[已解决]关于scrapy爬虫报错301重定向问题

amber1111

最近写了一个scrapy爬虫爬取淘宝数据，但是没次都爬不全，最后发现爬取部分网页都会报错301，已经伪装了user agent，加上Referer，但是还是有部分网页无法爬取，请问大家遇到这种情况是怎么解决的呢，以下是部分错误代码：
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://item.taobao.com/item.htm?id=548573777437> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
https://item.taobao.com/item.htm?id=552267374137
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://item.taobao.com/item.htm?id=571301219510> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
https://item.taobao.com/item.htm?id=558688071694
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://item.taobao.com/item.htm?id=570309089342> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
https://item.taobao.com/item.htm?id=521785652630
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://item.taobao.com/item.htm?id=571296540068> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
2018-11-06 14:27:39 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <301 https://item.taobao.com/item.htm?id=553289240364>: HTTP status code is not handled or not allowed
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://item.taobao.com/item.htm?id=537598464080> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://item.taobao.com/item.htm?id=546867778273> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
https://item.taobao.com/item.htm?id=561112325868
https://item.taobao.com/item.htm?id=566378917121
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://item.taobao.com/item.htm?id=45033959543> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://item.taobao.com/item.htm?id=570346798714> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://item.taobao.com/item.htm?id=578947695290> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
2018-11-06 14:27:39 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://item.taobao.com/item.htm?id=13298514524> (referer: https://s.taobao.com/search?q=%E6%98%BE%E7%A4%BA%E5%99%A8)
https://item.taobao.com/item.htm?id=523249394550
https://item.taobao.com/item.htm?id=548573777437

最佳答案

月排行榜 / 总排行榜

wongyusing

2018-11-6 19:26:35

请求头的参数不够。  
现在淘宝需要登录才能看到数据。
需要你登录后的曲奇（曲奇英文自行翻译）

还有就是需要把速度降下来。  
速度过快也会导致淘宝对你重定向到验证或者登录页面

跳转到最佳答案楼层

wongyusing

请求头的参数不够。  
现在淘宝需要登录才能看到数据。
需要你登录后的曲奇（曲奇英文自行翻译）

还有就是需要把速度降下来。  
速度过快也会导致淘宝对你重定向到验证或者登录页面