[已解决]关于scrapy抓取出现的问题

8916616

日志文件出现这个提示请问需要怎么解决...
ERROR: Spider error processing <GET https://www.zongheng.com/book/1208364.html> (referer: http://www.zongheng.com/)
Traceback (most recent call last)
这是爬取的第三级页面，第一、二级界面返回的数据都能正确打印，就第三级的数据没有反应了，用的pacharm，

最佳答案

月排行榜 / 总排行榜

isdkz

2023-2-5 23:38:02

8916616 发表于 2023-2-5 23:36
Traceback (most recent call last):
  File "E:\python\Lib\site-packages\scrapy%utils\defer.py", li ...

应该是这一句出错 scrapy.Request(two_url,callback = self.parse_three,headers = headers )

url 你传了一个列表，但是它只接收字符串

跳转到最佳答案楼层

isdkz

日志关键的部分没有放出来

要看 TraceBack 那一部分

方便的话就把整个项目文件夹压缩或者相关的截图放上来

8916616

isdkz 发表于 2023-2-5 23:25
日志关键的部分没有放出来

要看 TraceBack 那一部分

Traceback (most recent call last):
  File "E:\python\Lib\site-packages\scrapy\utils\defer.py", line 240, in iter_errback
    yield next(it)
          ^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\utils\python.py", line 338, in __next__
    return next(self.data)
           ^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\utils\python.py", line 338, in __next__
    return next(self.data)
           ^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\core\spidermw.py", line 79, in process_sync
    for r in iterable:
  File "E:\python\Lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in <genexpr>
    return (r for r in result or () if self._filter(r, spider))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\core\spidermw.py", line 79, in process_sync
    for r in iterable:
  File "E:\pycharmprojects\pythonproject\爬取纵横\zh\zh\middlewares.py", line 36, in process_spider_output
    for i in result:
  File "E:\python\Lib\site-packages\scrapy\core\spidermw.py", line 79, in process_sync
    for r in iterable:
  File "E:\python\Lib\site-packages\scrapy\spidermiddlewares\referer.py", line 336, in <genexpr>
    return (self._set_referer(r, response) for r in result or ())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\core\spidermw.py", line 79, in process_sync
    for r in iterable:
  File "E:\python\Lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 28, in <genexpr>
    return (r for r in result or () if self._filter(r, spider))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\core\spidermw.py", line 79, in process_sync
    for r in iterable:
  File "E:\python\Lib\site-packages\scrapy\spidermiddlewares\depth.py", line 32, in <genexpr>
    return (r for r in result or () if self._filter(r, response, spider))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\core\spidermw.py", line 79, in process_sync
    for r in iterable:
  File "E:\pycharmprojects\pythonproject\爬取纵横\zh\zh\spiders\zhpq.py", line 31, in parse_two
    yield scrapy.Request(two_url,callback = self.parse_three,headers = headers )
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\python\Lib\site-packages\scrapy\http\request\__init__.py", line 60, in __init__
    self._set_url(url)
  File "E:\python\Lib\site-packages\scrapy\http\request\__init__.py", line 98, in _set_url
    raise TypeError(f"Request url must be str, got {type(url).__name__}")
TypeError: Request url must be str, got list
2023-02-05 23:31:50 [scrapy.core.engine] INFO: Closing spider (finished)
2023-02-05 23:31:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 693,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 28209,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'elapsed_time_seconds': 0.758224,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2023, 2, 5, 15, 31, 50, 689839),
'httpcompression/response_bytes': 192444,
'httpcompression/response_count': 2,
'log_count/DEBUG': 9,
'log_count/ERROR': 1,
'log_count/INFO': 12,
'request_depth_max': 1,
'response_received_count': 2,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'spider_exceptions/TypeError': 1,
'start_time': datetime.datetime(2023, 2, 5, 15, 31, 49, 931615)}
2023-02-05 23:31:50 [scrapy.core.engine] INFO: Spider closed (finished)
这是TraceBack部分的

isdkz

8916616 发表于 2023-2-5 23:36
Traceback (most recent call last):
  File "E:\python\Lib\site-packages\scrapy%utils\defer.py", li ...

应该是这一句出错 scrapy.Request(two_url,callback = self.parse_three,headers = headers )

url 你传了一个列表，但是它只接收字符串

8916616

isdkz 发表于 2023-2-5 23:38
应该是这一句出错 scrapy.Request(two_url,callback = self.parse_three,headers = headers )

url  ...

谢谢了，自己找半天没找出问题

isdkz

8916616 发表于 2023-2-5 23:40
谢谢了，自己找半天没找出问题

看日志看 traceback 部分，越往上的是最先抛出的，往往出错的地方在后面（不一定是最后）抛出的地方，

因为前面抛出的错误一般都是你在调用接口的时候传的数据类型不对导致的

8916616

isdkz 发表于 2023-2-5 23:43
看日志看 traceback 部分，越往上的是最先抛出的，往往出错的地方在后面（不一定是最后）抛出的地方，
...

好的好的才开始学习，还不怎么会看这些