Scrapy框架中parse()函数中两个包含URL的文件没有被创建
spiderimport scrapy
class QuotesSpider(scrapy.Spider):
name = 'quotes' #蜘蛛的名字
def start_requests(self): #包含了spider在启动时进行爬取的URL列表，该函数必须返回Request对象或者生成器
allowed_domains = ['quotes.toscrape.com'] #蜘蛛的范围
urls = [
'http://www.quotes.toscrape.com/page/1/'
'http://www.quotes.toscrape.com/page/2/'
] #从哪里开始爬
for url in urls:
yield scrapy.Request(url=url,callback=self.parse)
def parse(self,response): #该函数负责解析返回的数据，提取数据以及生成需要进一步处理的URL的Request对象
page = response.url.split('/')[-2]
filename = 'quotes-%s.html'%page
with open(filename,"wb") as f:
f.write(response.body)
self.log('Save file %s'%filename)
爬：
(base) C:\Users\admin\Desktop\tutorial>scrapy crawl quotes
2020-06-09 19:57:23 INFO: Scrapy 2.1.0 started (bot: tutorial)
2020-06-09 19:57:23 INFO: Versions: lxml 4.5.1.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.6 (default, Jan8 2020, 20:23:39) , pyOpenSSL 19.1.0 (OpenSSL 1.1.1g21 Apr 2020), cryptography 2.8, Platform Windows-10-10.0.18362-SP0
2020-06-09 19:57:23 DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-06-09 19:57:23 INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
'NEWSPIDER_MODULE': 'tutorial.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['tutorial.spiders']}
2020-06-09 19:57:23 INFO: Telnet Password: b8b0c838b5723910
2020-06-09 19:57:23 INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2020-06-09 19:57:24 INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-06-09 19:57:24 INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-06-09 19:57:24 INFO: Enabled item pipelines:
[]
2020-06-09 19:57:24 INFO: Spider opened
2020-06-09 19:57:24 INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-06-09 19:57:24 INFO: Telnet console listening on 127.0.0.1:6023
2020-06-09 19:57:28 DEBUG: Retrying <GET http://www.quotes.toscrape.com/robots.txt> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:30 DEBUG: Retrying <GET http://www.quotes.toscrape.com/robots.txt> (failed 2 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:32 ERROR: Gave up retrying <GET http://www.quotes.toscrape.com/robots.txt> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:32 ERROR: Error downloading <GET http://www.quotes.toscrape.com/robots.txt>: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
Traceback (most recent call last):
File "C:\Users\admin\miniconda3\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:34 DEBUG: Retrying <GET http://www.quotes.toscrape.com/page/1/http://www.quotes.toscrape.com/page/2/> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:37 DEBUG: Retrying <GET http://www.quotes.toscrape.com/page/1/http://www.quotes.toscrape.com/page/2/> (failed 2 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:39 ERROR: Gave up retrying <GET http://www.quotes.toscrape.com/page/1/http://www.quotes.toscrape.com/page/2/> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:44 ERROR: Error downloading <GET http://www.quotes.toscrape.com/page/1/http://www.quotes.toscrape.com/page/2/>
Traceback (most recent call last):
File "C:\Users\admin\miniconda3\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2020-06-09 19:57:44 INFO: Closing spider (finished)
2020-06-09 19:57:44 INFO: Dumping Scrapy stats:
{'downloader/exception_count': 6,
'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 6,
'downloader/request_bytes': 1503,
'downloader/request_count': 6,
'downloader/request_method_count/GET': 6,
'elapsed_time_seconds': 20.184052,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 6, 9, 11, 57, 44, 311541),
'log_count/DEBUG': 4,
'log_count/ERROR': 4,
'log_count/INFO': 10,
'retry/count': 4,
'retry/max_reached': 2,
'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 4,
"robotstxt/exception_count/<class 'twisted.web._newclient.ResponseNeverReceived'>": 1,
'robotstxt/request_count': 1,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2020, 6, 9, 11, 57, 24, 127489)}
2020-06-09 19:57:44 INFO: Spider closed (finished)
最后，没找到quotes-1.html 和quotes-2.html
换了个网址可以了 {:10_267:}，有问题的网址，你还去请求，看教程，不看网址吧{:10_323:} Stubborn 发表于 2020-6-10 00:36
，有问题的网址，你还去请求，看教程，不看网址吧
{:5_109:} 莫待无花空折枝 发表于 2020-6-10 09:21
楼主换的什么网址啊
页:
[1]