|
马上注册,结交更多好友,享用更多功能^_^
您需要 登录 才可以下载或查看,没有账号?立即注册
x
本帖最后由 lzb1001 于 2022-5-8 00:19 编辑
D:\work\tutorial>scrapy shell 'http://dmoztools.net/Computers/Programming/Languages/Python/Books/'
2022-05-07 23:56:37 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: tutorial)
2022-05-07 23:56:37 [scrapy.utils.log] INFO: Versions: lxml 4.8.0.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 3.0.3 3 May 2022), cryptography 37.0.2, Platform Windows-10-10.0.19041-SP0
2022-05-07 23:56:37 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
'LOGSTATS_INTERVAL': 0,
'NEWSPIDER_MODULE': 'tutorial.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['tutorial.spiders']}
2022-05-07 23:56:37 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-05-07 23:56:37 [scrapy.extensions.telnet] INFO: Telnet Password: b1f68aed69443fa2
2022-05-07 23:56:37 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole']
2022-05-07 23:56:37 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-05-07 23:56:37 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-05-07 23:56:37 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-05-07 23:56:37 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-05-07 23:56:37 [scrapy.core.engine] INFO: Spider opened
2022-05-07 23:56:38 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://'http/robots.txt>: invalid hostname: 'http
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\core\downloader\middleware.py", line 49, in process_request
return (yield download_func(request=request, spider=spider))
ValueError: invalid hostname: 'http
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\Scripts\scrapy.exe\__main__.py", line 7, in <module>
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\cmdline.py", line 145, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\cmdline.py", line 100, in _run_print_help
func(*a, **kw)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\cmdline.py", line 153, in _run_command
cmd.run(args, opts)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\commands\shell.py", line 74, in run
shell.start(url=url, redirect=not opts.no_redirect)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\shell.py", line 43, in start
self.fetch(url, spider, redirect=redirect)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\shell.py", line 111, in fetch
reactor, self._schedule, request, spider)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\twisted\internet\threads.py", line 120, in blockingCallFromThread
result.raiseException()
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\twisted\python\failure.py", line 500, in raiseException
raise self.value.with_traceback(self.tb)
ValueError: invalid hostname: 'http
D:\work\tutorial>
按小甲鱼视频中讲解,运行最后将返回>>>,但看上面好像不是,最后又返回D:\work\tutorial>,而且还有一大堆错误,是哪里有问题哦?
我将命令行中的网址的单引号换成双引号:---是不是必须用双引号,不能用单引号啊???
D:\work\tutorial>scrapy shell "http://dmoztools.net/Computers/Programming/Languages/Python/Books/"
2022-05-08 00:00:37 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: tutorial)
2022-05-08 00:00:37 [scrapy.utils.log] INFO: Versions: lxml 4.8.0.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 3.0.3 3 May 2022), cryptography 37.0.2, Platform Windows-10-10.0.19041-SP0
2022-05-08 00:00:37 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
'LOGSTATS_INTERVAL': 0,
'NEWSPIDER_MODULE': 'tutorial.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['tutorial.spiders']}
2022-05-08 00:00:37 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-05-08 00:00:37 [scrapy.extensions.telnet] INFO: Telnet Password: bb2332e9ee9c5698
2022-05-08 00:00:37 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole']
2022-05-08 00:00:38 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-05-08 00:00:38 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-05-08 00:00:38 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-05-08 00:00:38 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-05-08 00:00:38 [scrapy.core.engine] INFO: Spider opened
2022-05-08 00:00:38 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://dmoztools.net/robots.txt> from <GET http://dmoztools.net/robots.txt>
2022-05-08 00:00:38 [py.warnings] WARNING: C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\site-packages\scrapy\core\engine.py:276: ScrapyDeprecationWarning: Passing a 'spider' argument to ExecutionEngine.download is deprecated
return self.download(result, spider) if isinstance(result, Request) else result
2022-05-08 00:00:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://dmoztools.net/robots.txt> (referer: None)
2022-05-08 00:00:40 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://dmoztools.net/Computers/Programming/Languages/Python/Books/> from <GET http://dmoztools.net/Computers/Programming/Languages/Python/Books/>
2022-05-08 00:00:40 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://dmoztools.net/Computers/Programming/Languages/Python/Books/> (referer: None)
2022-05-08 00:00:41 [asyncio] DEBUG: Using selector: SelectSelector
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x000001A105E874C8>
[s] item {}
[s] request <GET http://dmoztools.net/Computers/Programming/Languages/Python/Books/>
[s] response <404 https://dmoztools.net/Computers/Programming/Languages/Python/Books/>
[s] settings <scrapy.settings.Settings object at 0x000001A105E83F08>
[s] spider <DmoztoolsSpider 'dmoztools' at 0x1a10656c308>
[s] Useful shortcuts:
[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s] fetch(req) Fetch a scrapy.Request and update local objects
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser
2022-05-08 00:00:41 [asyncio] DEBUG: Using selector: SelectSelector
In [1]:
显示结果怎么有删除线?我附上返回的结果图吧:
|
|