[已解决]Scrapy 未自动初始化sel变量，如何手动

贰耇孜

网上都说 response会自动初始化一个sel变量，但是我的没有 提示NameError: name 'sel' is not defined
python 3.8.4   pip安装的scrapy


具体代码

>>> sel.xpath("//ul/li/h4/text()").extract()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: name 'sel' is not defined、

最佳答案

月排行榜 / 总排行榜

Twilight6

2020-8-22 16:55:27

请发完整代码

跳转到最佳答案楼层

Twilight6

请发完整代码

贰耇孜

C:\Users\Desktop\tutorial>scrapy shell "https://www.dmozdir.org/Category/?SmallPath=20"
2020-08-22 16:19:48 [scrapy.utils.log] INFO: Scrapy 2.3.0 started (bot: tutorial)
2020-08-22 16:19:48 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 3.0, Platform Windows-10-10.0.18362-SP0
2020-08-22 16:19:48 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-08-22 16:19:48 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
'LOGSTATS_INTERVAL': 0,
'NEWSPIDER_MODULE': 'tutorial.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['tutorial.spiders']}
2020-08-22 16:19:48 [scrapy.extensions.telnet] INFO: Telnet Password: 6fbd425e0ac85fd6
2020-08-22 16:19:48 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole']
2020-08-22 16:19:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-08-22 16:19:49 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-08-22 16:19:49 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-08-22 16:19:49 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-08-22 16:19:49 [scrapy.core.engine] INFO: Spider opened
2020-08-22 16:19:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.dmozdir.org/robots.txt> (referer: None)
2020-08-22 16:19:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.dmozdir.org/Category/?SmallPath=20> (referer: None)
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x00000275FC858280>
[s]   item       {}
[s]   request    <GET https://www.dmozdir.org/Category/?SmallPath=20>
[s]   response   <200 https://www.dmozdir.org/Category/?SmallPath=20>
[s]   settings   <scrapy.settings.Settings object at 0x00000275FC8580A0>
[s]   spider     <DmozSpider 'dmoz' at 0x275fcd18a90>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
>>> sel.xpath("//ul/li/h4/text()").extract()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: name 'sel' is not defined
>>>

sunrise085

python3.8.4不是有问题的么？
不兼容中文，现在问题解决了么？
很多人遇到这个问题，现在都升级到3.8.5或者降级到其他版本了。你没遇到这个问题？

贰耇孜

>>> response.xpath("//title")
[<Selector xpath='//title' data='<title>视频影视-娱乐休闲-目录分类-DMOZ中文网站分类目录</t...'>]
>>> response.xpath("//title/text()").extract
<bound method SelectorList.getall of [<Selector xpath='//title/text()' data='视频影视-娱乐休闲-目录分类-DMOZ中文网站分类目录'>]>
>>> response.xpath("//title/text()").extract()
['视频影视-娱乐休闲-目录分类-DMOZ中文网站分类目录']
>>> response.body
>>> sel.xpath("//ul/li/text()").extract()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: name 'sel' is not defined
>>> sel.xpath()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: name 'sel' is not defined

贰耇孜

Twilight6 发表于 2020-8-22 16:55
请发完整代码

代码在楼下

贰耇孜

sunrise085 发表于 2020-8-22 17:01
python3.8.4不是有问题的么？
不兼容中文，现在问题解决了么？
很多人遇到这个问题，现在都升级到3.8.5或 ...

我好像没遇到太多问题 就没升级 我升级一下试试

贰耇孜

sunrise085 发表于 2020-8-22 17:01
python3.8.4不是有问题的么？
不兼容中文，现在问题解决了么？
很多人遇到这个问题，现在都升级到3.8.5或 ...

看错了 我的就是3，85

贰耇孜

感谢各位大佬  已解决  手动sel=response 就搞定了