初级爬虫遇到问题

470459574

>>> import requests
>>> from lxml import etree
>>> url="https://book.douban.com/subject/34855046/comments/"
>>> r=requests.get(url).text
>>> s=etree.HTML(r)
>>> print(s.xpath('//div[@class="comment"]/h3/p/span/text()'))
Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    print(s.xpath('//div[@class="comment"]/h3/p/span/text()'))
AttributeError: 'NoneType' object has no attribute 'xpath'


idle里写的代码，刚学一天，按照老师的代码写的，不知道为什么每次都是提醒没有xpath对象，然后，如果对了，也没有text文字，都是空的"[]"符号显示，真的不懂为什么。。。求救啊！

wp231957

把headers加上再试一下

wp231957

xpath 也不对

>>> print(s.xpath('//div[@class="comment"]/p/span/text()'))
['比起你只记得伤害过我，我宁愿你忘了曾经伤害过我，这样，我也许会好过一点。\r以及，绘制作者太强了...分镜强，蒙太奇也强， 把主角被回忆吞噬的那种漩涡感描绘地非常极致，方方面面的强。', '也许有一天，我会忘记全世界，但我还是会拼凑着记忆的碎片，寻找你。', '面粉变成面包很容易，面包变回面粉不太可能。阿兹海默就是不可逆的一件事。世界的增墒中，苦是一个增长的变量。', '半夜看完快哭了，选的细节太戳人，讲述顺序也是全新的。阿兹海默是另一种绝症，总看到那些患病老人出去找孩子的新闻，从没想过他们正被记忆里的情感反复绞杀，这有得多痛苦。', '还原了阿尔兹海默症视角下的记忆，很喜欢那些克制的独白。一想到阿尔兹海默症的生活中每天都是这样的错落感，再看到那句“这样的事情会发生在我们身上吗”就觉得好难过', '黄色水笔与黑色铅笔分别勾勒两种世界，一种展现阿尔茨海默症的独特艺术形式。阿尔茨海默的世界脆弱、易逝，时时刻刻有被擦去的风险。', '如果有一天我的大脑忘记了你\n不要怕\n我的心还记得你\n因为记忆会衰退，爱不会', '每次醒来就像经历了一场生离死别']
>>>

470459574

wp231957 发表于 2019-12-17 15:10
把headers加上再试一下

请问一下，headers应该加在哪？我把headers,加了之后，还是报错了

470459574

wp231957 发表于 2019-12-17 15:10
把headers加上再试一下

报错：AttributeError: 'set' object has no attribute 'items'
大神，应该怎么做？

wp231957

470459574 发表于 2019-12-17 17:21
报错：AttributeError: 'set' object has no attribute 'items'
大神，应该怎么做？

上代码啊
你不拿代码，我咋知道

470459574

wp231957 发表于 2019-12-17 17:24
上代码啊
你不拿代码，我咋知道

>>> import requests
>>> from lxml import etree
>>> url="https://book.douban.com/subject/34855046/comments/"
>>> headers={'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding:gzip, deflate, sdch, br',
'Accept-Language:zh-CN,zh;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Cookie:bid=BVPnSw1A4hc; douban-fav-remind=1; __guid=81379588.3227039460351705600.1576473377722.9248; ll="118190"; viewed="34855046"; gr_user_id=626edbd4-0f62-4005-9b6b-db4210750454; __yadk_uid=eiuTNT42na7BVZx37SHKpzYjhVSUJ0FO; monitor_count=5; __utmt_douban=1; __utma=30149280.163854509.1571388038.1576559777.1576573458.4; __utmb=30149280.1.10.1576573458; __utmc=30149280; __utmz=30149280.1576559777.3.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmt=1; __utma=81379588.1939606557.1576473379.1576559777.1576573458.3; __utmb=81379588.1.10.1576573458; __utmc=81379588; __utmz=81379588.1576559777.2.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; ap_v=0,6.0; _pk_ref.100001.3ac3=%5B%22%22%2C%22%22%2C1576573458%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_id.100001.3ac3=fe9cb3c0780ea04c.1576473379.3.1576573458.1576559814.; _pk_ses.100001.3ac3=*',
'Host:book.douban.com',
'Upgrade-Insecure-Requests:1',
'User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}
>>> r=requests.get(url,headers=headers)
Traceback (most recent call last):
  File "<pyshell#13>", line 1, in <module>
    r=requests.get(url,headers=headers)
  File "C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\models.py", line 314, in prepare
    self.prepare_headers(headers)
  File "C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\models.py", line 446, in prepare_headers
    for header in headers.items():
AttributeError: 'set' object has no attribute 'items'

470459574

wp231957 发表于 2019-12-17 17:24
上代码啊
你不拿代码，我咋知道

>>> import requests
>>> from lxml import etree
>>> url="https://book.douban.com/subject/34855046/comments/"
>>> headers={'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding:gzip, deflate, sdch, br',
'Accept-Language:zh-CN,zh;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Cookie:bid=BVPnSw1A4hc; douban-fav-remind=1; __guid=81379588.3227039460351705600.1576473377722.9248; ll="118190"; viewed="34855046"; gr_user_id=626edbd4-0f62-4005-9b6b-db4210750454; __yadk_uid=eiuTNT42na7BVZx37SHKpzYjhVSUJ0FO; monitor_count=5; __utmt_douban=1; __utma=30149280.163854509.1571388038.1576559777.1576573458.4; __utmb=30149280.1.10.1576573458; __utmc=30149280; __utmz=30149280.1576559777.3.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmt=1; __utma=81379588.1939606557.1576473379.1576559777.1576573458.3; __utmb=81379588.1.10.1576573458; __utmc=81379588; __utmz=81379588.1576559777.2.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; ap_v=0,6.0; _pk_ref.100001.3ac3=%5B%22%22%2C%22%22%2C1576573458%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_id.100001.3ac3=fe9cb3c0780ea04c.1576473379.3.1576573458.1576559814.; _pk_ses.100001.3ac3=*',
'Host:book.douban.com',
'Upgrade-Insecure-Requests:1',
'User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}
>>> r=requests.get(url,headers=headers)

流羽星

你的xpath路径不对吧

470459574

流羽星 发表于 2019-12-17 20:40
你的xpath路径不对吧

路径已经参照一楼改过了，不知道怎么的还是不行

gywjj

这 个问题应该一级一级的查，你首先看一下你 的r有没有出来结果，这个网页有反爬机制，如果没有出来结果，那么后面的xpath有可能也会出错，出来［］的时候，应该是r里 ，全是乱码，我爬出来就是乱码，还在研究

szbin

触发了反爬机制了。

470459574

szbin 发表于 2019-12-18 09:33
触发了反爬机制了。

应该是触发反扒机制了，我增加延迟试一试

XiaoPaiShen

1. import requests
   
2. from lxml import etree
   
3. 
   
4. url="https://book.douban.com/subject/34855046/comments/"
   
5. 
   
6. headers={'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
   
7. 'Accept-Encoding':'gzip, deflate',
   
8. 'Accept-Language':'zh-CN,zh;q=0.8',
   
9. 'Cache-Control':'max-age=0',
   
10. 'Connection':'keep-alive',
    
11. 'Cookie':'bid=BVPnSw1A4hc; douban-fav-remind=1; __guid=81379588.3227039460351705600.1576473377722.9248; ll="118190"; viewed="34855046"; gr_user_id=626edbd4-0f62-4005-9b6b-db4210750454; __yadk_uid=eiuTNT42na7BVZx37SHKpzYjhVSUJ0FO; monitor_count=5; __utmt_douban=1; __utma=30149280.163854509.1571388038.1576559777.1576573458.4; __utmb=30149280.1.10.1576573458; __utmc=30149280; __utmz=30149280.1576559777.3.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmt=1; __utma=81379588.1939606557.1576473379.1576559777.1576573458.3; __utmb=81379588.1.10.1576573458; __utmc=81379588; __utmz=81379588.1576559777.2.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; ap_v=0,6.0; _pk_ref.100001.3ac3=%5B%22%22%2C%22%22%2C1576573458%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_id.100001.3ac3=fe9cb3c0780ea04c.1576473379.3.1576573458.1576559814.; _pk_ses.100001.3ac3=*',
    
12. 'Host':'book.douban.com',
    
13. 'Upgrade-Insecure-Requests':'1',
    
14. 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}
    
15. 
    
16. req = requests.get(url, headers=headers)
    
17. result = etree.HTML(req.text)
    
18. 
    
19. print(result.xpath('//div[@class="comment"]/p/span/text()'))

复制代码

XiaoPaiShen

我贴代码居然需要审核？

header 都写错了，
格式 是： 'Accept-Encoding':'gzip, deflate',
key 和 value 都是字符串

'Accept-Encoding' 这一行应该是
'Accept-Encoding':'gzip, deflate',

xpath 写的也不对
应该是：'//div[@class="comment"]/p/span/text()'

pachong

_2_

s 的值为 None ，没有 xpath 属性

470459574

XiaoPaiShen 发表于 2019-12-21 12:40
我贴代码居然需要审核？

header 都写错了，

Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import requests
>>> from lxml import etree
>>> url='https://book.douban.com/subject/34855046/comments/'
>>> header=({})
>>> header=({'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, br',
'Accept-Language':'zh-CN,zh;q=0.9',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Cookie':'ll="118190"; bid=Jf2I4eQKZ3M; __utmz=30149280.1576501369.1.1.utmcsr=so.com|utmccn=(referral)|utmcmd=referral|utmcct=/link; _vwo_uuid_v2=D902CEF0CAFD9DD2ED5C1FC29F0341359|60ff20567c843d281177ed36db3b1b47; __guid=81379588.1208348864452897300.1576502199254.9558; __utmz=81379588.1576583158.2.2.utmcsr=so.com|utmccn=(organic)|utmcmd=organic|utmctr=%e8%b1%86%e7%93%a3; gr_user_id=0716dd15-0d93-4c8a-9393-22da4fadc96b; __yadk_uid=T7IH2ohDF4WpJgvN0a0cZ1j1TSZselhl; viewed="25862578_34856416"; monitor_count=1; ap_v=0,6.0; _pk_ref.100001.3ac3=%5B%22%22%2C%22%22%2C1576997204%2C%22https%3A%2F%2Fwww.so.com%2Fs%3Fq%3D%25e8%25b1%2586%25e7%2593%25a3%26src%3D360_wsyjsw%26ls%3Dsn2321298%26lm_extend%3Dctype%3A4%26plat_info%3DGUANJIA.1497610398.c5299c5fba%22%5D; _pk_id.100001.3ac3=a4c68420a8ede957.1576502200.5.1576997204.1576844669.; _pk_ses.100001.3ac3=*; __utma=30149280.675138352.1576501369.1576844669.1576997204.5; __utmc=30149280; __utmt_douban=1; __utmb=30149280.1.10.1576997204; __utma=81379588.1499720999.1576502200.1576844669.1576997204.5; __utmc=81379588; __utmt=1; __utmb=81379588.1.10.1576997204',
'Host':'book.douban.com',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'})
>>> r=requests.get(url,headers=headers)
Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    r=requests.get(url,headers=headers)
NameError: name 'headers' is not defined
我这里怎么说headers没有定义

zltzlt

470459574 发表于 2019-12-22 14:58
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
Type "c ...

应该是 r=requests.get(url, headers=header)，多了个 s

XiaoPaiShen

470459574 发表于 2019-12-22 14:58
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
Type "c ...

前面已经有代码了，存到 .py 文件里执行不是更方便吗？