爬取百度百科"猪八戒"页面词条及其副标题和链接,Python交流,编程语言专区,鱼C论坛

lzb1001 发表于 2022-4-9 15:20:06

爬取百度百科"猪八戒"页面词条及其副标题和链接

【四个问题】

1、下面两个URL好像都可以，为什么会这样？主要应该看哪里用哪个呢？
url = '……/search/word?%s' % param
url = '……/item/%s' % param

2、用了列表分片后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或小于5比如或仅用[:-1]还会报错无法运行，这是为什么？

3、如果要加入伪装的header，上述headers用法是否正确？
headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}

4、输入关键词“黄埔军校”，刚开始正常抓取，快结束时会报如下错误，不知道什么原因
Traceback (most recent call last):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
self.send(msg)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
self.connect()
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\work\p14_92.py", line 35, in <module>
main()
  File "D:\work\p14_92.py", line 26, in main
response2 = urllib.request.urlopen(req2)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
response = self._open(req, data)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
'_open', req)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error getaddrinfo failed>

suchocolate 发表于 2022-4-10 16:13:55

代码发出来。

lzb1001 发表于 2022-4-10 20:02:12

suchocolate 发表于 2022-4-10 16:13
代码发出来。

代码发出来需要很久才能显示，不知道是不是因为代码里含有网址需要人工审核的原因

lzb1001 发表于 2022-4-10 20:14:02

suchocolate 发表于 2022-4-10 16:13
代码发出来。

我的python版本是3.7.6

import urllib.request
import urllib.parse
import re
from bs4 import BeautifulSoup

def main():
keyword = input('请输入检索关键词：')
param = urllib.parse.urlencode({'word':keyword}) # word=%E7%8C%AA%E5%85%AB%E6%88%92
url = 'https://baike.baidu.com/search/word?%s' % param # ->%E7%8C%AA%E5%85%AB%E6%88%92
#url = 'https://baike.baidu.com/item/%s' % param

#headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'} # 可否隐藏headers???

req = urllib.request.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36') # 修改User-Agent隐藏
response = urllib.request.urlopen(req)
html = response.read().decode('utf-8') # .decode('utf-8')此处可不需要
soup = BeautifulSoup(html, 'html.parser')
print(req.headers) # 查看隐藏后的headers

for each in soup.find_all(href = re.compile('item')):
   content = ''.join() # 问题1：这里没看懂什么意思？为什么要用join()方法？另外text这个单词从哪里来的呢？
   url2 = ''.join(['http://baike.baidu.com', each['href'] if "%" in each['href'] else urllib.parse.quote(each['href'])]) # 判断是否有进行了 url 编码，没有的话对它进行编码
   req2 = urllib.request.Request(url2)
   response2 = urllib.request.urlopen(req2)
   html2 = response2.read().decode('utf-8') # .decode('utf-8')此处可不需要
   soup2 = BeautifulSoup(html2, 'html.parser')
   if soup2.h2:
         content = ''.join() # 问题2：输出结果词条名（副标题），副标题的括号哪里产生的呢？
   content = ''.join()
   print(content)


if __name__ == '__main__':
main()

问题3：经测试，以下两个URL好像都可以：
（1）url = '……/item/%s' % param
（2）url = '……/search/word?%s' % param
为什么？一般看哪里&用哪个？

问题4：关于for each in soup.find_all(href = re.compile('item')):
经测试，完全不用分片或定义分片[:-1]都无法正常运行，而且需要才能把开头无用结果完全过滤，为什么？

问题5：上面两种设置headers的方法都可以吗？

问题6：关键词“黄埔军校”，开始正常爬取，但最后返回错误，经查该词条最后四个链接无法爬取和显示，该如何处理？

问题比较多，麻烦大神不厌其烦耐心解答，不甚感谢！

lzb1001 发表于 2022-4-10 20:15:29

suchocolate 发表于 2022-4-10 16:13
代码发出来。

代码已发，麻烦审核并解答下，谢谢大神。

suchocolate 发表于 2022-4-16 09:46:43

1、下面两个URL好像都可以，为看哪什么会这样？主要应该里用哪个呢？
正常的URL一般格式
http://www.jianshu.com/search;user?q=python&page=1&type=collections#chapter1
:// 前面的就是 scheme，代表协议，上面是http
第3个/ 前的www.jianshu.com网站域名
第3个/后面是你要访问的资源路径
分号；后面是 params，代表参数，但实际当中用的少，而是用下面的query作为参数
问号？后面是查询条件 query，一般用作 GET 类型的 URL；格式是key=value，多个查询条件&分隔
井号 #后面是锚点，用于直接定位页面内部的下拉位置。

按照规范，（未编码前）你的url应该是 https://baike.baidu.com/search/word?word='猪八戒'，如果你了之后，就去掉了word=，导致url不正确，不确定百度能正确解析你的URL。

URL解析参考 3.1.3章节：https://github.com/Python3WebSpider/Python3WebSpider/blob/master/3.1-%E4%BD%BF%E7%94%A8urllib.md

2、用了列表分片后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或小于5比如或仅用[:-1]还会报错无法运行，这是为什么？
哪里分片，结果？

3、如果要加入伪装的header，上述headers用法是否正确？
header是http规范要求的，其中user-agent必须携带的字段，但规范没有要求必须用什么，只是表示请求人的信息。
如果我们不设置，默认是 python-urllib 或 python-requests，看你使用的库。
但目前实际情况，有的网站会检查user-agent，如果发现其中带python或者urllib等字样，就拒绝服务，返回418给你。所以目前使用都会填一个不包含python或urllib字样的信息，想逼真一点就用代码里的内容没有错。

4、输入关键词“黄埔军校”，刚开始正常抓取，快结束时会报如下错误，不知道什么原因
抓太频繁了，重启python

5.content = ''.join() # 问题1：这里没看懂什么意思？为什么要用join()方法？另外text这个单词从哪里来的呢？
join的目的是让不连续的列表组成一个长字符串，为后续操作做准备。
each 对象是soup对象，text是soup对象自带属性，包含soup的实际文本。

6.content = ''.join() # 问题2：输出结果词条名（副标题），副标题的括号哪里产生的呢？
官网带的

建议你从基础系统学起，事半功倍，没有基础直接上手，浪费时间，学不扎实。

lzb1001 发表于 2022-4-19 10:43:12

suchocolate 发表于 2022-4-16 09:46
1、下面两个URL好像都可以，为看哪什么会这样？主要应该里用哪个呢？
正常的URL一般格式
http://www.jian ...

感谢解答。

2、用了列表分片后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或小于5比如或仅用[:-1]还会报错无法运行，这是为什么？
哪里分片，结果？

我说错了，不是，是
整句代码for each in soup.find_all(href = re.compile('item')):

suchocolate 发表于 2022-4-20 08:39:07

lzb1001 发表于 2022-4-19 10:43
感谢解答。

2、用了列表分片后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或 ...

把你遇到的报错发出来。

lzb1001 发表于 2022-4-20 15:16:35

suchocolate 发表于 2022-4-20 08:39
把你遇到的报错发出来。

当把代码中的分片改为：

for each in soup.find_all(href = re.compile('item')):或for each in soup.find_all(href = re.compile('item'))[:-1]:时，

运行代码：

请输入检索关键词：王新衡
word=%E7%8E%8B%E6%96%B0%E8%A1%A1
https://baike.baidu.com/search/word?word=%E7%8E%8B%E6%96%B0%E8%A1%A1
https://baike.baidu.com/item/%E7%8E%8B%E6%96%B0%E8%A1%A1
{'User-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
self.connect()
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\work\p14_92.py", line 48, in <module>
main()
File "D:\work\p14_92.py", line 36, in main
response2 = urllib.request.urlopen(req2) # 将Request对象req2传给urlopen函数
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
response = self._open(req, data)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
'_open', req)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error getaddrinfo failed>

lzb1001 发表于 2022-4-20 18:19:55

本帖最后由 lzb1001 于 2022-4-20 18:21 编辑

4、输入关键词“黄埔军校”，刚开始正常抓取，快结束时会报如下错误，不知道什么原因
抓太频繁了，重启python

试了还是不行，该词条最后四个链接（如下）还是无法抓取：

黄埔军校西点军校桑赫斯特皇家军事学院伏龙芝军事学院

可否请大神进入该词条看下以上四个链接？

附：返回错误提示如下
Traceback (most recent call last):
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
self.connect()
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\work\p14_92.py", line 48, in <module>
main()
File "D:\work\p14_92.py", line 36, in main
response2 = urllib.request.urlopen(req2) # 将Request对象req2传给urlopen函数
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
response = self._open(req, data)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
'_open', req)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error getaddrinfo failed>

页: [1]

鱼C论坛's Archiver

爬取百度百科"猪八戒"页面词条及其副标题和链接