爬取百度百科"猪八戒"页面词条及其副标题和链接

lzb1001

【四个问题】

1、下面两个URL好像都可以，为什么会这样？主要应该看哪里用哪个呢？
url = '……/search/word?%s' % param
url = '……/item/%s' % param[5:]

2、用了列表分片[5:-1]后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或小于5比如[4:-1]或仅用[:-1]还会报错无法运行，这是为什么？

3、如果要加入伪装的header，上述headers用法是否正确？
headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}

4、输入关键词“黄埔军校”，刚开始正常抓取，快结束时会报如下错误，不知道什么原因
Traceback (most recent call last):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
    self.connect()
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 707, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
&#160;&#160;File "D:\work\p14_92.py", line 35, in <module>
&#160; &#160; main()
&#160;&#160;File "D:\work\p14_92.py", line 26, in main
&#160; &#160; response2 = urllib.request.urlopen(req2)
&#160;&#160;File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
&#160; &#160; return opener.open(url, data, timeout)
&#160;&#160;File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
&#160; &#160; response = self._open(req, data)
&#160;&#160;File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
&#160; &#160; '_open', req)
&#160;&#160;File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
&#160; &#160; result = func(*args)
&#160;&#160;File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1347, in http_open
&#160; &#160; return self.do_open(http.client.HTTPConnection, req)
&#160;&#160;File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1321, in do_open
&#160; &#160; raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

suchocolate

代码发出来。

lzb1001

suchocolate 发表于 2022-4-10 16:13
代码发出来。

代码发出来需要很久才能显示，不知道是不是因为代码里含有网址需要人工审核的原因

lzb1001

suchocolate 发表于 2022-4-10 16:13
代码发出来。

我的python版本是3.7.6

import urllib.request
import urllib.parse
import re
from bs4 import BeautifulSoup

def main():
    keyword = input('请输入检索关键词：')
    param = urllib.parse.urlencode({'word':keyword}) # word=%E7%8C%AA%E5%85%AB%E6%88%92
    url = 'https://baike.baidu.com/search/word?%s' % param # [5:]->%E7%8C%AA%E5%85%AB%E6%88%92
    #url = 'https://baike.baidu.com/item/%s' % param[5:]

    #headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'} # 可否隐藏headers???

    req = urllib.request.Request(url)
    req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36') # 修改User-Agent隐藏
    response = urllib.request.urlopen(req)
    html = response.read().decode('utf-8') # .decode('utf-8')此处可不需要
    soup = BeautifulSoup(html, 'html.parser')
    print(req.headers) # 查看隐藏后的headers

    for each in soup.find_all(href = re.compile('item'))[11:-1]:
        content = ''.join([each.text]) # 问题1：这里没看懂什么意思？为什么要用join()方法？另外text这个单词从哪里来的呢？
        url2 = ''.join(['http://baike.baidu.com', each['href'] if "%" in each['href'] else urllib.parse.quote(each['href'])]) # 判断是否有进行了 url 编码，没有的话对它进行编码
        req2 = urllib.request.Request(url2)
        response2 = urllib.request.urlopen(req2)
        html2 = response2.read().decode('utf-8') # .decode('utf-8')此处可不需要
        soup2 = BeautifulSoup(html2, 'html.parser')
        if soup2.h2:
            content = ''.join([content, soup2.h2.text]) # 问题2：输出结果词条名（副标题），副标题的括号哪里产生的呢？
        content = ''.join([content, ' -> ', url2])
        print(content)
        

if __name__ == '__main__':
    main()

问题3：经测试，以下两个URL好像都可以：
（1）url = '……/item/%s' % param[5:]
（2）url = '……/search/word?%s' % param
为什么？一般看哪里&用哪个？

问题4：关于for each in soup.find_all(href = re.compile('item'))[11:-1]:
经测试，完全不用分片或定义分片[:-1]都无法正常运行，而且需要[11:-1]才能把开头无用结果完全过滤，为什么？

问题5：上面两种设置headers的方法都可以吗？

问题6：关键词“黄埔军校”，开始正常爬取，但最后返回错误，经查该词条最后四个链接无法爬取和显示，该如何处理？

问题比较多，麻烦大神不厌其烦耐心解答，不甚感谢！

lzb1001

suchocolate 发表于 2022-4-10 16:13
代码发出来。

代码已发，麻烦审核并解答下，谢谢大神。

suchocolate

1、下面两个URL好像都可以，为看哪什么会这样？主要应该里用哪个呢？
正常的URL一般格式
http://www.jianshu.com/search;user?q=python&page=1&type=collections#chapter1
:// 前面的就是 scheme，代表协议，上面是http
第3个/ 前的www.jianshu.com网站域名
第3个/后面是你要访问的资源路径
分号；后面是 params，代表参数，但实际当中用的少，而是用下面的query作为参数
问号？后面是查询条件 query，一般用作 GET 类型的 URL；格式是key=value，多个查询条件&分隔
井号 #后面是锚点，用于直接定位页面内部的下拉位置。

按照规范，（未编码前）你的url应该是 https://baike.baidu.com/search/word?word='猪八戒'，如果你[5:]了之后，就去掉了word=，导致url不正确，不确定百度能正确解析你的URL。

URL解析参考 3.1.3章节：https://github.com/Python3WebSpider/Python3WebSpider/blob/master/3.1-%E4%BD%BF%E7%94%A8urllib.md


2、用了列表分片[5:-1]后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或小于5比如[4:-1]或仅用[:-1]还会报错无法运行，这是为什么？
哪里分片[5:-1]，结果？

3、如果要加入伪装的header，上述headers用法是否正确？
header是http规范要求的，其中user-agent必须携带的字段，但规范没有要求必须用什么，只是表示请求人的信息。
如果我们不设置，默认是 python-urllib 或 python-requests，看你使用的库。
但目前实际情况，有的网站会检查user-agent，如果发现其中带python或者urllib等字样，就拒绝服务，返回418给你。所以目前使用都会填一个不包含python或urllib字样的信息，想逼真一点就用代码里的内容没有错。

4、输入关键词“黄埔军校”，刚开始正常抓取，快结束时会报如下错误，不知道什么原因
抓太频繁了，重启python


5.content = ''.join([each.text]) # 问题1：这里没看懂什么意思？为什么要用join()方法？另外text这个单词从哪里来的呢？
join的目的是让不连续的列表组成一个长字符串，为后续操作做准备。
each 对象是soup对象，text是soup对象自带属性，包含soup的实际文本。


6.content = ''.join([content, soup2.h2.text]) # 问题2：输出结果词条名（副标题），副标题的括号哪里产生的呢？
官网带的




建议你从基础系统学起，事半功倍，没有基础直接上手，浪费时间，学不扎实。

lzb1001

suchocolate 发表于 2022-4-16 09:46
1、下面两个URL好像都可以，为看哪什么会这样？主要应该里用哪个呢？
正常的URL一般格式
http://www.jian ...

感谢解答。

2、用了列表分片[5:-1]后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或小于5比如[4:-1]或仅用[:-1]还会报错无法运行，这是为什么？
哪里分片[5:-1]，结果？

我说错了，不是[5:-1]，是[10:-1]
整句代码for each in soup.find_all(href = re.compile('item'))[10:-1]:

suchocolate

lzb1001 发表于 2022-4-19 10:43
感谢解答。

2、用了列表分片[5:-1]后，仅过滤最后一条无用记录，开头几条无法过滤；若不用列表分片或 ...

把你遇到的报错发出来。

lzb1001

suchocolate 发表于 2022-4-20 08:39
把你遇到的报错发出来。

当把代码中的分片改为：

for each in soup.find_all(href = re.compile('item'))[4:-1]:或for each in soup.find_all(href = re.compile('item'))[:-1]:时，

运行代码：

请输入检索关键词：王新衡
word=%E7%8E%8B%E6%96%B0%E8%A1%A1
https://baike.baidu.com/search/w ... B%E6%96%B0%E8%A1%A1
https://baike.baidu.com/item/%E7%8E%8B%E6%96%B0%E8%A1%A1
{'User-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}
Traceback (most recent call last):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
    self.connect()
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 707, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\work\p14_92.py", line 48, in <module>
    main()
  File "D:\work\p14_92.py", line 36, in main
    response2 = urllib.request.urlopen(req2) # 将Request对象req2传给urlopen函数
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1347, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1321, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

lzb1001

4、输入关键词“黄埔军校”，刚开始正常抓取，快结束时会报如下错误，不知道什么原因
抓太频繁了，重启python

试了还是不行，该词条最后四个链接（如下）还是无法抓取：

黄埔军校        西点军校        桑赫斯特皇家军事学院        伏龙芝军事学院

可否请大神进入该词条看下以上四个链接？

附：返回错误提示如下
Traceback (most recent call last):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
    self.connect()
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 707, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\work\p14_92.py", line 48, in <module>
    main()
  File "D:\work\p14_92.py", line 36, in main
    response2 = urllib.request.urlopen(req2) # 将Request对象req2传给urlopen函数
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1347, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "C:\Users\dell\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1321, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>