第055讲：论一只爬虫的自我修养3：隐藏
本帖最后由 Twilight6 于 2020-11-17 12:23 编辑import urllib.request
import re
from bs4 import BeautifulSoup
def main():
url = "http://baike.baidu.com/view/284853.htm"
response = urllib.request.urlopen(url)
html = response.read()
soup = BeautifulSoup(html, "html.parser") # 使用 Python 默认的解析器
for each in soup.find_all(href=re.compile("view")):
print(each.text, "->", ''.join(["http://baike.baidu.com", each["href"]]))
# 上边用 join() 不用 + 直接拼接，是因为 join() 被证明执行效率要高很多
if __name__ == "__main__":
main()
代码时小甲鱼的源代码，但是编译时候出现了以下问题：
File "D:\python\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "D:\python\lib\urllib\request.py", line 525, in open
response = self._open(req, data)
File "D:\python\lib\urllib\request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "D:\python\lib\urllib\request.py", line 502, in _call_chain
result = func(*args)
File "D:\python\lib\urllib\request.py", line 1379, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "D:\python\lib\urllib\request.py", line 1350, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "D:\python\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "D:\python\lib\http\client.py", line 1266, in _send_request
self.putrequest(method, url, **skips)
File "D:\python\lib\http\client.py", line 1104, in putrequest
self._output(self._encode_request(request))
File "D:\python\lib\http\client.py", line 1184, in _encode_request
return request.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 36-39: ordinal not in range(128)
python3.x的默认编码是‘utf-8’,这里没问题，查了很久没有解决，希望大神帮帮忙！！
发求助帖子时候，不要设置主题售价哈
主题售价是别人支出给你的
这里获取到的源码有部分数据没办法被解码，参考下面的帖子应该能正常的运行：
[已解决] 055课 爬百度百科“网络爬虫”的词条 问题
https://fishc.com.cn/thread-169631-1-1.html
(出处: 鱼C论坛)
Twilight6 发表于 2020-11-17 12:24
发求助帖子时候，不要设置主题售价哈
主题售价是别人支出给你的
啊？我以为是我给别人的，谢谢大佬！！解决了！
页:
[1]