如何爬取百度百科的内容
我刚才试了，结果直接返回的链接失效，把请求头全部都加上了，还是不行，大佬们，百度加入了什么可怕的反爬虫逻辑，我的代码如下：import bs4
import requests
header = {
'Host':'image.baidu.com',
'Connection':'keep-alive',
'Cache-Control':'max-age=0',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0;win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/siqned-exchange;v=b3;q=0.9'}
def request(url, encode):
r = requests.get(url)
r.encoding = encode
print(r.text)
soup = bs4.BeautifulSoup(r.text, 'html.parser')
#print(soup)
soup.find_all('')
request('https://baike.baidu.com/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB', 'utf-8') 请求头别乱加，删掉host
https://dd-static.jd.com/ddimg/jfs/t1/212632/7/9683/154010/61cffeb0E9a070cd1/82c9ee223967bf52.png
页:
[1]