[已解决]关于爬虫的urllib.error.HTTPError: HTTP Error 418: 报错

fc5igm

在做爬虫第一课的练习时，题目要求爬豆瓣的首页内容
结果遇到了报错

1. Traceback (most recent call last):
   
2.   File "D:/python/works/test.py", line 11, in <module>
   
3.     html=urr.urlopen(ret).read()
   
4.   File "D:\python\lib\urllib\request.py", line 214, in urlopen
   
5.     return opener.open(url, data, timeout)
   
6.   File "D:\python\lib\urllib\request.py", line 523, in open
   
7.     response = meth(req, response)
   
8.   File "D:\python\lib\urllib\request.py", line 632, in http_response
   
9.     response = self.parent.error(
   
10.   File "D:\python\lib\urllib\request.py", line 561, in error
    
11.     return self._call_chain(*args)
    
12.   File "D:\python\lib\urllib\request.py", line 494, in _call_chain
    
13.     result = func(*args)
    
14.   File "D:\python\lib\urllib\request.py", line 641, in http_error_default
    
15.     raise HTTPError(req.full_url, code, msg, hdrs, fp)
    
16. urllib.error.HTTPError: HTTP Error 418:

复制代码

查了下网上，有人说加个headers里的User-Agent就可以模拟成浏览器访问
我加了之后却依然还是报错
之后我用了人家帖子里提供的User-Agent，成功爬虫。
然而问题是，我用我的浏览器访问豆瓣实际上是没有问题的。但是为什么爬虫时用我浏览器的User-Agent就不行呢？
是我写错了吗？

网上帖子提供的

1. headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}

复制代码

我自己写的

1. headers={'User=Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Safari/537.36 Edg/91.0.864.48'}

复制代码

程序全代码：

1. import urllib.request as urr
   
2. import chardet as char
   
3. with open('urls.txt') as urls:
   
4.     num=0
   
5.     for urllink in urls.readlines():
   
6.         print(urllink)
   
7.         num+=1
   
8.         #headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
   
9.         headers={'User=Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Safari/537.36 Edg/91.0.864.48'}
   
10.         ret=urr.Request(urllink,headers=headers)
    
11.         html=urr.urlopen(ret).read()
    
12.         urlcode=char.detect(html)['encoding']
    
13.         if urlcode=='GB2312':
    
14.             urlcode='GBK'
    
15.         html=html.decode(urlcode,'ignore')
    
16.         with open(f'url_{num}.txt','w',encoding=urlcode) as file:
    
17.             file.write(html)
    
18. 
    

复制代码

最佳答案

月排行榜 / 总排行榜

fish_游鱼

2021-6-18 23:06:54

...

跳转到最佳答案楼层

fish_游鱼

...

fc5igm

fish_游鱼 发表于 2021-6-18 23:06
...

谢谢

BanCan笑

headers=之后的缩进也有坑，之前没注意过啊，留意下