[已解决]用requests.get打开的时候报错

hostmi

我用  res = requests.get("http://jobs.51job.com/all/co456898.html")

打开的时候出错了，换其他网址又不会报错，是这个网站有反爬机制吗

1. Traceback (most recent call last):
   
2.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
   
3.     chunked=chunked)
   
4.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
   
5.     six.raise_from(e, None)
   
6.   File "<string>", line 2, in raise_from
   
7.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
   
8.     httplib_response = conn.getresponse()
   
9.   File "E:\soft\Python36\lib\http\client.py", line 1331, in getresponse
   
10.     response.begin()
    
11.   File "E:\soft\Python36\lib\http\client.py", line 297, in begin
    
12.     version, status, reason = self._read_status()
    
13.   File "E:\soft\Python36\lib\http\client.py", line 258, in _read_status
    
14.     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    
15.   File "E:\soft\Python36\lib\socket.py", line 586, in readinto
    
16.     return self._sock.recv_into(b)
    
17. ConnectionResetError: [WinError 10054] 远程主机强迫关闭了一个现有的连接。
    
18. 
    
19. During handling of the above exception, another exception occurred:
    
20. 
    
21. Traceback (most recent call last):
    
22.   File "E:\soft\Python36\lib\site-packages\requests\adapters.py", line 440, in send
    
23.     timeout=timeout
    
24.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
    
25.     _stacktrace=sys.exc_info()[2])
    
26.   File "E:\soft\Python36\lib\site-packages\urllib3\util\retry.py", line 357, in increment
    
27.     raise six.reraise(type(error), error, _stacktrace)
    
28.   File "E:\soft\Python36\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    
29.     raise value.with_traceback(tb)
    
30.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    
31.     chunked=chunked)
    
32.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
    
33.     six.raise_from(e, None)
    
34.   File "<string>", line 2, in raise_from
    
35.   File "E:\soft\Python36\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
    
36.     httplib_response = conn.getresponse()
    
37.   File "E:\soft\Python36\lib\http\client.py", line 1331, in getresponse
    
38.     response.begin()
    
39.   File "E:\soft\Python36\lib\http\client.py", line 297, in begin
    
40.     version, status, reason = self._read_status()
    
41.   File "E:\soft\Python36\lib\http\client.py", line 258, in _read_status
    
42.     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    
43.   File "E:\soft\Python36\lib\socket.py", line 586, in readinto
    
44.     return self._sock.recv_into(b)
    
45. urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))
    
46. 
    
47. During handling of the above exception, another exception occurred:
    
48. 
    
49. Traceback (most recent call last):
    
50.   File "<pyshell#2>", line 1, in <module>
    
51.     res = requests.get("http://jobs.51job.com/all/co456898.html")
    
52.   File "E:\soft\Python36\lib\site-packages\requests\api.py", line 72, in get
    
53.     return request('get', url, params=params, **kwargs)
    
54.   File "E:\soft\Python36\lib\site-packages\requests\api.py", line 58, in request
    
55.     return session.request(method=method, url=url, **kwargs)
    
56.   File "E:\soft\Python36\lib\site-packages\requests\sessions.py", line 508, in request
    
57.     resp = self.send(prep, **send_kwargs)
    
58.   File "E:\soft\Python36\lib\site-packages\requests\sessions.py", line 618, in send
    
59.     r = adapter.send(request, **kwargs)
    
60.   File "E:\soft\Python36\lib\site-packages\requests\adapters.py", line 490, in send
    
61.     raise ConnectionError(err, request=request)
    
62. requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))

复制代码

最佳答案

月排行榜 / 总排行榜

Teagle

2017-11-4 17:40:59

1. headers = {
   
2.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0'
   
3.     }

复制代码

response = requests.get(url,headers=headers)

你爬取时默认的爬虫头是python-requests/x.xx.x
诸如此类，服务器发现携带这类headers的数据包的时候，会直接拒绝访问

解决方法就是，直接伪造User-Agent信息

'User-Agent':'BaiduSipder'便是百度爬虫的header信息

跳转到最佳答案楼层

新手·ing

是的，这样只能伪装一下你的爬虫了，使用代理IP或者伪装成浏览器吧。

BngThea

反爬越来越普遍了

Teagle

1. headers = {
   
2.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0'
   
3.     }

复制代码

response = requests.get(url,headers=headers)

你爬取时默认的爬虫头是python-requests/x.xx.x
诸如此类，服务器发现携带这类headers的数据包的时候，会直接拒绝访问

解决方法就是，直接伪造User-Agent信息

'User-Agent':'BaiduSipder'便是百度爬虫的header信息

hostmi

Teagle 发表于 2017-11-4 17:40
response = requests.get(url,headers=headers)

你爬取时默认的爬虫头是python-requests/x.xx.x

嗯，谢谢