关于爬虫超时的问题。
如题，在进行爬虫爬取的时候（单线程），一直提示requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))
期间，在request请求中，添加了timeout=25没有解决。
网上搜索，说是要引入importsocket，然后加上socket.setdefaulttimeout(25)，设置超时时间。也没有解决。
单次循环暂停time.sleep(25)，暂停25秒也未解决。
再后来，说是要保持窗口的cookies， 使用了 request.Session()也依然没有解决。
暂时不知道怎么搞了。
求解决
Traceback (most recent call last):
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
conn.connect()
File "D:\Python38\lib\site-packages\urllib3\connection.py", line 362, in connect
self.sock = ssl_wrap_socket(
File "D:\Python38\lib\site-packages\urllib3\util\ssl_.py", line 386, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "D:\Python38\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "D:\Python38\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "D:\Python38\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: 远程主机强迫关闭了一个现有的连接。
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Python38\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "D:\Python38\lib\site-packages\urllib3\util\retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "D:\Python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "D:\Python38\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
conn.connect()
File "D:\Python38\lib\site-packages\urllib3\connection.py", line 362, in connect
self.sock = ssl_wrap_socket(
File "D:\Python38\lib\site-packages\urllib3\util\ssl_.py", line 386, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "D:\Python38\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "D:\Python38\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "D:\Python38\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:/MyPythonFiles/PT/mteam.py", line 21, in <module>
res=requests.get(url,headers=headers,timeout=25)
File "D:\Python38\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "D:\Python38\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "D:\Python38\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "D:\Python38\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "D:\Python38\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))
Process finished with exit code 1
提供一下源代码 这是遭遇反爬
页:
[1]