[已解决]关于爬取斗图啦网站的表情包的速度问题和异常问题(可能被封IP了)

风尘岁月

异常:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1331, in getresponse
    response.begin()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\util\retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1331, in getresponse
    response.begin()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:/python_fruit/表情包/表情包_robot.py", line 31, in <module>
    img_data = requests.get(url = img_url, headers = headers).content
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

最佳答案

月排行榜 / 总排行榜

Twilight6

2020-5-30 17:13:57

Twilight6 发表于 2020-5-30 17:09
连接中断，这种错误，建议多复制几个 UA头放到 headers 里去，然后用 choice 随机挑选，还有挂个代理IP

或者用 fake_useragent 模块
这个模块自带各种浏览器的 UA 头使用的时候直接

from fake_useragent import UserAgent

# 随机 谷歌 UA
headers = {
'User-Agent':UserAgent().chrome
}

# 随机 IE UA
headers = {
'User-Agent':UserAgent().ie
}

跳转到最佳答案楼层

xiaosi4081

挂 v pn

Twilight6

requests.exceptions.ConnectionError

连接中断，这种错误，建议多复制几个 UA头放到 headers 里去，然后用 choice 随机挑选，还有挂个代理IP

Twilight6

Twilight6 发表于 2020-5-30 17:09
连接中断，这种错误，建议多复制几个 UA头放到 headers 里去，然后用 choice 随机挑选，还有挂个代理IP

或者用 fake_useragent 模块
这个模块自带各种浏览器的 UA 头使用的时候直接

from fake_useragent import UserAgent

# 随机 谷歌 UA
headers = {
'User-Agent':UserAgent().chrome
}

# 随机 IE UA
headers = {
'User-Agent':UserAgent().ie
}

风尘岁月

谢谢 能用了 换了个代理（背锅）IP