刚学到爬虫XXOO，遇到403forbidden

nihaoz

其实是小甲鱼哥哥把顺序弄错了，因为网站有反爬虫  所以小甲鱼视频写的response = urllib.request.urlopen(url)，但是url是最初始的东西就是http://jandan.net/ooxx！所以小甲鱼写的视频上写的程序是没有加入user-agent的，所以姐姐办法就是response = urllib.request.urlopen(req)

nihaoz

1. req = urllib.request.Request(url)
   
2.     user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  
   
3.     #headers = { 'User-Agent' : user_agent }
   
4.     req.add_header('User-Agent', user_agent )
   
5.     response = urllib.request.urlopen(req)
   
6.     html = response.read().decode('utf-8')
   
7.     a = html.find('current-comment-page')+23
   
8.     b = html.find(']',a)
   
9.     print(html[a:b])

复制代码

贴上小甲鱼小改动的代码 原理在上一楼

worry921

同问楼主的问题

ebola11

能爬，煎蛋网不止只检查headers中的‘user-agent’,还检查headers中的'cookie’,如果老是一个‘cookie’爬，就会抛出503异常，而且爬图片的‘host’不是煎蛋的,是新浪的微相册，所以打开图片url时候，需要修改‘host’。另，如果抛出UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte异常，试着删除headers中'Accept-Encoding'的gzip。搞了一天了，才看了些遮遮掩掩的光屁股，还不如上毛站打瘾得多。

萌萌哒小鲨鱼

RUCJack 发表于 2016-3-22 15:27
10楼的方法试了一下，还是不行。使用代理服务器，亲测可行：
proxy_support = urllib.request.ProxyHandle ...

亲~ 咋整的？

400a6

21正解，小甲鱼上课的时候他将header加入req后，传给urlopen（）的却是没有加header的初始url，所以爬虫被拦截了。17/2/13

一只河蟹出墙来

今年5月初淘宝上买了小甲鱼老师的《零基础学习Python》(正版哟)，目前自学到了西瓜皮。
昨天看到OOXX的视频，把小甲鱼老师的代码改了下，加入了一些捕捉URL和HTTP报错的代码，自已找了个妹子网站，我是每抓取哪个地址，都会打印出来，但是像下面这种错误还是经常出现，这些地址确实不存，但网站会把它自动重定向到主页，大部分这种情况都是自动忽略的，但有时却会解决发下面这种报错。
自已搜索了下，说是因为浏览器在测试期间过早关闭造成的，但不知道怎么解决，
本来想自已发贴呢，但发现没权限，囧......
召唤@小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...小甲鱼老师...

Traceback (most recent call last):
  File "D:\python_xiaojiayu\Crawler\download_mm.py", line 70, in <module>
    download_mm()
  File "D:\python_xiaojiayu\Crawler\download_mm.py", line 66, in download_mm
    img_addrs = find_imgs(page_url)
  File "D:\python_xiaojiayu\Crawler\download_mm.py", line 15, in find_imgs
    html = url_open(url).decode('utf-8')
  File "D:\python_xiaojiayu\Crawler\download_mm.py", line 10, in url_open
    html = urllib.request.urlopen(req).read()
  File "D:\Python\lib\urllib\request.py", line 162, in urlopen
    return opener.open(url, data, timeout)
  File "D:\Python\lib\urllib\request.py", line 465, in open
    response = self._open(req, data)
  File "D:\Python\lib\urllib\request.py", line 483, in _open
    '_open', req)
  File "D:\Python\lib\urllib\request.py", line 443, in _call_chain
    result = func(*args)
  File "D:\Python\lib\urllib\request.py", line 1283, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "D:\Python\lib\urllib\request.py", line 1243, in do_open
    r = h.getresponse()
  File "D:\Python\lib\http\client.py", line 1174, in getresponse
    response.begin()
  File "D:\Python\lib\http\client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "D:\Python\lib\http\client.py", line 251, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response