设为首页收藏本站
切换到窄版

鱼C论坛

 找回密码
 立即注册
鱼C论坛»论坛 › 编程语言专区 › Python交流 › Mac下python爬虫多次失败
返回列表 发新帖
查看: 2000|回复: 7

[已解决]Mac下python爬虫多次失败

[复制链接]
ten$1
发表于 2020-7-11 20:54:45 | 显示全部楼层 |阅读模式
10鱼币
本帖最后由 ten$1 于 2020-7-11 22:21 编辑

第一次实验：
  1. import urllib.request
  2. from bs4 import BeautifulSoup
  3. import os

  5. def Download(url,picAlt,name):
  6.     path = 'D:\\pythonD爬虫妹子图\\'+picAlt+'\\'
  7.     if not os.path.exists(path):
  8.         os.makedirs(path)
  9.     urllib.request.urlretrieve( url, '{0}{1}.jpg'.format(path, name))

  11. header = {
  12.     "User-Agent":'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36',
  13.     'Accept': '*/*',
  14.     'Accept-Language': 'en-US,en;q=0.8',
  15.     'Cache-Control': 'max-age=0',
  16.     'Connection': 'keep-alive'
  17.     }

  19. def run(targetUrl, beginNUM ,endNUM):
  20.     req = urllib.request.Request(url=targetUrl,headers=header)
  21.     response = urllib.request.urlopen(req)
  22.     html = response.read().decode('gb2312','ignore')
  23.     soup = BeautifulSoup(html, 'html.parser')
  24.     Divs = soup.find_all('div',attrs={'id':'big-pic' })
  25.     nowpage = soup.find('span',attrs={'class':'nowpage'}).get_text()
  26.     totalpage= soup.find('span',attrs={'class':'totalpage'}).get_text()
  27.     if beginNUM ==endNUM :
  28.         return
  29.     for div in Divs:
  30.         beginNUM = beginNUM+1

  32.         if div.find("a") is None :
  33.             print("没有下一张了")
  34.             return
  35.         elif div.find("a")['href'] is None or div.find("a")['href']=="":
  36.             print("没有下一张了None")
  37.             return
  38.         print("下载信息：总进度：",beginNUM,"/",endNUM," ，正在下载套图：(",nowpage,"/",totalpage,")")

  40.         if int(nowpage)<int(totalpage):
  41.             nextPageLink ="http://www.mmonly.cc/mmtp/qcmn/" +(div.find('a')['href'])
  42.         elif int(nowpage)==int(totalpage):
  43.             nextPageLink = (div.find('a')['href'])

  45.         picLink = (div.find('a').find('img')['src'])
  46.         picAlt = (div.find('a').find('img'))['alt']
  47.         print('下载的图片链接:',picLink)
  48.         print('套图名：[ ', picAlt , ' ] ')
  49.         print('开始下载...........')
  50.         Download(picLink,picAlt, nowpage)
  51.         print("下载成功！")
  52.         print('下一页链接:',nextPageLink)
  53.         run(nextPageLink,beginNUM ,endNUM)
  54.         return


  57. if __name__ == '__main__':
  58.     targetUrl ="http://www.mmonly.cc/mmtp/qcmn/237269.html"
  59.     run(targetUrl,beginNUM=0,endNUM=70)
  60.     print(" OVER")
  61.    
复制代码

结果
  1. Traceback (most recent call last):
  2.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1350, in do_open
  3.     h.request(req.get_method(), req.selector, req.data, headers,
  4.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1240, in request
  5.     self._send_request(method, url, body, headers, encode_chunked)
  6.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1286, in _send_request
  7.     self.endheaders(body, encode_chunked=encode_chunked)
  8.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1235, in endheaders
  9.     self._send_output(message_body, encode_chunked=encode_chunked)
  10.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1006, in _send_output
  11.     self.send(msg)
  12.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 946, in send
  13.     self.connect()
  14.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1409, in connect
  15.     self.sock = self._context.wrap_socket(self.sock,
  16.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
  17.     return self.sslsocket_class._create(
  18.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
  19.     self.do_handshake()
  20.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
  21.     self._sslobj.do_handshake()
  22. ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

  24. During handling of the above exception, another exception occurred:

  26. Traceback (most recent call last):
  27.   File "/Users/xiaojiayudeapple/Desktop/My Code/python/A.py", line 59, in <module>
  28.     run(targetUrl,beginNUM=0,endNUM=70)
  29.   File "/Users/xiaojiayudeapple/Desktop/My Code/python/A.py", line 21, in run
  30.     response = urllib.request.urlopen(req)
  31.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 222, in urlopen
  32.     return opener.open(url, data, timeout)
  33.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 531, in open
  34.     response = meth(req, response)
  35.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 640, in http_response
  36.     response = self.parent.error(
  37.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 563, in error
  38.     result = self._call_chain(*args)
  39.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
  40.     result = func(*args)
  41.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 755, in http_error_302
  42.     return self.parent.open(new, timeout=req.timeout)
  43.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 525, in open
  44.     response = self._open(req, data)
  45.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 542, in _open
  46.     result = self._call_chain(self.handle_open, protocol, protocol +
  47.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
  48.     result = func(*args)
  49.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1393, in https_open
  50.     return self.do_open(http.client.HTTPSConnection, req,
  51.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1353, in do_open
  52.     raise URLError(err)
  53. urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>
复制代码




                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

                               
登录/注册后可看大图

第二次：
  1. import urllib.request
  2. import os
  3. import re

  5. #打开url操作
  6. def url_open(url):
  7.         headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36',
  8.                    'Referer': 'http://wwww.mzitu.com'}
  9.         req = urllib.request.Request(url,headers = headers)
  10.         response = urllib.request.urlopen(req)
  11.         html = response.read()
  12.         return html

  14. #获取当前图片组的最大页码数
  15. def get_maxpage(url):
  16.         html = url_open(url).decode('utf-8')
  17.         pages = re.findall(r'<span>\d{1,2}',html)
  18.         return pages[-1][6:len(pages[-1])]

  20. #传入当前页面url，返回当前页面所有图片组链接地址列表
  21. def find_imgs(url):
  22.         html = url_open(url).decode('utf-8')
  23.         imgs_url = re.findall(r'http://www.mzitu.com/\d{6}',html)
  24.         return imgs_url

  26. #传入图片组url，返回图片组中所有图片链接地址列表
  27. def find_img(url,page):
  28.         html = url_open(url + '/' + str(page)).decode('utf-8')
  29.         img_addrs = []

  31.         a = html.find('img src=')
  32.         while a != -1:
  33.                 b = html.find('.jpg" alt="',a,a+255)
  34.                 if b!= -1:
  35.                         img_addrs.append(html[a+9:b+4])
  36.                 else:
  37.                         b =a + 9
  38.                 a = html.find('img src=',b)
  39.         return img_addrs[0]


  42. #根据图片地址列表，将图片保存到folder中
  43. def save_img(folder,img_addrs):
  44.         for each in img_addrs:
  45.                 filename = each.split('/')[-1]
  46.                 print(filename)
  47.                 with open(filename,'wb') as f:
  48.                         img = url_open(each)
  49.                         f.write(img)

  51. def download(folder = 'meizi',*pages):
  52.         if not os.path.exists(folder):
  53.                 os.mkdir(folder)
  54.         os.chdir(folder)

  56.         url = 'http://www.mzitu.com'
  57.         # page_num = int(get_page(url))        #获取当前页数

  59.         for page in pages:
  60.                 page_url = url + '/page/' + str(page) + '/'
  61.                 #创建页文件夹
  62.                 pagefolder = "page-" + str(page)
  63.                 if not os.path.exists(pagefolder):
  64.                         os.mkdir(pagefolder)
  65.                 os.chdir(pagefolder)
  66.                 #获取图片组地址列表
  67.                 img_group_addrs = find_imgs(page_url)
  68.                 #对于每个图片组，获取图片地址并保存
  69.                 group = 0
  70.                 for addr in img_group_addrs:
  71.                         group += 1
  72.                         img_addrs = [find_img(addr,x) for x in range(int(get_maxpage(addr)))]
  73.                         #创建组文件夹
  74.                         groupfolder = str(page) + "-" + str(group)
  75.                         if not os.path.exists(groupfolder):
  76.                                 os.mkdir(groupfolder)
  77.                         os.chdir(groupfolder)
  78.                         save_img(groupfolder,img_addrs)
  79.                         os.chdir(os.pardir)
  80.                 os.chdir(os.pardir)

  82. if __name__ == '__main__':
  83.         download('meizi',1)#第一个参数为文件夹名，第二个参数为要爬取的页码
复制代码

结果
  1. Traceback (most recent call last):
  2.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1350, in do_open
  3.     h.request(req.get_method(), req.selector, req.data, headers,
  4.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1240, in request
  5.     self._send_request(method, url, body, headers, encode_chunked)
  6.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1286, in _send_request
  7.     self.endheaders(body, encode_chunked=encode_chunked)
  8.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1235, in endheaders
  9.     self._send_output(message_body, encode_chunked=encode_chunked)
  10.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1006, in _send_output
  11.     self.send(msg)
  12.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 946, in send
  13.     self.connect()
  14.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1409, in connect
  15.     self.sock = self._context.wrap_socket(self.sock,
  16.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
  17.     return self.sslsocket_class._create(
  18.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
  19.     self.do_handshake()
  20.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
  21.     self._sslobj.do_handshake()
  22. ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

  24. During handling of the above exception, another exception occurred:

  26. Traceback (most recent call last):
  27.   File "/Users/xiaojiayudeapple/Desktop/My Code/python/A.py", line 83, in <module>
  28.     download('meizi',1)#第一个参数为文件夹名，第二个参数为要爬取的页码
  29.   File "/Users/xiaojiayudeapple/Desktop/My Code/python/A.py", line 67, in download
  30.     img_group_addrs = find_imgs(page_url)
  31.   File "/Users/xiaojiayudeapple/Desktop/My Code/python/A.py", line 22, in find_imgs
  32.     html = url_open(url).decode('utf-8')
  33.   File "/Users/xiaojiayudeapple/Desktop/My Code/python/A.py", line 10, in url_open
  34.     response = urllib.request.urlopen(req)
  35.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 222, in urlopen
  36.     return opener.open(url, data, timeout)
  37.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 531, in open
  38.     response = meth(req, response)
  39.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 640, in http_response
  40.     response = self.parent.error(
  41.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 563, in error
  42.     result = self._call_chain(*args)
  43.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
  44.     result = func(*args)
  45.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 755, in http_error_302
  46.     return self.parent.open(new, timeout=req.timeout)
  47.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 525, in open
  48.     response = self._open(req, data)
  49.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 542, in _open
  50.     result = self._call_chain(self.handle_open, protocol, protocol +
  51.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
  52.     result = func(*args)
  53.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1393, in https_open
  54.     return self.do_open(http.client.HTTPSConnection, req,
  55.   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1353, in do_open
  56.     raise URLError(err)
  57. urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>
  58. >>>
复制代码
最佳答案
月排行榜 / 总排行榜
Twilight6
2020-7-11 20:54:46
ten$1 发表于 2020-7-11 22:23
就是 原厂封装 的IDE啊。。证书错误怎么解决？


安装了不会报没有模块 bs4 的错误才对呀，你是不是电脑不止一个版本的 Python 呢？ 安装的时候看见 Successfully 才算是成功的哈

证书错误加上这串代码试试看，取消证书验证：
  1. import ssl
  2. ssl._create_default_https_context = ssl._create_unverified_context
复制代码

最佳答案

Twilight6

查看完整内容

安装了不会报没有模块 bs4 的错误才对呀，你是不是电脑不止一个版本的 Python 呢？ 安装的时候看见 Successfully 才算是成功的哈 证书错误加上这串代码试试看，取消证书验证：
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

Twilight6
发表于 2020-7-11 20:54:46 | 显示全部楼层    本楼为最佳答案   
ten$1 发表于 2020-7-11 22:23
就是 原厂封装 的IDE啊。。证书错误怎么解决？


安装了不会报没有模块 bs4 的错误才对呀，你是不是电脑不止一个版本的 Python 呢？ 安装的时候看见 Successfully 才算是成功的哈

证书错误加上这串代码试试看，取消证书验证：
  1. import ssl
  2. ssl._create_default_https_context = ssl._create_unverified_context
复制代码
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

ten$1
 楼主| 发表于 2020-7-11 20:55:45 | 显示全部楼层
这是怎么回事啊？
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

Twilight6
发表于 2020-7-11 21:08:40 | 显示全部楼层
第一个报错是你没安装 BeautifulSoup 模块吧？

第二个报错是证书错误吧？SSL

评分

参与人数 1鱼币 +2 收起 理由
ten$1 + 2

查看全部评分

小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

ten$1
 楼主| 发表于 2020-7-11 22:20:54 | 显示全部楼层
Twilight6 发表于 2020-7-11 21:08
第一个报错是你没安装 BeautifulSoup 模块吧？

第二个报错是证书错误吧？SSL

BeautifulSoup装好了
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

Twilight6
发表于 2020-7-11 22:22:09 | 显示全部楼层
ten$1 发表于 2020-7-11 22:20
BeautifulSoup装好了

你用的 是 PyCharm ?
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

ten$1
 楼主| 发表于 2020-7-11 22:23:23 | 显示全部楼层
Twilight6 发表于 2020-7-11 22:22
你用的 是 PyCharm ?

就是 原厂封装 的IDE啊。。证书错误怎么解决？
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

ten$1
 楼主| 发表于 2020-7-11 22:28:26 | 显示全部楼层
Twilight6 发表于 2020-7-11 22:26
安装了不会报没有模块 bs4 的错误才对呀，你是不是电脑不止一个版本的 Python 呢？ 安装的时候看见 Suc ...

好了，谢谢谢谢
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

返回列表 发新帖
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

小黑屋|手机版|Archiver|鱼C工作室 ( 粤ICP备18085999号-1 | 粤公网安备 44051102000585号)

GMT+8, 2025-6-23 09:56

Powered by Discuz! X3.4

© 2001-2023 Discuz! Team.

快速回复 返回顶部 返回列表