很惭愧、写了一个小爬虫——小草社区~

罗同学 · 发表于 2018-11-21 10:15:44

miladuo 发表于 2018-11-21 00:42
执行后，报这个错误。encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x0 ...

可以给个报错的截图嘛、谢谢。

miyazai · 发表于 2018-11-21 11:08:35

yanxi920 · 发表于 2018-11-21 11:19:06

学习学习

miladuo · 发表于 2018-11-21 11:37:04

罗同学发表于 2018-11-21 10:15
可以给个报错的截图嘛、谢谢。

现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
File "/Users/abc/Library/Mobile Documents/com~apple~CloudDocs/Python/日常练习/爬虫练习/2.测试中.py", line 65, in <module>
link.append(urlhead + each.h3.a.get('href'))
AttributeError: 'NoneType' object has no attribute 'a'

Process finished with exit code 1

没看明白这个表示什么意思？each.h3.a

shierhan · 发表于 2018-11-21 11:57:34

素衣起风尘 · 发表于 2018-11-21 12:08:35

看看

海风zZ · 发表于 2018-11-21 14:05:03

啥好东西我瞅瞅

miladuo · 发表于 2018-11-21 14:23:01

miladuo 发表于 2018-11-21 11:37
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
Fi ...

/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/abc/Library/Mobile Documents/com~apple~CloudDocs/Python/日常练习/爬虫练习/2.测试中.py"
http://cl.kp9.xyz/read.php?tid=5877
http://cl.kp9.xyz/htm_data/16/1106/524942.html
http://cl.kp9.xyz/htm_data/16/1706/2424348.html
http://cl.kp9.xyz/htm_data/16/1808/344501.html
http://cl.kp9.xyz/htm_data/16/1110/622028.html
http://cl.kp9.xyz/htm_data/16/1707/2519480.html
http://cl.kp9.xyz/htm_data/16/0805/136474.html
http://cl.kp9.xyz/htm_data/16/1109/594741.html
http://cl.kp9.xyz/htm_data/16/1811/3334067.html
http://cl.kp9.xyz/htm_data/16/1811/3335464.html
http://cl.kp9.xyz/htm_data/16/1811/3334354.html
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
Done

现在报错成这样了。还是encoding的错误。

罗同学 · 发表于 2018-11-21 16:36:21

miladuo 发表于 2018-11-21 11:37
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
Fi ...

each.h3.a 这个是提取标签、看下网页的代码可以发现页面中子网页的地址全都是在each.h3.a 标签下的。
至于那个encoding的报错、emmmm，我去发个帖子求助去。

罗同学 · 发表于 2018-11-24 12:36:13

miladuo 发表于 2018-11-21 14:23
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/abc/ ...

emmm、今天休息又试了一下、好用的啊、没有报错额。
注意49、50行的地址前面现在是“http://cl.kp9.xyz”，把代码里面改一下
QQ图片20181124123306.png

miladuo · 发表于 2018-11-26 10:21:47

罗同学发表于 2018-11-24 12:36
emmm、今天休息又试了一下、好用的啊、没有报错额。
注意49、50行的地址前面现在是“http://cl.kp9.xyz ...

你运行我的看看。麻烦了。我修改了存储的路径和URL链接。

import requests
from bs4 import BeautifulSoup as bs
import re
import os
import socket
import time
import threading

def url_open(url):
socket.setdefaulttimeout(20)
headers = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'}
res = requests.get(url, headers=headers)
return res

def save(url):  # 传入每个子网页链接
res = url_open(url)
res.encoding = 'gbk'
soup = bs(res.text, 'lxml')
title = soup.find('title').text.split('-')[0]  # 标题
# os.mkdir(title)
# os.chdir(title)
temp = soup.find_all('tr', class_='tr3')
img = re.findall(r'data-src="(.*?jpg)" type', str(temp))

imglist = []
for each in img:
      imglist.append(each)
for each in imglist:
      filename = each.split('/')[-1]
      img = url_open(each)
      print('saving...+%s' % filename)

      with open(title + filename, 'wb')as f:
         f.write(img.content)
         # os.chdir('..')

if __name__ == '__main__':
os.makedirs(r'/Users/abc/spider/training/1024', exist_ok=True)
os.chdir(r'/Users/abc/spider/training/1024')
url = 'http://cl.kp9.xyz/thread0806.php?fid=16&search=&page=1'  # 默认爬取第一个页面，毕竟要注意身体，需要多个页面的话，自己加个for循环也不是什么难事~
urlhead = 'http://cl.kp9.xyz/'  # 页面解析出来的连接前面需要加上这个头才能打开，根据多年经验这个头是会变的，如果哪天不能用了自己看下是不是这个头变了
res = url_open(url)
res.encoding = 'gbk'

'''找到页面中的所有子网页'''
soup = bs(res.text, 'lxml')
temp = soup.find_all('td', class_="tal")
link = []
for each in temp:
      urls1 = urlhead + each.h3.a.get('href')
      link.append(urls1)
# del link[0:10]

downloads = []
for each in link:
      down = threading.Thread(target=save, args=[each])
      downloads.append(down)
      down.start()

for each in downloads:
      each.join()
print('Done')

运行结果：
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
Done

罗同学 · 发表于 2018-11-26 10:59:43

miladuo 发表于 2018-11-26 10:21
你运行我的看看。麻烦了。我修改了存储的路径和URL链接。

emmm、修改一下url_open函数，如果还不行、我就也不会了

def url_open(url):
    socket.setdefaulttimeout(20)
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'}
    res = requests.get(url, headers=headers).decode('gbk', 'replace').encode('gbk')
    return res

小强工作室 · 发表于 2018-11-26 11:25:21

学习一下

sunnyrain · 发表于 2018-11-26 13:34:08

以学习的名义，哈哈

白蓝 · 发表于 2018-11-26 15:40:51

学习

junpengAI · 发表于 2018-11-26 15:43:35

学习啦

woduhaitao · 发表于 2018-11-26 16:08:44

1024

xlxl · 发表于 2018-11-26 17:47:34

脸红是怎么样的

fishc_py · 发表于 2018-11-26 20:51:45

队长别开枪是我 · 发表于 2018-11-27 00:01:14

是什么没看懂啊

账号		自动登录	找回密码
密码			立即注册

[作品展示] 很惭愧、写了一个小爬虫——小草社区~