最近都沉了呢、自己动手捞一下
1
执行后，报这个错误。encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
我理解是encoding不对吧，但是查看网页的确是gbk呀。
miladuo 发表于 2018-11-21 00:42
执行后，报这个错误。encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x0 ...
可以给个报错的截图嘛、谢谢。
1
学习学习{:5_109:}
罗同学 发表于 2018-11-21 10:15
可以给个报错的截图嘛、谢谢。
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
File "/Users/abc/Library/Mobile Documents/com~apple~CloudDocs/Python/日常练习/爬虫练习/2.测试中.py", line 65, in <module>
link.append(urlhead + each.h3.a.get('href'))
AttributeError: 'NoneType' object has no attribute 'a'
Process finished with exit code 1
没看明白这个表示什么意思？each.h3.a
{:10_269:}
看看
啥好东西我瞅瞅
miladuo 发表于 2018-11-21 11:37
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
Fi ...
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/abc/Library/Mobile Documents/com~apple~CloudDocs/Python/日常练习/爬虫练习/2.测试中.py"
http://cl.kp9.xyz/read.php?tid=5877
http://cl.kp9.xyz/htm_data/16/1106/524942.html
http://cl.kp9.xyz/htm_data/16/1706/2424348.html
http://cl.kp9.xyz/htm_data/16/1808/344501.html
http://cl.kp9.xyz/htm_data/16/1110/622028.html
http://cl.kp9.xyz/htm_data/16/1707/2519480.html
http://cl.kp9.xyz/htm_data/16/0805/136474.html
http://cl.kp9.xyz/htm_data/16/1109/594741.html
http://cl.kp9.xyz/htm_data/16/1811/3334067.html
http://cl.kp9.xyz/htm_data/16/1811/3335464.html
http://cl.kp9.xyz/htm_data/16/1811/3334354.html
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
Done
现在报错成这样了。还是encoding的错误。
miladuo 发表于 2018-11-21 11:37
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
Fi ...
each.h3.a这个是提取标签、看下网页的代码可以发现页面中子网页的地址全都是在each.h3.a标签下的。
至于那个encoding的报错、emmmm，我去发个帖子求助去。
miladuo 发表于 2018-11-21 14:23
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/abc/ ...
emmm、今天休息又试了一下、好用的啊、没有报错额。
注意49、50行的地址前面现在是“http://cl.kp9.xyz”，把代码里面改一下
罗同学 发表于 2018-11-24 12:36
emmm、今天休息又试了一下、好用的啊、没有报错额。
注意49、50行的地址前面现在是“http://cl.kp9.xyz ...
你运行我的看看。麻烦了。我修改了存储的路径和URL链接。
import requests
from bs4 import BeautifulSoup as bs
import re
import os
import socket
import time
import threading
def url_open(url):
socket.setdefaulttimeout(20)
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'}
res = requests.get(url, headers=headers)
return res
def save(url):# 传入每个子网页链接
res = url_open(url)
res.encoding = 'gbk'
soup = bs(res.text, 'lxml')
title = soup.find('title').text.split('-')# 标题
# os.mkdir(title)
# os.chdir(title)
temp = soup.find_all('tr', class_='tr3')
img = re.findall(r'data-src="(.*?jpg)" type', str(temp))
imglist = []
for each in img:
imglist.append(each)
for each in imglist:
filename = each.split('/')[-1]
img = url_open(each)
print('saving...+%s' % filename)
with open(title + filename, 'wb')as f:
f.write(img.content)
# os.chdir('..')
if __name__ == '__main__':
os.makedirs(r'/Users/abc/spider/training/1024', exist_ok=True)
os.chdir(r'/Users/abc/spider/training/1024')
url = 'http://cl.kp9.xyz/thread0806.php?fid=16&search=&page=1'# 默认爬取第一个页面，毕竟要注意身体，需要多个页面的话，自己加个for循环也不是什么难事~
urlhead = 'http://cl.kp9.xyz/'# 页面解析出来的连接前面需要加上这个头才能打开，根据多年经验这个头是会变的，如果哪天不能用了自己看下是不是这个头变了
res = url_open(url)
res.encoding = 'gbk'
'''找到页面中的所有子网页'''
soup = bs(res.text, 'lxml')
temp = soup.find_all('td', class_="tal")
link = []
for each in temp:
urls1 = urlhead + each.h3.a.get('href')
link.append(urls1)
# del link
downloads = []
for each in link:
down = threading.Thread(target=save, args=)
downloads.append(down)
down.start()
for each in downloads:
each.join()
print('Done')
运行结果：
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
Done
miladuo 发表于 2018-11-26 10:21
你运行我的看看。麻烦了。我修改了存储的路径和URL链接。
emmm、修改一下url_open函数，如果还不行、我就也不会了{:10_243:}
def url_open(url):
socket.setdefaulttimeout(20)
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'}
res = requests.get(url, headers=headers).decode('gbk', 'replace').encode('gbk')
return res
学习一下
以学习的名义，哈哈
学习
{:5_109:}学习啦
1024