很惭愧、写了一个小爬虫——小草社区~

罗同学

miladuo 发表于 2018-11-21 00:42
执行后，报这个错误。encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x0 ...

可以给个报错的截图嘛、谢谢。

miyazai

1

yanxi920

学习学习

miladuo

罗同学 发表于 2018-11-21 10:15
可以给个报错的截图嘛、谢谢。

现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
  File "/Users/abc/Library/Mobile Documents/com~apple~CloudDocs/Python/日常练习/爬虫练习/2.测试中.py", line 65, in <module>
    link.append(urlhead + each.h3.a.get('href'))
AttributeError: 'NoneType' object has no attribute 'a'

Process finished with exit code 1

没看明白这个表示什么意思？each.h3.a  

shierhan

素衣起风尘

看看

海风zZ

啥好东西我瞅瞅

miladuo

miladuo 发表于 2018-11-21 11:37
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
  Fi ...

/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/abc/Library/Mobile Documents/com~apple~CloudDocs/Python/日常练习/爬虫练习/2.测试中.py"
http://cl.kp9.xyz/read.php?tid=5877
http://cl.kp9.xyz/htm_data/16/1106/524942.html
http://cl.kp9.xyz/htm_data/16/1706/2424348.html
http://cl.kp9.xyz/htm_data/16/1808/344501.html
http://cl.kp9.xyz/htm_data/16/1110/622028.html
http://cl.kp9.xyz/htm_data/16/1707/2519480.html
http://cl.kp9.xyz/htm_data/16/0805/136474.html
http://cl.kp9.xyz/htm_data/16/1109/594741.html
http://cl.kp9.xyz/htm_data/16/1811/3334067.html
http://cl.kp9.xyz/htm_data/16/1811/3335464.html
http://cl.kp9.xyz/htm_data/16/1811/3334354.html
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x93 0x00 0x00 0x3A
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
Done

现在报错成这样了。还是encoding的错误。

罗同学

miladuo 发表于 2018-11-21 11:37
现在在公司运行，报的又不是encoding错误了。报错信息如下：
Traceback (most recent call last):
  Fi ...

each.h3.a  这个是提取标签、看下网页的代码可以发现页面中子网页的地址全都是在each.h3.a  标签下的。
至于那个encoding的报错、emmmm，我去发个帖子求助去。

罗同学

miladuo 发表于 2018-11-21 14:23
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/abc/ ...

emmm、今天休息又试了一下、好用的啊、没有报错额。
注意49、50行的地址前面现在是“http://cl.kp9.xyz”，把代码里面改一下

miladuo

罗同学 发表于 2018-11-24 12:36
emmm、今天休息又试了一下、好用的啊、没有报错额。
注意49、50行的地址前面现在是“http://cl.kp9.xyz ...

你运行我的看看。麻烦了。我修改了存储的路径和URL链接。


import requests
from bs4 import BeautifulSoup as bs
import re
import os
import socket
import time
import threading


def url_open(url):
    socket.setdefaulttimeout(20)
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'}
    res = requests.get(url, headers=headers)
    return res


def save(url):  # 传入每个子网页链接
    res = url_open(url)
    res.encoding = 'gbk'
    soup = bs(res.text, 'lxml')
    title = soup.find('title').text.split('-')[0]  # 标题
    # os.mkdir(title)
    # os.chdir(title)
    temp = soup.find_all('tr', class_='tr3')
    img = re.findall(r'data-src="(.*?jpg)" type', str(temp))


    imglist = []
    for each in img:
        imglist.append(each)
    for each in imglist:
        filename = each.split('/')[-1]
        img = url_open(each)
        print('saving...+%s' % filename)

        with open(title + filename, 'wb')as f:
            f.write(img.content)
            # os.chdir('..')


if __name__ == '__main__':
    os.makedirs(r'/Users/abc/spider/training/1024', exist_ok=True)
    os.chdir(r'/Users/abc/spider/training/1024')
    url = 'http://cl.kp9.xyz/thread0806.php?fid=16&search=&page=1'  # 默认爬取第一个页面，毕竟要注意身体，需要多个页面的话，自己加个for循环也不是什么难事~
    urlhead = 'http://cl.kp9.xyz/'  # 页面解析出来的连接前面需要加上这个头才能打开，根据多年经验这个头是会变的，如果哪天不能用了自己看下是不是这个头变了
    res = url_open(url)
    res.encoding = 'gbk'

    '''找到页面中的所有子网页'''
    soup = bs(res.text, 'lxml')
    temp = soup.find_all('td', class_="tal")
    link = []
    for each in temp:
        urls1 = urlhead + each.h3.a.get('href')
        link.append(urls1)
    # del link[0:10]

    downloads = []
    for each in link:
        down = threading.Thread(target=save, args=[each])
        downloads.append(down)
        down.start()

    for each in downloads:
        each.join()
    print('Done')

运行结果：
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x48
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x9C 0x00 0x00 0x29
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0xFE
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0x97 0x00 0x00 0x8B
encoding error : input conversion failed due to input error, bytes 0xFF 0x00 0x00 0x3C
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA6 0x85 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0xA9 0x5F 0x00 0x00
encoding error : input conversion failed due to input error, bytes 0x8F 0x00 0x00 0x20
Done

罗同学

miladuo 发表于 2018-11-26 10:21
你运行我的看看。麻烦了。我修改了存储的路径和URL链接。

emmm、修改一下url_open函数，如果还不行、我就也不会了

1. def url_open(url):
   
2.     socket.setdefaulttimeout(20)
   
3.     headers = {
   
4.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'}
   
5.     res = requests.get(url, headers=headers).decode('gbk', 'replace').encode('gbk')
   
6.     return res

复制代码

小强工作室

学习一下

sunnyrain

以学习的名义，哈哈

白蓝

学习

junpengAI

  学习啦

woduhaitao

1024

xlxl

脸红是怎么样的

fishc_py

队长别开枪是我

是什么没看懂啊