[已解决]关于爬虫的问题

哈哈哈ha1

这是代码，我想要爬取我文件中的几个网站的源代码
import urllib.request
import chardet
def main():
    i = 0
    with open('C:\\Users\\Lenovo\\Desktop\\urls.txt','r')  as f:
        # 读取待访问的网址
        # 由于urls.txt每一行一个URL
        # 所以按换行符'\n'分割
        urls = f.read().splitlines()
    for each_url in urls:
        responrs = urllib.request.urlopen(urls)
        html = responrs.read()
        # 识别网页编码
        encode = chardet.detect(html)['encoding']
        if encode == 'GB2312':
            encode = 'GBK'
            i += 1
            filename = "url_%d" % i
            with open(filename,'w',encoding=encode):
                each_file.write(html.decode(encode,'ingnore'))
    if __name__ == '__main__':
        main()


main()


这是报错内容
Traceback (most recent call last):
  File "C:/Users/Lenovo/PycharmProjects/pythonProject8/venv/Scripts/爬虫课后作业.py", line 27, in <module>
    main()
  File "C:/Users/Lenovo/PycharmProjects/pythonProject8/venv/Scripts/爬虫课后作业.py", line 13, in main
    responrs = urllib.request.urlopen(urls)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 515, in open
    req.timeout = timeout
AttributeError: 'list' object has no attribute 'timeout'

求大佬帮忙指出问题所在，并且能否给一个更改si'k'l

最佳答案

月排行榜 / 总排行榜

suchocolate

2021-11-1 15:23:59

哈哈哈ha1 发表于 2021-11-1 14:55
我改了之后又报错了
Traceback (most recent call last):
  File "C:/Users/Lenovo/PycharmProjects/py ...

要改header，防止反扒。

headers = {'User-Agent': 'Mozilla'}
for each_url in urls:
    req = urllib.request.Request(each_url, headers=headers)
    responrs = urllib.request.urlopen(req)
    html = responrs.read()

跳转到最佳答案楼层

sh1t灬

responrs = urllib.request.urlopen(urls)
这句话有问题，urls是个待爬取列表，你在for循环里面应该使用each_url

哈哈哈ha1

sh1t灬 发表于 2021-10-31 12:18
responrs = urllib.request.urlopen(urls)
这句话有问题，urls是个待爬取列表，你在for循环里面应该使用ea ...

我改了之后又报错了
Traceback (most recent call last):
  File "C:/Users/Lenovo/PycharmProjects/pythonProject8/venv/Scripts/爬虫课后作业.py", line 27, in <module>
    main()
  File "C:/Users/Lenovo/PycharmProjects/pythonProject8/venv/Scripts/爬虫课后作业.py", line 13, in main
    responrs = urllib.request.urlopen(each_url)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 640, in http_response
    response = self.parent.error(
  File "C:\Program Files\Python38\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 502, in _call_chain
    result = func(*args)
  File "C:\Program Files\Python38\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 418:

suchocolate

哈哈哈ha1 发表于 2021-11-1 14:55
我改了之后又报错了
Traceback (most recent call last):
  File "C:/Users/Lenovo/PycharmProjects/py ...

要改header，防止反扒。

headers = {'User-Agent': 'Mozilla'}
for each_url in urls:
    req = urllib.request.Request(each_url, headers=headers)
    responrs = urllib.request.urlopen(req)
    html = responrs.read()