|
3鱼币
在做53讲作业时,最后一题复制了小甲鱼的代码,运行了一次遇到418错误,之后添加user-agent还是会报418错误,求大佬指点
- import urllib.request
- import chardet
- head = {'User - Agent': 'Mozilla / 5.0(WindowsNT10.0;Win64;x64) AppleWebKit / 537.36(KHTML, likeGecko) Chrome / '
- '80.0.3987.116Safari / 537.36'}
- def main():
- i = 0
- with open("urls.txt", "r") as f:
- # 读取待访问的网址
- # 由于urls.txt每一行一个URL
- # 所以按换行符'\n'分割
- urls = f.read().splitlines()
- for each_url in urls:
- rep = urllib.request.Request(each_url, headers=head)
- response = urllib.request.urlopen(rep)
- html = response.read()
- # 识别网页编码
- encode = chardet.detect(html)['encoding']
- if encode == 'GB2312':
- encode = 'GBK'
- i += 1
- filename = "url_%d.txt" % i
- with open(filename, "w", encoding=encode) as each_file:
- each_file.write(html.decode(encode, "ignore"))
- if __name__ == "__main__":
- main()
复制代码
把UA头里面两个空格去了
- head = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'}
复制代码
|
|