717986230 发表于 2021-7-18 19:25:07

python爬取诗词网乱码

感谢大佬，不知道哪里与问题了。。请求帮助{:5_102:} {:10_266:} {:10_266:} {:10_266:}

代码如下：
url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
page_text = requests.get(url=url,headers=headers).text
fp = open('./sanguo.txt','w',encoding='utf-8')
soup = BeautifulSoup(page_text,'lxml')
a_list = soup.select('.book-mulu > ul > li > a')
for a in a_list:
    title = a.string
    detail_url = 'https://www.shicimingju.com'+a['href']
    page_text_detail = requests.get(url=detail_url,headers=headers).text
    soup = BeautifulSoup(page_text_detail,'lxml')
    div_tag = soup.find('div',class_="chapter_content")
    content = div_tag.text
    fp.write(title+':'+content+'\n')
    print(title,'保存成功！！！')
fp.close()

717986230 发表于 2021-7-18 19:28:40

{:10_266:}{:10_266:}{:10_266:}{:10_266:}{:10_266:}

717986230 发表于 2021-7-18 19:33:01

乱码{:10_266:}

大马强 发表于 2021-7-18 19:58:45

import requests
from bs4 import BeautifulSoup

url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
# 以下为修改的代码 该网页是用utf-8 来编码，所以要进行编码处理，这是第一处
req = requests.get(url=url, headers=headers)
req.encoding = ("utf-8")
page_text = req.text

fp = open('./sanguo.txt', 'w', encoding='utf-8')
soup = BeautifulSoup(page_text, 'lxml')
a_list = soup.select('.book-mulu > ul > li > a')
for a in a_list:
    title = a.string
    detail_url = 'https://www.shicimingju.com'+a['href']
    # 这是第二处
    req2 = requests.get(url=detail_url, headers=headers)
    req2.encoding = ("utf-8")
    page_text_detail = req2.text
    soup = BeautifulSoup(page_text_detail, 'lxml')
    div_tag = soup.find('div', class_="chapter_content")
    content = div_tag.text
    fp.write(title+':'+content+'\n')
    print(title, '保存成功！！！')
fp.close()

乱码一般都是你的编码有问题，参考这篇文章
https://blog.csdn.net/wyb199026/article/details/52562538?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522162660896416780262529495%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=162660896416780262529495&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_v2~rank_v29-2-52562538.first_rank_v2_pc_rank_v29&utm_term=%E5%AF%B9%E7%88%AC%E5%8F%96%E5%88%B0%E7%9A%84%E7%BD%91%E9%A1%B5%E8%BF%9B%E8%A1%8C%E7%BC%96%E7%A0%81&spm=1018.2226.3001.4187

南归 发表于 2021-7-18 20:02:00

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}
page_text = requests.get(url=url, headers=headers, verify=False)
page_text.encoding = page_text.apparent_encoding
page_text = page_text.text #以上两行代码为添加的

717986230 发表于 2021-7-18 20:57:54

大马强 发表于 2021-7-18 19:58
乱码一般都是你的编码有问题，参考这篇文章
https://blog.csdn.net/wyb199026/article/details/52562538 ...

感谢大佬{:10_266:}

717986230 发表于 2021-7-18 20:59:50

南归 发表于 2021-7-18 20:02


感谢大佬{:10_298:}
页: [1]
查看完整版本: python爬取诗词网乱码