[已解决]python爬取诗词网乱码

717986230

感谢大佬，不知道哪里与问题了。。请求帮助

代码如下：
url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
page_text = requests.get(url=url,headers=headers).text
fp = open('./sanguo.txt','w',encoding='utf-8')
soup = BeautifulSoup(page_text,'lxml')
a_list = soup.select('.book-mulu > ul > li > a')
for a in a_list:
    title = a.string
    detail_url = 'https://www.shicimingju.com'+a['href']
    page_text_detail = requests.get(url=detail_url,headers=headers).text
    soup = BeautifulSoup(page_text_detail,'lxml')
    div_tag = soup.find('div',class_="chapter_content")
    content = div_tag.text
    fp.write(title+':'+content+'\n')
    print(title,'保存成功！！！')
fp.close()

最佳答案

月排行榜 / 总排行榜

大马强

2021-7-18 19:58:45

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
   
5. headers = {
   
6.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
   
7. # 以下为修改的代码 该网页是用utf-8 来编码，所以要进行编码处理，这是第一处
   
8. req = requests.get(url=url, headers=headers)
   
9. req.encoding = ("utf-8")
   
10. page_text = req.text
    
11. 
    
12. fp = open('./sanguo.txt', 'w', encoding='utf-8')
    
13. soup = BeautifulSoup(page_text, 'lxml')
    
14. a_list = soup.select('.book-mulu > ul > li > a')
    
15. for a in a_list:
    
16.     title = a.string
    
17.     detail_url = 'https://www.shicimingju.com'+a['href']
    
18.     # 这是第二处
    
19.     req2 = requests.get(url=detail_url, headers=headers)
    
20.     req2.encoding = ("utf-8")
    
21.     page_text_detail = req2.text
    
22.     soup = BeautifulSoup(page_text_detail, 'lxml')
    
23.     div_tag = soup.find('div', class_="chapter_content")
    
24.     content = div_tag.text
    
25.     fp.write(title+':'+content+'\n')
    
26.     print(title, '保存成功！！！')
    
27. fp.close()
    

复制代码

乱码一般都是你的编码有问题，参考这篇文章
https://blog.csdn.net/wyb199026/ ... 1018.2226.3001.4187

跳转到最佳答案楼层

717986230

717986230

乱码

大马强

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
   
5. headers = {
   
6.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
   
7. # 以下为修改的代码 该网页是用utf-8 来编码，所以要进行编码处理，这是第一处
   
8. req = requests.get(url=url, headers=headers)
   
9. req.encoding = ("utf-8")
   
10. page_text = req.text
    
11. 
    
12. fp = open('./sanguo.txt', 'w', encoding='utf-8')
    
13. soup = BeautifulSoup(page_text, 'lxml')
    
14. a_list = soup.select('.book-mulu > ul > li > a')
    
15. for a in a_list:
    
16.     title = a.string
    
17.     detail_url = 'https://www.shicimingju.com'+a['href']
    
18.     # 这是第二处
    
19.     req2 = requests.get(url=detail_url, headers=headers)
    
20.     req2.encoding = ("utf-8")
    
21.     page_text_detail = req2.text
    
22.     soup = BeautifulSoup(page_text_detail, 'lxml')
    
23.     div_tag = soup.find('div', class_="chapter_content")
    
24.     content = div_tag.text
    
25.     fp.write(title+':'+content+'\n')
    
26.     print(title, '保存成功！！！')
    
27. fp.close()
    

复制代码

乱码一般都是你的编码有问题，参考这篇文章
https://blog.csdn.net/wyb199026/ ... 1018.2226.3001.4187

南归

1. headers = {
   
2.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
   
3. }
   
4. page_text = requests.get(url=url, headers=headers, verify=False)
   
5. page_text.encoding = page_text.apparent_encoding
   
6. page_text = page_text.text #以上两行代码为添加的

复制代码

717986230

大马强 发表于 2021-7-18 19:58
乱码一般都是你的编码有问题，参考这篇文章
https://blog.csdn.net/wyb199026/article/details/52562538 ...

感谢大佬

717986230

南归 发表于 2021-7-18 20:02

感谢大佬