[已解决]urllib 获取html问题

wangwang123

import urllib.request

url = "https://www.baidu.com/s?ie=utf-8&medium=0&rtt=4&bsst=1&rsv_dl=news_t_sk&cl=2&wd=阿里巴巴"

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                        'Chrome/95.0.4638.54 Safari/537.36'}

request = urllib.request.Request(url, headers=header)

response = urllib.request.urlopen(request)

bs = BeautifulSoup(response, "html.parser")

问题： urllib函数不能识别中文？ 只能通过requests模块获取吗，是不是可以在Request那里增加一个参数就可以识别了呢？有点疑惑。

谢谢大佬帮忙解答！

最佳答案

月排行榜 / 总排行榜

suchocolate

2021-10-31 19:30:08

1. from urllib.parse import quote
   
2. keyword = '壁纸'
   
3. url = 'https://www.baidu.com/s?wd=' + quote(keyword)
   
4. print(url)

复制代码

跳转到最佳答案楼层

hrpzcf

1. https://blog.csdn.net/mouday/article/details/80278938

复制代码

wangwang123

hrpzcf 发表于 2021-10-31 16:34

from urllib request import quote,unquote 没有这个库啊，在setting添加也没有

hrpzcf

from urllib.request import quote,unquote

wangwang123

hrpzcf 发表于 2021-10-31 17:32
from urllib.request import quote,unquote

Cannot find reference 'quote' in 'request.pyi' 一样的 有这个问题

suchocolate

1. from urllib.parse import quote
   
2. keyword = '壁纸'
   
3. url = 'https://www.baidu.com/s?wd=' + quote(keyword)
   
4. print(url)

复制代码

hrpzcf

1. import urllib.request
   
2. # from urllib.request import quote
   
3. from urllib.parse import quote # 上面一句不行就试试这个，我用两个都正常工作
   
4. 
   
5. from bs4 import BeautifulSoup
   
6. 
   
7. cn = quote("阿里巴巴")
   
8. url = (
   
9.     "https://www.baidu.com/s?ie=utf-8&medium=0&rtt=4&bsst=1&rsv_dl=news_t_sk&cl=2&wd=%s"
   
10.     % cn
    
11. )
    
12. 
    
13. header = {
    
14.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
    
15.     "Chrome/95.0.4638.54 Safari/537.36"
    
16. }
    
17. 
    
18. request = urllib.request.Request(url, headers=header)
    
19. 
    
20. response = urllib.request.urlopen(request)
    
21. 
    
22. # bs = BeautifulSoup(response, "html.parser")
    
23. 
    
24. print(response.read().decode("utf-8"))
    

复制代码

redforce

from urllib.parse import quote

redforce

1. import urllib.request
   
2. from urllib.parse import quote
   
3. from bs4 import BeautifulSoup
   
4. 
   
5. wd = quote('阿里巴巴')
   
6. 
   
7. url = f"https://www.baidu.com/s?ie=utf-8&medium=0&rtt=4&bsst=1&rsv_dl=news_t_sk&cl=2&wd={wd}"
   
8. 
   
9. header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
   
10.                         'Chrome/95.0.4638.54 Safari/537.36'}
    
11. 
    
12. req = urllib.request.Request(url, headers=header)
    
13. 
    
14. response = urllib.request.urlopen(req)
    
15. 
    
16. bs = BeautifulSoup(response, "html.parser")
    
17. print(bs)

复制代码

wangwang123

hrpzcf 发表于 2021-10-31 19:33

谢谢！