[已解决]当BeautifulSoup解析完网页后，发现所需的url不在里边

937135952

当BeautifulSoup解析完网页后，发现所需的url不在里边，应该是被隐藏了，应该怎么办，求教各位大佬。如下代码所示，想findall（'a'）来查出所有url，可是返回值为空

1. 
   
2. liebiao=[]
   
3.     headers = {
   
4.     'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
   
5.     'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   
6.     'Accept-Language':'en-US,en;q=0.5',
   
7.     'Accept-Encoding':'gzip',
   
8.     'DNT':'1',
   
9.     'Connection':'close'
   
10.     }
    
11.     page = requests.get("https://search.bilibili.com/all?keyword=%E7%8C%AB%E5%92%AA%E6%97%A5%E5%B8%B8%E7%94%A8%E5%93%81&from_source=webtop_search&spm_id_from=333.1007&search_source=5", headers=headers)
    
12.     print(page)
    
13.    
    
14.     soup_obj=BeautifulSoup(page.content,'html.parser')
    
15.     print(soup_obj)
    
16. 
    
17. 
    
18.     for link in soup_obj.findAll('a'):#含a
    
19.         if "href" in link.attrs:#且以href作为特征/如果link的特征项里有href.
    
20.             a=link.attrs['href']
    
21.             if 'www.bilibili.com/video' in a:
    
22.                 #print(a)
    
23.                 #以下存储url到列表里
    
24.                 liebiao.append("https:"+a)
    

复制代码

最佳答案

月排行榜 / 总排行榜

suchocolate

2022-10-26 00:13:31

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. import re
   
4. 
   
5. 
   
6. def main():
   
7.     result = []
   
8.     headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
   
9.     url = "https://search.bilibili.com/all?keyword=%E7%8C%AB%E5%92%AA%E6%97%A5%E5%B8%B8%E7%94%A8%E5%93%81&from_source=" \
   
10.           "webtop_search&spm_id_from=333.1007&search_source=5"
    
11.     r = requests.get(url, headers=headers)
    
12.     r.encoding = 'utf-8'
    
13.     soup = BeautifulSoup(r.text, 'html.parser')
    
14.     for item in soup.find_all('a', attrs={'href': re.compile('bili.com/video')}):
    
15.         result.append(f"https:{item['href']}")
    
16.     print(result)
    
17. 
    
18. 
    
19. if __name__ == "__main__":
    
20.     main()
    
21. 
    

复制代码

跳转到最佳答案楼层

xiaosi4081

1.打开网页
2.摁下f12
3.找到netword->xhr
4.在一堆json文件里找到av号或url

jmy_286501

想爬哔哩哔哩，不用这么麻烦吧

jmy_286501

缩进也有问题

jmy_286501

page.content要解码吧，page.content.decode('utf8')

937135952

jmy_286501 发表于 2022-10-25 14:58
page.content要解码吧，page.content.decode('utf8')

没有用

suchocolate

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. import re
   
4. 
   
5. 
   
6. def main():
   
7.     result = []
   
8.     headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
   
9.     url = "https://search.bilibili.com/all?keyword=%E7%8C%AB%E5%92%AA%E6%97%A5%E5%B8%B8%E7%94%A8%E5%93%81&from_source=" \
   
10.           "webtop_search&spm_id_from=333.1007&search_source=5"
    
11.     r = requests.get(url, headers=headers)
    
12.     r.encoding = 'utf-8'
    
13.     soup = BeautifulSoup(r.text, 'html.parser')
    
14.     for item in soup.find_all('a', attrs={'href': re.compile('bili.com/video')}):
    
15.         result.append(f"https:{item['href']}")
    
16.     print(result)
    
17. 
    
18. 
    
19. if __name__ == "__main__":
    
20.     main()
    
21. 
    

复制代码