[已解决]爬取网页源码与审查元素不一致如何解决？（cookies已加）

丶霁灵

我想爬取音乐排行榜的歌曲名称，

1. import requests
   
2. 
   
3. def get_url(url):
   
4.     headers = {'user_agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
   
5.     'referer':'https://music.163.com'}
   
6.     cookies = dict(cookies_are=' _ntes_nnid=f5291a60e8322f06589d07f84941da7b,1580733444127; _ntes_nuid=f5291a60e8322f06589d07f84941da7b; P_I********820c03.1.0.1584157473499.0.1584157480981; login_target=163')
   
7.     res = requests.get(url,headers = headers,cookies=cookies)
   
8.     html = res.text
   
9.     return(html)
   
10. def main(url):
    
11.     html = get_url(url)
    
12.     with open("5.txt",'w',encoding='utf-8') as f:
    
13.         f.write(html)
    
14. if __name__ == '__main__':
    
15.     url = 'https://music.163.com/discover/toplist'
    
16.     main(url)

复制代码

cookies已加，但是得到的网页源代码还是只有200多行

并且内容明确是在GET包里面，查资料说是F12看到是源码被js加载后的结果，
请问该怎么办呀~~~给一个自学的方向也可以呀

最佳答案

月排行榜 / 总排行榜

叙白

2020-4-1 22:06:27

这样

1. import requests
   
2. from lxml import etree
   
3. url = 'https://music.163.com/discover/toplist'
   
4. headers = {
   
5.     'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
   
6. }
   
7. response = requests.get(url,headers = headers)
   
8. text = response.text
   
9. 
   
10. html = etree.HTML(text)
    
11. songname = html.xpath('//ul[@class="f-hide"]//a/text()')
    
12. address = html.xpath('//ul[@class="f-hide"]//a/@href')
    
13. print(songname)
    
14. print(address)

复制代码

跳转到最佳答案楼层

丶霁灵

救命呀

叙白

这样

1. import requests
   
2. from lxml import etree
   
3. url = 'https://music.163.com/discover/toplist'
   
4. headers = {
   
5.     'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
   
6. }
   
7. response = requests.get(url,headers = headers)
   
8. text = response.text
   
9. 
   
10. html = etree.HTML(text)
    
11. songname = html.xpath('//ul[@class="f-hide"]//a/text()')
    
12. address = html.xpath('//ul[@class="f-hide"]//a/@href')
    
13. print(songname)
    
14. print(address)

复制代码

丶霁灵

叙白 发表于 2020-4-1 22:06
这样

我试了你的代码能爬出来，但是我的代码就是爬不出来