[已解决]爬取网页源码与审查元素不一致如何解决？（cookies已加）

丶霁灵

我想爬取音乐排行榜的歌曲名称，

import requests

def get_url(url):
    headers = {'user_agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
    'referer':'https://music.163.com'}
    cookies = dict(cookies_are=' _ntes_nnid=f5291a60e8322f06589d07f84941da7b,1580733444127; _ntes_nuid=f5291a60e8322f06589d07f84941da7b; P_I********820c03.1.0.1584157473499.0.1584157480981; login_target=163')
    res = requests.get(url,headers = headers,cookies=cookies)
    html = res.text
    return(html)
def main(url):
    html = get_url(url)
    with open("5.txt",'w',encoding='utf-8') as f:
        f.write(html)
if __name__ == '__main__':
    url = 'https://music.163.com/discover/toplist'
    main(url)

cookies已加，但是得到的网页源代码还是只有200多行

并且内容明确是在GET包里面，查资料说是F12看到是源码被js加载后的结果，
请问该怎么办呀~~~给一个自学的方向也可以呀

最佳答案

月排行榜 / 总排行榜

叙白

2020-4-1 22:06:27

这样

import requests
from lxml import etree
url = 'https://music.163.com/discover/toplist'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
}
response = requests.get(url,headers = headers)
text = response.text

html = etree.HTML(text)
songname = html.xpath('//ul[@class="f-hide"]//a/text()')
address = html.xpath('//ul[@class="f-hide"]//a/@href')
print(songname)
print(address)

跳转到最佳答案楼层

丶霁灵

救命呀

叙白

这样

import requests
from lxml import etree
url = 'https://music.163.com/discover/toplist'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
}
response = requests.get(url,headers = headers)
text = response.text

html = etree.HTML(text)
songname = html.xpath('//ul[@class="f-hide"]//a/text()')
address = html.xpath('//ul[@class="f-hide"]//a/@href')
print(songname)
print(address)

丶霁灵

叙白 发表于 2020-4-1 22:06
这样

我试了你的代码能爬出来，但是我的代码就是爬不出来