爬取网页源码与审查元素不一致如何解决？（cookies已加）
我想爬取音乐排行榜的歌曲名称，import requests
def get_url(url):
headers = {'user_agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
'referer':'https://music.163.com'}
cookies = dict(cookies_are=' _ntes_nnid=f5291a60e8322f06589d07f84941da7b,1580733444127; _ntes_nuid=f5291a60e8322f06589d07f84941da7b; P_I********820c03.1.0.1584157473499.0.1584157480981; login_target=163')
res = requests.get(url,headers = headers,cookies=cookies)
html = res.text
return(html)
def main(url):
html = get_url(url)
with open("5.txt",'w',encoding='utf-8') as f:
f.write(html)
if __name__ == '__main__':
url = 'https://music.163.com/discover/toplist'
main(url)
cookies已加，但是得到的网页源代码还是只有200多行
并且内容明确是在GET包里面，查资料说是F12看到是源码被js加载后的结果，
请问该怎么办呀~~~给一个自学的方向也可以呀 救命呀 这样import requests
from lxml import etree
url = 'https://music.163.com/discover/toplist'
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
}
response = requests.get(url,headers = headers)
text = response.text
html = etree.HTML(text)
songname = html.xpath('//ul[@class="f-hide"]//a/text()')
address = html.xpath('//ul[@class="f-hide"]//a/@href')
print(songname)
print(address) 叙白 发表于 2020-4-1 22:06
这样
我试了你的代码能爬出来，但是我的代码就是爬不出来
页:
[1]