[已解决]爬虫有关的问题

RIXO

一个动态网页，怎么抓取想要的字符串啊
就比如说http://www.25xz.com/player/1.shtml
这个网址
我找到了他的mp3地址为
http://bama.25xz.com/小卓玛/故乡迭部.mp3

小卓玛和故乡迭部分别是他的专辑名字和歌曲名字

但是如果用urlopen返回的html是一个静态网址，没有相应的专辑名字和歌曲名字
怎么应该获取啊

最佳答案

月排行榜 / 总排行榜

wongyusing

2018-9-21 21:44:33

1. import requests     # requests是python的一个轻量级爬虫框架
   
2. from bs4 import BeautifulSoup as bs  # BeautifulSoup这个名字太长了，简写成bs
   
3. 
   
4. # 打开网页函数
   
5. def get_response(url):
   
6.     headers = {
   
7. 
   
8. 'Host': 'www.25xz.com',
   
9. 'Connection': 'keep-alive',
   
10. 'Accept': 'application/json, text/javascript, */*; q=0.01',
    
11. 'X-Requested-With': 'XMLHttpRequest',
    
12. 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
    
13. 'Referer': 'http://www.25xz.com/player/1.shtml',
    
14. 'Accept-Encoding': 'gzip, deflate',
    
15. 'Accept-Language': 'zh-CN,zh;q=0.9',
    
16. 'Cookie': 'ASP.NET_SessionId=qhgwada3kxhnrckccypdxgah',
    
17.         }
    
18.     response = requests.get(url, headers)
    
19.     response.encoding = 'utf-8'
    
20.     #response.encoding = 'gbk'
    
21.     return response
    
22. 
    
23. 
    
24. def main():
    
25.     base_url = 'http://www.25xz.com/ajax/musicList.shtml@1'
    
26.     response = get_response(base_url)
    
27.     print(response.text)
    
28.     #soup = bs(response.text,'lxml')
    
29.     #print(soup.select('.play_musicname'))
    
30. 
    
31. if __name__ == '__main__':
    
32.     main()

复制代码

还有事情忙着，有空再写注释

跳转到最佳答案楼层

wongyusing

1. import requests     # requests是python的一个轻量级爬虫框架
   
2. from bs4 import BeautifulSoup as bs  # BeautifulSoup这个名字太长了，简写成bs
   
3. 
   
4. # 打开网页函数
   
5. def get_response(url):
   
6.     headers = {
   
7. 
   
8. 'Host': 'www.25xz.com',
   
9. 'Connection': 'keep-alive',
   
10. 'Accept': 'application/json, text/javascript, */*; q=0.01',
    
11. 'X-Requested-With': 'XMLHttpRequest',
    
12. 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
    
13. 'Referer': 'http://www.25xz.com/player/1.shtml',
    
14. 'Accept-Encoding': 'gzip, deflate',
    
15. 'Accept-Language': 'zh-CN,zh;q=0.9',
    
16. 'Cookie': 'ASP.NET_SessionId=qhgwada3kxhnrckccypdxgah',
    
17.         }
    
18.     response = requests.get(url, headers)
    
19.     response.encoding = 'utf-8'
    
20.     #response.encoding = 'gbk'
    
21.     return response
    
22. 
    
23. 
    
24. def main():
    
25.     base_url = 'http://www.25xz.com/ajax/musicList.shtml@1'
    
26.     response = get_response(base_url)
    
27.     print(response.text)
    
28.     #soup = bs(response.text,'lxml')
    
29.     #print(soup.select('.play_musicname'))
    
30. 
    
31. if __name__ == '__main__':
    
32.     main()

复制代码

还有事情忙着，有空再写注释

幽梦三影

对这个URL发送请求