[已解决]这个总统数据库怎么爬？

故里

https://www.worldpresidentsdb.com/Yang-Shangkun/
https://www.worldpresidentsdb.com/list/gender/female/

最佳答案

月排行榜 / 总排行榜

suchocolate

2021-10-13 18:22:16

1. import requests
   
2. from lxml import etree
   
3. 
   
4. 
   
5. def main():
   
6.     base_url = 'https://www.worldpresidentsdb.com'
   
7.     url = 'https://www.worldpresidentsdb.com/list/gender/female/'
   
8.     headers = {'user-agent': 'firefox'}
   
9.     r = requests.get(url, headers=headers)
   
10.     html = etree.HTML(r.text)
    
11.     psts = html.xpath('//div[@class="list-group"]//@href')
    
12.     for pst in psts:
    
13.         url = f'{base_url}{pst}'
    
14.         r = requests.get(url, headers=headers)
    
15.         html = etree.HTML(r.text)
    
16.         info = html.xpath('//div[contains(@class,"col-md-8")]/p//text()')
    
17.         print(info)
    
18. 
    
19. 
    
20. if __name__ == '__main__':
    
21.     main()

复制代码

跳转到最佳答案楼层

suchocolate

1. import requests
   
2. from lxml import etree
   
3. 
   
4. 
   
5. def main():
   
6.     base_url = 'https://www.worldpresidentsdb.com'
   
7.     url = 'https://www.worldpresidentsdb.com/list/gender/female/'
   
8.     headers = {'user-agent': 'firefox'}
   
9.     r = requests.get(url, headers=headers)
   
10.     html = etree.HTML(r.text)
    
11.     psts = html.xpath('//div[@class="list-group"]//@href')
    
12.     for pst in psts:
    
13.         url = f'{base_url}{pst}'
    
14.         r = requests.get(url, headers=headers)
    
15.         html = etree.HTML(r.text)
    
16.         info = html.xpath('//div[contains(@class,"col-md-8")]/p//text()')
    
17.         print(info)
    
18. 
    
19. 
    
20. if __name__ == '__main__':
    
21.     main()

复制代码