[已解决]爬虫

kygschp

有没有大哥知道怎么爬谷歌学术的数据？

import requests
import bs4

url = 'https://scholar.google.com.hk/citations?hl=zh-CN&user=lyQCGAEAAAAJ'

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'}

response = requests.get(url, headers=headers)


ValueError: check_hostname requires server_hostname

最佳答案

月排行榜 / 总排行榜

redforce

2021-10-31 19:39:47

1. import requests
   
2. import bs4
   
3. 
   
4. url = 'https://scholar.google.com.hk/citations?hl=zh-CN&user=lyQCGAEAAAAJ'
   
5. 
   
6. # 定义代理服务器地址，此地址需要自己去找，通过代理能访问https://scholar.google.com.hk即可
   
7. proxies = {'http': '127.0.0.1:8080', 'https': '127.0.0.1:8080'}
   
8. 
   
9. headers = {
   
10.     'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'}
    
11. 
    
12. response = requests.get(url, headers=headers, proxy=proxies)
    
13. print(response)
    

复制代码

跳转到最佳答案楼层

redforce

1. import requests
   
2. import bs4
   
3. 
   
4. url = 'https://scholar.google.com.hk/citations?hl=zh-CN&user=lyQCGAEAAAAJ'
   
5. 
   
6. # 定义代理服务器地址，此地址需要自己去找，通过代理能访问https://scholar.google.com.hk即可
   
7. proxies = {'http': '127.0.0.1:8080', 'https': '127.0.0.1:8080'}
   
8. 
   
9. headers = {
   
10.     'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'}
    
11. 
    
12. response = requests.get(url, headers=headers, proxy=proxies)
    
13. print(response)
    

复制代码