[已解决]爬虫——网站响应502求助

非凡

目标链接
https://cattendee.abstractsonlin ... &searchId=70400


页面上的信息是动态加载的，数据包的url,我也抓到了
https://www.abstractsonline.com/ ... ort=1&order=asc
但是当我用python发送请求的时候，总是502报错，请问下大佬这问题怎么解决

1. 
   
2. import requests
   
3. 
   
4. url = r'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
   
5. #修改heeader信息模拟浏览器访问
   
6. headers = {'UserAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39'}
   
7. res = requests.get(url, headers=headers, timeout=3)
   
8. res.encoding = 'utf-8'
   

复制代码

最佳答案

月排行榜 / 总排行榜

YunGuo

2022-5-13 16:44:39

1. import requests
   
2. 
   
3. url = 'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
   
4. headers = {
   
5.     'Accept': 'application/json',
   
6.     'Backpack': '7d701d87-aeb0-4f7b-8b60-c98e784a5030',
   
7.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36',
   
8. }
   
9. response = requests.get(url, headers=headers)
   
10. print(response.json())

复制代码

跳转到最佳答案楼层

YunGuo

1. import requests
   
2. 
   
3. url = 'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
   
4. headers = {
   
5.     'Accept': 'application/json',
   
6.     'Backpack': '7d701d87-aeb0-4f7b-8b60-c98e784a5030',
   
7.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36',
   
8. }
   
9. response = requests.get(url, headers=headers)
   
10. print(response.json())

复制代码

wp231957

想抓取哪部分数据

1q23w31

没添加headers必须字段，例如防盗链
至于那些是必须的，你得自己试

非凡

YunGuo 发表于 2022-5-13 16:44

鱼从来没有让我失望过，拜谢大佬

非凡

1q23w31 发表于 2022-5-13 20:36
没添加headers必须字段，例如防盗链
至于那些是必须的，你得自己试

感谢您的解答。谢谢谢谢谢谢谢！！！！！1