[已解决]爬虫——网站响应502求助

非凡

目标链接
https://cattendee.abstractsonlin ... &searchId=70400


页面上的信息是动态加载的，数据包的url,我也抓到了
https://www.abstractsonline.com/ ... ort=1&order=asc
但是当我用python发送请求的时候，总是502报错，请问下大佬这问题怎么解决

import requests

url = r'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
#修改heeader信息模拟浏览器访问
headers = {'UserAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39'}
res = requests.get(url, headers=headers, timeout=3)
res.encoding = 'utf-8'

最佳答案

月排行榜 / 总排行榜

YunGuo

2022-5-13 16:44:39

import requests

url = 'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
headers = {
    'Accept': 'application/json',
    'Backpack': '7d701d87-aeb0-4f7b-8b60-c98e784a5030',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36',
} 
response = requests.get(url, headers=headers)
print(response.json())

跳转到最佳答案楼层

YunGuo

import requests

url = 'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
headers = {
    'Accept': 'application/json',
    'Backpack': '7d701d87-aeb0-4f7b-8b60-c98e784a5030',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36',
} 
response = requests.get(url, headers=headers)
print(response.json())

wp231957

想抓取哪部分数据

1q23w31

没添加headers必须字段，例如防盗链
至于那些是必须的，你得自己试

非凡

YunGuo 发表于 2022-5-13 16:44

鱼从来没有让我失望过，拜谢大佬

非凡

1q23w31 发表于 2022-5-13 20:36
没添加headers必须字段，例如防盗链
至于那些是必须的，你得自己试

感谢您的解答。谢谢谢谢谢谢谢！！！！！1