非凡 发表于 2022-5-13 16:44:38

爬虫——网站响应502求助

目标链接
https://cattendee.abstractsonline.com/meeting/10517/search?query=%40AllPosters%7EYes&type=presentation&view=appendToCards&initialSearchId=70400&searchId=70400


页面上的信息是动态加载的，数据包的url,我也抓到了
https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc
但是当我用python发送请求的时候，总是502报错，请问下大佬这问题怎么解决


import requests

url = r'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
#修改heeader信息模拟浏览器访问
headers = {'UserAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39'}
res = requests.get(url, headers=headers, timeout=3)
res.encoding = 'utf-8'

YunGuo 发表于 2022-5-13 16:44:39

import requests

url = 'https://www.abstractsonline.com/oe3/Program/10517/Search/70400/Results?page=1&pagesize=25&sort=1&order=asc'
headers = {
    'Accept': 'application/json',
    'Backpack': '7d701d87-aeb0-4f7b-8b60-c98e784a5030',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36',
}
response = requests.get(url, headers=headers)
print(response.json())

wp231957 发表于 2022-5-13 19:08:00

想抓取哪部分数据

1q23w31 发表于 2022-5-13 20:36:31

没添加headers必须字段，例如防盗链
至于那些是必须的，你得自己试

非凡 发表于 2022-5-13 22:58:01

YunGuo 发表于 2022-5-13 16:44


鱼从来没有让我失望过，拜谢大佬{:9_227:}

非凡 发表于 2022-5-13 22:58:49

1q23w31 发表于 2022-5-13 20:36
没添加headers必须字段，例如防盗链
至于那些是必须的，你得自己试

感谢您的解答。谢谢谢谢谢谢谢！！！！！1
页: [1]
查看完整版本: 爬虫——网站响应502求助