[已解决]爬虫求助

fc5igm

目标是爬取下图中的网页信息


网址链接：https://www.csindex.com.cn/#/indices/family/list?index_series=5

为此，写了如下代码

import urllib
import json

headers={'Accept': r'application/json, text/plain, */*', 'User-Agent': r'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.78'}
url=r'https://www.csindex.com.cn/csindex-home/index-list/query-index-item'
data='{"sorter":{"sortField":"null","sortOrder":null},"pager":{"pageNum":1,"pageSize":10},"indexFilter":{"ifCustomized":null,"ifTracked":null,"ifWeightCapped":null,"indexCompliance":null,"hotSpot":null,"indexClassify":["16","18"],"currency":null,"region":["china_mainland"],"indexSeries":["6","1","2","3"],"undefined":null}}'
data = urllib.parse.urlencode(json.loads(data)).encode('utf-8')
opener = urllib.request.build_opener()
request = urllib.request.Request(url, headers=headers,data=data)
response = opener.open(request,timeout=10).read().decode('utf-8', 'ignore')

然后获得的反馈是
'{"code":"500","msg":"服务器异常,请联系管理员","data":null,"success":false}'

尝试在headers中加入cookie也毫无变化。请问这种情况下应该怎么办？

最佳答案

月排行榜 / 总排行榜

isdkz

2023-2-6 17:16:15

我试了一下，这个接口并没有反爬机制，所以 UA、Referer、Cookie 都不需要

data你直接原始数据用字符串编码就好，然后 headers 通过 Content-Type 告诉他 data 里面的是什么类型

import urllib.request
import json

headers={'Content-Type': 'application/json;charset=UTF-8'}
url=r'https://www.csindex.com.cn/csindex-home/index-list/query-index-item'
data='{"sorter":{"sortField":"null","sortOrder":null},"pager":{"pageNum":1,"pageSize":10},"indexFilter":{"ifCustomized":null,"ifTracked":null,"ifWeightCapped":null,"indexCompliance":null,"hotSpot":null,"indexClassify":null,"currency":null,"region":null,"indexSeries":["5"],"undefined":null}}'
data = data.encode('utf-8')
opener = urllib.request.build_opener()
request = urllib.request.Request(url, headers=headers,data=data)
response = opener.open(request,timeout=10).read().decode('utf-8', 'ignore')
print(response)

跳转到最佳答案楼层

isdkz

我试了一下，这个接口并没有反爬机制，所以 UA、Referer、Cookie 都不需要

data你直接原始数据用字符串编码就好，然后 headers 通过 Content-Type 告诉他 data 里面的是什么类型

import urllib.request
import json

headers={'Content-Type': 'application/json;charset=UTF-8'}
url=r'https://www.csindex.com.cn/csindex-home/index-list/query-index-item'
data='{"sorter":{"sortField":"null","sortOrder":null},"pager":{"pageNum":1,"pageSize":10},"indexFilter":{"ifCustomized":null,"ifTracked":null,"ifWeightCapped":null,"indexCompliance":null,"hotSpot":null,"indexClassify":null,"currency":null,"region":null,"indexSeries":["5"],"undefined":null}}'
data = data.encode('utf-8')
opener = urllib.request.build_opener()
request = urllib.request.Request(url, headers=headers,data=data)
response = opener.open(request,timeout=10).read().decode('utf-8', 'ignore')
print(response)

fc5igm

isdkz 发表于 2023-2-6 17:46
我试了一下，这个接口并没有反爬机制，所以 UA、Referer、Cookie 都不需要

data你直接原始数据用字符串 ...

我试了一下，问题好像出在了

data = urllib.parse.urlencode(json.loads(data)).encode('utf-8')

这行代码上。

你用的

data = data.encode('utf-8')

返回的数据是

b'{"sorter":{"sortField":"null","sortOrder":null},"pager":{"pageNum":1,"pageSize":10},"indexFilter":{"ifCustomized":null,"ifTracked":null,"ifWeightCapped":null,"indexCompliance":null,"hotSpot":null,"indexClassify":["16","18"],"currency":null,"region":["china_mainland"],"indexSeries":["6","1","2","3"],"undefined":null}}'

而如果用我原文写的那个，返回的则是

b'sorter=%7B%27sortField%27%3A+%27null%27%2C+%27sortOrder%27%3A+None%7D&pager=%7B%27pageNum%27%3A+1%2C+%27pageSize%27%3A+10%7D&indexFilter=%7B%27ifCustomized%27%3A+None%2C+%27ifTracked%27%3A+None%2C+%27ifWeightCapped%27%3A+None%2C+%27indexCompliance%27%3A+None%2C+%27hotSpot%27%3A+None%2C+%27indexClassify%27%3A+%5B%2716%27%2C+%2718%27%5D%2C+%27currency%27%3A+None%2C+%27region%27%3A+%5B%27china_mainland%27%5D%2C+%27indexSeries%27%3A+%5B%276%27%2C+%271%27%2C+%272%27%2C+%273%27%5D%2C+%27undefined%27%3A+None%7D'

可是我看网上都是用的

data = urllib.parse.urlencode(data).encode('utf-8')

这样的代码。为什么在这里用反而会错误了呢？
什么时候应该用

data = data.encode('utf-8')

又什么时候应该用parse呢？

isdkz

fc5igm 发表于 2023-2-6 18:01
我试了一下，问题好像出在了

这行代码上。

首先，用 urllib 这个库是基于字节来操作的，所以不管怎样都要编码成字节码

然后就是数据的类型了，urllib.parse.urlencode 是将一个字典变成一个 url 参数的格式（a=1&b=2这种格式)，数据类型也就是 application/x-www-form-urlencoded

而通过开发者工具看到的post数据的原始格式（raw）就是一个类似于字典的数据，也就是 json，而 json 数据对应的 Content-Type 就是 application/json，因为 Content-Type 的默认值不是这个，所以要自己在 headers 中指出，不然也是会出错的

总而言之，在开发者工具中看 payload 的时候看 raw 格式，raw 是怎样的就要保持不变，然后通过 Content-Type 告诉服务器传的 payload 的格式

fc5igm

isdkz 发表于 2023-2-6 18:13
首先，用 urllib 这个库是基于字节来操作的，所以不管怎样都要编码成字节码

然后就是数据的类型了， ...

十分感谢。

LYWX

isdkz 发表于 2023-2-6 17:16
我试了一下，这个接口并没有反爬机制，所以 UA、Referer、Cookie 都不需要

data你直接原始数据用字符串 ...

6