新年快乐，小白求助，怎样爬取”掌上高考“网页中的院校信息？

wingplay

先祝大家新年快乐财源滚滚日进斗金
python新手小白求教，需要爬取的是”掌上高考 “网站中“汉语言文学”专业的所有开设院校。试着用soup.find_all("div",class_="school-tab_schoolName__uLwTK")得出的是空的结果。分析网页元素发现，里面嵌套了很多层的div class，所以直接get得不到里面的院校名字的内容。查了网上说要用lxml或是CSS，就找了一些类似的lxml的tree内容，修改了部分代码，运行解译了很久，结果还是空。代码如下。 求助各位鱼油们，非常感谢
import requests
from bs4 import BeautifulSoup
from lxml import etree


url = "”掌上高考网站 “中“汉语言文学”专业的所有开设院校的链接"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3880.400 QQBrowser/10.8.4554.400 '}
response = requests.get(url,headers=headers)
html = response.content.decode('utf-8')
#print( html)
soup = BeautifulSoup(response.text, "html.parser")
#print(soup)

#print(html )
_element = etree.HTML(html)
print(_element.text)

root = etree.parse(html)


text = _element.xpath('*//div')
ent_text = etree.tostring(text[0],method='text',encoding='utf-8').decode()
ent_text = ent_text.replace(" ","")
ent_text = [i for i in ent_text.split("\n") if i]
print(f"最后结果：{ent_text}")

'''target_div = soup.select("div#target-div")
'''
'''
data = []
for div in soup.find_all("div",class_="school-tab_schoolName__uLwTK"):
    print(div)
    content = div.text.strip()
    data.append(content)
'''

FishC

新年快乐！让我来帮助你解决这个问题。

根据你提供的代码，有几个问题需要修改。首先，你从网页中获取到的HTML内容应当是`response.text`而不是`response.content.decode('utf-8')`。其次，在使用lxml解析器时，你应该使用`etree.HTML(response.text)`而不是`etree.HTML(html)`。此外，你还可以尝试使用CSS选择器来查找目标元素。

下面是修改后的代码示例：

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. url = "掌上高考网站中汉语言文学专业的所有开设院校的链接"
   
5. headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3880.400 QQBrowser/10.8.4554.400 '}
   
6. response = requests.get(url, headers=headers)
   
7. html = response.text
   
8. 
   
9. soup = BeautifulSoup(html, "html.parser")
   
10. 
    
11. # 使用CSS选择器来查找目标元素
    
12. data = []
    
13. for div in soup.select("div.school-tab_schoolName__uLwTK"):
    
14.     content = div.text.strip()
    
15.     data.append(content)
    
16. 
    
17. print("最终结果：")
    
18. for item in data:
    
19.     print(item)

复制代码

希望这可以帮助你成功爬取到目标网页中的院校信息。祝你好运！

以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。

wingplay

按上述的代码，运行还是空的结果。。。再修改，还是没结果

FishC

wingplay 发表于 2024-02-17 13:28
按上述的代码，运行还是空的结果。。。再修改，还是没 ...

非常抱歉你仍然面临问题。可能有几个原因导致你无法获取到数据。首先，确保你提供的网页链接是正确的，并且确保你的请求头部信息是有效的。其次，尝试使用`soup.prettify()`来查看BeautifulSoup解析后的HTML内容，确保你能够正确地找到目标元素。最后，检查目标元素的CSS选择器是否正确。

另外，根据你提供的代码和描述，我注意到你在使用lxml解析器时，将`_element`转换为文本格式时使用了`.text`，这可能会导致你无法正确获取目标元素的内容。你可以尝试使用`.xpath('string()')`而不是`.text`来获取文本内容。下面是修改后的代码示例：

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. from lxml import etree
   
4. 
   
5. url = "掌上高考网站中汉语言文学专业的所有开设院校的链接"
   
6. headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3880.400 QQBrowser/10.8.4554.400'}
   
7. 
   
8. response = requests.get(url, headers=headers)
   
9. html = response.text
   
10. 
    
11. soup = BeautifulSoup(html, "html.parser")
    
12. 
    
13. # 使用CSS选择器来查找目标元素
    
14. data = []
    
15. for div in soup.select("div.school-tab_schoolName__uLwTK"):
    
16.     content = div.text.strip()
    
17.     data.append(content)
    
18. 
    
19. print("最终结果：")
    
20. for item in data:
    
21.     print(item)

复制代码

如果仍然无法获取到数据，请提供更多关于目标网页结构和要抓取内容的详细信息，这样我就能更好地帮助你了。祝你好运！

以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。

阿奇_o

页面是动态加载的，懂？  不懂也没关系，若对速度没啥要求的，用selenium就行了。

wingplay

阿奇_o 发表于 2024-2-17 19:08
页面是动态加载的，懂？  不懂也没关系，若对速度没啥要求的，用selenium就行了。

初学小白，真不懂。对速度没啥要求，请问有没有selenium的类似的代码段可参考？

allen-xy

这个网站需要通过逆向计算出请求中带的参数signsafe，其他参数可以是固定的。
网页（官网url）初始加载时，不会带着数据一起过来。
数据是通过ajax请求（另外一个url）动态获取到的，获取时会有一些验证过程，服务器会进行校验。
通过分析得到关键参数的计算方式，再将计算逻辑转换变python代码，之后就是常规的python爬虫代码了。

关键点是分析出signsafe的获取方式，其他都是基本操作。

bs仅限于获取网页源代码中有数据的情况，本例中的数据是通过动态加载的，所以不适用。
经过验证发现：
1、在请求时，不带signsafe这个字段也可以正常获取数据，可能网站并没有严格校验这个值，但是建议还是按照浏览器正常的请求参数来进行获取数据。
2、网站在生成signsafe时，并不是固定长度的，可能还需要进一步探索。

大概思路就是这样，楼主可以再深入研究。

免责声明：本脚本只用作学习使用，不得用于非法用途。

1. import base64
   
2. from hashlib import sha1
   
3. from hmac import new as hmac_new
   
4. from urllib.parse import unquote
   
5. import requests
   
6. import time
   
7. 
   
8. 
   
9. # signsafe的计算过程。
   
10. def v(t):
    
11.     n = t['SIGN']
    
12.     t = t['str']
    
13.     t = unquote(t)
    
14.     n = hmac_new(n.encode(), t.encode(), sha1).digest()
    
15.     n = base64.b64encode(n).decode()
    
16.     return n
    
17. 
    
18. 
    
19. session = requests.session()
    
20. session.headers = {
    
21.     'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
    
22. }
    
23. 
    
24. for i in range(1, 6):
    
25.     # 取前5页数据
    
26.     url_1 = f"https://api.zjzw.cn/web/api/?is_single=2&local_province_id=12&page={i}&province_id=&request_type=1&size=10&special_id=40&top_school_id=2498&type=&uri=apidata/api/gk/special/school"
    
27. 
    
28.     # 计算signsafe，每次请求都需要带一个signsafe，否则无法取到数据。
    
29.     t = {
    
30.         'SIGN': "D23ABC@#56",
    
31.         'str': url_1
    
32.     }
    
33.     signsafe = v(t)
    
34.     print("==========")
    
35. 
    
36.     # 准备请求url和post请求的参数。
    
37.     url_full = f"{url_1}&signsafe={signsafe}"
    
38.     url_params = {
    
39.         "is_single": "2",
    
40.         "local_province_id": "12",
    
41.         "page": i,
    
42.         "province_id": "",
    
43.         "request_type": "1",
    
44.         "signsafe": signsafe,
    
45.         "size": "10",
    
46.         "special_id": "40",
    
47.         "top_school_id": "2498",
    
48.         "type": "",
    
49.         "uri": "apidata/api/gk/special/school",
    
50.     }
    
51. 
    
52.     # 发送请求，获取数据。
    
53.     resp = session.post(url_full, params=url_params)
    
54.     # 得到的结果是json格式数据
    
55.     res = resp.json()
    
56. 
    
57.     for data in res['data']['item']:
    
58.         # 这里根据需求取内容
    
59.         print(f"学校：{data['name']}\t城市：{data['city_name']}")
    
60. 
    
61.     # 他好我也好
    
62.     time.sleep(2)

复制代码

结果如下：

1. ==========
   
2. 学校：成都锦城学院      城市：成都市
   
3. 学校：北京师范大学      城市：北京市
   
4. 学校：南京大学  城市：南京市        
   
5. 学校：北京大学  城市：北京市        
   
6. 学校：复旦大学  城市：上海市        
   
7. 学校：武汉大学  城市：武汉市        
   
8. 学校：四川大学  城市：成都市        
   
9. 学校：华东师范大学      城市：上海市
   
10. 学校：浙江大学  城市：杭州市        
    
11. 学校：中国人民大学      城市：北京市
    
12. ==========
    
13. 学校：陕西师范大学      城市：西安市
    
14. 学校：中山大学  城市：广州市
    
15. 学校：山东大学  城市：济南市
    
16. 学校：南开大学  城市：天津市
    
17. 学校：暨南大学  城市：广州市
    
18. 学校：南京师范大学      城市：南京市
    
19. 学校：吉林大学  城市：长春市
    
20. 学校：苏州大学  城市：苏州市
    
21. 学校：华中师范大学      城市：武汉市
    
22. 学校：首都师范大学      城市：北京市
    
23. ==========
    
24. 学校：福建师范大学      城市：福州市
    
25. 学校：上海师范大学      城市：上海市
    
26. 学校：北京语言大学      城市：北京市
    
27. 学校：浙江师范大学      城市：金华市
    
28. 学校：西南大学  城市：重庆市
    
29. 学校：湖南师范大学      城市：长沙市
    
30. 学校：上海大学  城市：上海市
    
31. 学校：东北师范大学      城市：长春市
    
32. 学校：西北大学  城市：西安市
    
33. 学校：华南师范大学      城市：广州市
    
34. ==========
    
35. 学校：中央民族大学      城市：北京市
    
36. 学校：上海交通大学      城市：上海市
    
37. 学校：江苏师范大学      城市：徐州市
    
38. 学校：扬州大学  城市：扬州市
    
39. 学校：山东师范大学      城市：济南市
    
40. 学校：河南大学  城市：开封市
    
41. 学校：四川师范大学      城市：成都市
    
42. 学校：兰州大学  城市：兰州市
    
43. 学校：云南大学  城市：昆明市
    
44. 学校：华中科技大学      城市：武汉市
    
45. ==========
    
46. 学校：河北大学  城市：保定市
    
47. 学校：北京外国语大学    城市：北京市
    
48. 学校：郑州大学  城市：郑州市
    
49. 学校：广西师范大学      城市：桂林市
    
50. 学校：黑龙江大学        城市：哈尔滨市
    
51. 学校：安徽大学  城市：合肥市
    
52. 学校：厦门大学  城市：厦门市
    
53. 学校：中国传媒大学      城市：北京市
    
54. 学校：内蒙古大学        城市：呼和浩特市
    
55. 学校：南昌大学  城市：南昌市

复制代码

@管理员，不知道这样回复帖子有没有问题。如有违规等问题，请联系我更正或烦请版主更正。

wingplay

非常感谢！请问您给的代码中url_1 = f"...."   这段的原网页地址是什么？我换了个page=1以后打开是这样的

1. {"code":"0000","message":"成功---success","data":{"item":[{"admissions":"2","central":"2","city_name":"成都市","department":"2","doublehigh":"0","dual_class":"38003","dual_class_name":"","f211":2,"f985":2,"id":"gkspecialschool40/2498","is_top":1,"level_name":"普通本科","name":"成都锦城学院","nature_name":"民办","province_name":"四川","ruanke_level":"","ruanke_rank":"","school_id":2498,"tag_name":"","type_name":"综合类","xueke_rank":9999,"xueke_rank_score":""},{"admissions":"1","central":"2","city_name":"北京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/52","is_top":2,"level_name":"普通本科","name":"北京师范大学","nature_name":"公办","province_name":"北京","ruanke_level":"A+","ruanke_rank":"1","school_id":52,"tag_name":"教育部直属","type_name":"师范类","xueke_rank":"2","xueke_rank_score":"A+"},{"admissions":"1","central":"2","city_name":"南京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/111","is_top":2,"level_name":"普通本科","name":"南京大学","nature_name":"公办","province_name":"江苏","ruanke_level":"A+","ruanke_rank":"2","school_id":111,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"5","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"北京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/31","is_top":2,"level_name":"普通本科","name":"北京大学","nature_name":"公办","province_name":"北京","ruanke_level":"A+","ruanke_rank":"3","school_id":31,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"1","xueke_rank_score":"A+"},{"admissions":"1","central":"2","city_name":"上海市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/132","is_top":2,"level_name":"普通本科","name":"复旦大学","nature_name":"公办","province_name":"上海","ruanke_level":"A+","ruanke_rank":"4","school_id":132,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"3","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"武汉市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/42","is_top":2,"level_name":"普通本科","name":"武汉大学","nature_name":"公办","province_name":"湖北","ruanke_level":"A+","ruanke_rank":"5","school_id":42,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"13","xueke_rank_score":"A-"},{"admissions":"1","central":"2","city_name":"成都市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/99","is_top":2,"level_name":"普通本科","name":"四川大学","nature_name":"公办","province_name":"四川","ruanke_level":"A+","ruanke_rank":"6","school_id":99,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"8","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"上海市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/131","is_top":2,"level_name":"普通本科","name":"华东师范大学","nature_name":"公办","province_name":"上海","ruanke_level":"A+","ruanke_rank":"7","school_id":131,"tag_name":"教育部直属","type_name":"师范类","xueke_rank":"4","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"杭州市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/114","is_top":2,"level_name":"普通本科","name":"浙江大学","nature_name":"公办","province_name":"浙江","ruanke_level":"A+","ruanke_rank":"8","school_id":114,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"6","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"北京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/46","is_top":2,"level_name":"普通本科","name":"中国人民大学","nature_name":"公办","province_name":"北京","ruanke_level":"A+","ruanke_rank":"9","school_id":46,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"9","xueke_rank_score":"A-"}],"numFound":626},"location":"","encrydata":""}

复制代码

wingplay

allen-xy 发表于 2024-2-19 18:19
这个网站需要通过逆向计算出请求中带的参数signsafe，其他参数可以是固定的。
网页（官网url）初始加载时 ...

非常感谢！请问您给的代码中url_1 = f"...."   这段的原网页地址是什么？我换了个page=1以后打开是这样的

1. {"code":"0000","message":"成功---success","data":{"item":[{"admissions":"2","central":"2","city_name":"成都市","department":"2","doublehigh":"0","dual_class":"38003","dual_class_name":"","f211":2,"f985":2,"id":"gkspecialschool40/2498","is_top":1,"level_name":"普通本科","name":"成都锦城学院","nature_name":"民办","province_name":"四川","ruanke_level":"","ruanke_rank":"","school_id":2498,"tag_name":"","type_name":"综合类","xueke_rank":9999,"xueke_rank_score":""},{"admissions":"1","central":"2","city_name":"北京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/52","is_top":2,"level_name":"普通本科","name":"北京师范大学","nature_name":"公办","province_name":"北京","ruanke_level":"A+","ruanke_rank":"1","school_id":52,"tag_name":"教育部直属","type_name":"师范类","xueke_rank":"2","xueke_rank_score":"A+"},{"admissions":"1","central":"2","city_name":"南京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/111","is_top":2,"level_name":"普通本科","name":"南京大学","nature_name":"公办","province_name":"江苏","ruanke_level":"A+","ruanke_rank":"2","school_id":111,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"5","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"北京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/31","is_top":2,"level_name":"普通本科","name":"北京大学","nature_name":"公办","province_name":"北京","ruanke_level":"A+","ruanke_rank":"3","school_id":31,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"1","xueke_rank_score":"A+"},{"admissions":"1","central":"2","city_name":"上海市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/132","is_top":2,"level_name":"普通本科","name":"复旦大学","nature_name":"公办","province_name":"上海","ruanke_level":"A+","ruanke_rank":"4","school_id":132,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"3","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"武汉市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/42","is_top":2,"level_name":"普通本科","name":"武汉大学","nature_name":"公办","province_name":"湖北","ruanke_level":"A+","ruanke_rank":"5","school_id":42,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"13","xueke_rank_score":"A-"},{"admissions":"1","central":"2","city_name":"成都市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/99","is_top":2,"level_name":"普通本科","name":"四川大学","nature_name":"公办","province_name":"四川","ruanke_level":"A+","ruanke_rank":"6","school_id":99,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"8","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"上海市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/131","is_top":2,"level_name":"普通本科","name":"华东师范大学","nature_name":"公办","province_name":"上海","ruanke_level":"A+","ruanke_rank":"7","school_id":131,"tag_name":"教育部直属","type_name":"师范类","xueke_rank":"4","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"杭州市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/114","is_top":2,"level_name":"普通本科","name":"浙江大学","nature_name":"公办","province_name":"浙江","ruanke_level":"A+","ruanke_rank":"8","school_id":114,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"6","xueke_rank_score":"A"},{"admissions":"1","central":"2","city_name":"北京市","department":"1","doublehigh":"0","dual_class":"38000","dual_class_name":"双一流","f211":1,"f985":1,"id":"gkspecialschool40/46","is_top":2,"level_name":"普通本科","name":"中国人民大学","nature_name":"公办","province_name":"北京","ruanke_level":"A+","ruanke_rank":"9","school_id":46,"tag_name":"教育部直属","type_name":"综合类","xueke_rank":"9","xueke_rank_score":"A-"}],"numFound":626},"location":"","encrydata":""}

复制代码

allen-xy

wingplay 发表于 2024-2-21 10:24
非常感谢！请问您给的代码中url_1 = f"...."   这段的原网页地址是什么？我换了个page=1以后打开是这样的

这个url链接就是网页上面每页具体数据的请求链接（见附件截图），这个url并不是前台展示的初始url（你在浏览器地址栏处写的），而是网站后台自动发起请求时使用的链接（返回的是json数据，就是字典，从字典里拿你想要的数据就行。这种数据不是在浏览器中直接展示用的，而是需要浏览器渲染完成后才展示在初始url的页面中。），你得进入后再具体分析他使用的是哪个url来获取数据的。
你需要第2页数据就把page写成2，第3页写3，依次类推。
页码可以通过for循环实现。

wingplay

allen-xy 发表于 2024-2-21 10:37
这个url链接就是网页上面每页具体数据的请求链接（见附件截图），这个url并不是前台展示的初始url（你 ...

再请问signsafe的计算过程是怎么定的呢？

allen-xy

wingplay 发表于 2024-2-22 13:05
再请问signsafe的计算过程是怎么定的呢？

通过查找url链接的载荷数据，找到每次请求的不同点，再去流量器抓包中找这个点的位置，大概率会在js代码中，然后再从js代码中反着查找，最终找到生成位置。
在这里不可能一点一点说清楚，内容太多。。。
这个过程就是逆向，阅读网站设计逻辑，反向生成python代码。
需要懂html、JavaScript等前端技术，一点一点推导结果。