python怎么实现获取下拉后页面加载的内容？
steam评测页面下拉后每次出现十条新内容，代码运行时无法获取下拉后的加载内容，求教import requests
from bs4 import BeautifulSoup
import json
import time
# from selenium import webdriver
#
# driver = webdriver.Chrome(r'C:\Users\m1359\AppData\Local\Google\Chrome\Application\chromedriver.exe')
def sen_from_text(text):
SENTIMENT_URL = 'http://api.bosonnlp.com/sentiment/analysis'
h = {'X-Token': 'balbala'}# your token
data = json.dumps(text)
resp = requests.post(SENTIMENT_URL, headers=h, data=data.encode('utf-8'))
resp = json.loads(resp.text)# print(resp)
front = float(resp)
return front
headers = {"Accept-Language": "zh-CN,zh;q=0.9"}
file = open('steam.txt', 'w+', encoding='utf-8')
for i in range(1, 10): # 动态获取评测页面
url = 'http://steamcommunity.com/app/578080/homecontent/?userreviewsoffset=' + str(10 * (i - 1)) + '&p=' + str(
i) + '&workshopitemspage=' + str(i) + '&readytouseitemspage=' + str(i) + '&mtxitemspage=' + str(
i) + '&itemspage=' + str(i) + '&screenshotspage=' + str(i) + '&videospage=' + str(i) + '&artpage=' + str(
i) + '&allguidepage=' + str(i) + '&webguidepage=' + str(i) + '&integratedguidepage=' + str(
i) + '&discussionspage=' + str(
i) + '&numperpage=10&browsefilter=toprated&browsefilter=toprated&appid=578080&l=schinese&appHubSubSection=10' \
'&filterLanguage=default&searchText=&forceanon=1 ' # steam评测页面的url，参考链接https://www.tinymind.net.cn/articles/6c517fc1b33931写的
html = requests.get(url, headers=headers, timeout=10).text
soup = BeautifulSoup(html, 'html.parser')
reviews = soup.find_all('div', {'class': "apphub_Card modalContentLink interactable"})
for review in reviews:
# nick = review.find('div', {'class': 'apphub_CardContentAuthorName offline ellipsis'})
title = review.find('div', {'class': 'title'}).text # 获取评价（推荐or不推荐）
hour = review.find('div', {'class': 'hours'}).text.split(' ') # 获取游玩时间
link = review.find('a').attrs['href'] # 获取用户主页链接
comment = review.find('div', {'class': 'apphub_CardTextContent'}).text.split('\n').strip('\t') # 获取评价内容
# sen = sen_from_text(comment)
print(title, hour, link, comment) 下拉一般是ajax，【浏览器f12】-【网络】-【xhr】-【下拉一次】-【学ajax的头模拟get】 本帖最后由 Twilight6 于 2020-5-27 21:52 编辑
下拉页面后加载的内容是 ajax 现象
看这个视频教程吧，这个是爬取豆瓣下拉时刷新页面的案例
https://www.bilibili.com/video/BV1z541167mu?p=8 suchocolate 发表于 2020-5-27 21:36
下拉一般是ajax，【浏览器f12】-【网络】-【xhr】-【下拉一次】-【学ajax的头模拟get】
就是模拟的那个url
Request URL: https://steamcommunity.com/app/578080/homecontent/?userreviewscursor=AoIFP%2FU1fmAAAABwo565AQ%3D%3D&userreviewsoffset=10&p=2&workshopitemspage=2&readytouseitemspage=2&mtxitemspage=2&itemspage=2&screenshotspage=2&videospage=2&artpage=2&allguidepage=2&webguidepage=2&integratedguidepage=2&discussionspage=2&numperpage=10&browsefilter=toprated&browsefilter=toprated&l=schinese&appHubSubSection=10&filterLanguage=default&searchText=
我是按照文章里的方法拼接的，应该是没有问题的吧 诶呀二傻子是我 发表于 2020-5-27 22:14
就是模拟的那个url
我是按照文章里的方法拼接的，应该是没有问题的吧
照着实际的url拼接应该没问题，不过ajax的head也得写全，最好是原封不动复制head。ajax的head里头最重要的参数是"X-Requested-With": "XMLHttpRequest" 用requests-html直接模拟下拉操作更简单 {:7_146:} driver.find_element_by_xpath('//li').click()
页:
[1]