[已解决]爬取豆瓣电影信息

摆动的呆毛

import requests
import bs4
import time

with open('./豆瓣.txt',mode='w+',encoding='utf-8') as f:
    headers = {
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
        }
    q = input("输入要爬几页")
    for i in range(int(q)):
        url = "https://movie.douban.com/top250"
        params = {q : (i * 25)}
        res = requests.get(url=url, headers=headers, params=params)
        soup = bs4.BeautifulSoup(res.text, "html.parser")
        targets = soup.find_all("div", class_="hd")
        for each in targets:
            f.write(each.a.text)
        time.sleep(1)

为什么无法爬取下一页，一直在第一页里循环

最佳答案

月排行榜 / 总排行榜

qiuyouzhi

2020-8-7 22:01:17

import requests
import bs4
import time

with open('./豆瓣.txt',mode='w+',encoding='utf-8') as f:
    headers = {
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
        }
    q = input("输入要爬几页")
    for i in range(int(q)):
        url = f"https://movie.douban.com/top250?start={i * 25}&filter="
        res = requests.get(url=url, headers=headers)
        soup = bs4.BeautifulSoup(res.text, "html.parser")
        targets = soup.find_all("div", class_="hd")
        for each in targets:
            f.write(each.a.text)
        time.sleep(1)

跳转到最佳答案楼层

qiuyouzhi

import requests
import bs4
import time

with open('./豆瓣.txt',mode='w+',encoding='utf-8') as f:
    headers = {
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
        }
    q = input("输入要爬几页")
    for i in range(int(q)):
        url = f"https://movie.douban.com/top250?start={i * 25}&filter="
        res = requests.get(url=url, headers=headers)
        soup = bs4.BeautifulSoup(res.text, "html.parser")
        targets = soup.find_all("div", class_="hd")
        for each in targets:
            f.write(each.a.text)
        time.sleep(1)

1q23w31

你一直请求第一页当然不变

陈尚涵

一直是第1页

心驰神往

鱼币

a~qie

qiuyouzhi 发表于 2020-8-7 22:01

现在没用了吗？