爬虫
本帖最后由 kygschp 于 2021-7-24 19:39 编辑https://movie.douban.com/typerank?type_name=%E6%82%AC%E7%96%91&type=10&interval_id=100:90&action=
想要爬取豆瓣排行榜上电影的名字
但是下面的这段代码貌似有点问题，targets那里获得不了东西，打印出来是个空列表，请问下各位大佬是什么原因导致的？
def find_movies(soup):
movie_names = []
targets = soup.find_all('div', class_="movie_name")
print(targets)
for each in targets:
movie_name = each.span.a.text
print(movie_name)
movie_names.append(movie_name)
完整代码如下
import requests
import bs4
def open_url(url):
header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
response = requests.get(url, headers=header)
return response
def find_movies(soup):
movie_names = []
targets = soup.find_all('div', class_="movie_name")
print(targets)
for each in targets:
movie_name = each.span.a.text
print(movie_name)
movie_names.append(movie_name)
movie_shorts = []
targets = soup.find_all('div', class_='movie-misc')
for each in targets:
movie_short = each.text
movie_shorts.append(movie_short)
movie_infos = []
length = len(movie_names)
for i in range(length):
movie_infos.append(movie_names+movie_shorts+'\n')
return movie_infos
def save(movie_infos):
file = r'D:\python\movies.txt'
with open(file, 'w') as f:
for each in movie_infos:
f.write(each)
def main():
url = 'https://movie.douban.com/typerank?type_name=%E6%82%AC%E7%96%91&type=10&interval_id=100:90&action='
response = open_url(url)
soup = bs4.BeautifulSoup(response.text, 'html.parser')
movie_infos = find_movies(soup)
save(movie_infos)
if __name__ == '__main__':
main() 本帖最后由 suchocolate 于 2021-7-25 10:50 编辑
你的这个url和其他 top250之类的榜单不同，这个url的页面的数据是通过ajax加载的。
url不同，代码改成：
import requests
def main():
f = open('result.txt', 'w', encoding='utf-8')
headers = {'user-agent': 'firefox'}
for n in range(10):
url = f'https://movie.douban.com/j/chart/top_list?type=10&interval_id=100%3A90&action=&start={n}&limit=20'
r = requests.get(url, headers=headers)
for item in r.json():
print(item['title'], item['types'])
f.write(f'{item["title"]} {item["types"]}\n')
f.close()
if __name__ == '__main__':
main() 本帖最后由 kygschp 于 2021-7-25 11:41 编辑
suchocolate 发表于 2021-7-25 10:48
你的这个url和其他 top250之类的榜单不同，这个url的页面的数据是通过ajax加载的。
url不同，代码改成：
感谢大佬，想知道url前面的那个f是什么意思，刚学还没接触过{:9_239:} kygschp 发表于 2021-7-25 11:34
感谢大佬，想知道url前面的那个f是什么意思，刚学还没接触过
额，已经明白了，感谢大佬 {:5_95:}
页:
[1]