[已解决]爬虫时的问题

柿子饼同学

我爬豆瓣250的那个网站，它不回应是怎么回事

1. import requests as r
   
2. import bs4 as b
   
3. 
   
4. res = r.get(r'https://movie.douban.com/top250')
   
5. soup = b.BeautifulSoup(res.text, 'html.parser')
   
6. target = soup.find_all('div', class_='hd')
   
7. for i in target:
   
8.      print(each.a.span.text)
   

复制代码

这是我写的，运行之后什么都不出来，这是什么情况

最佳答案

月排行榜 / 总排行榜

青出于蓝

2021-2-25 15:06:16

草率草率，没在编译器上运行

1. import requests as r
   
2. import bs4 as b
   
3. headers={"User-Agent":
   
4.         "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0"}
   
5. res = r.get('https://movie.douban.com/top250', headers=headers)
   
6. soup = b.BeautifulSoup(res.text, 'html.parser')
   
7. target = soup.find_all('div', class_='hd')
   
8. for each in target:    # 改成each
   
9.      print(each.a.span.text)
   

复制代码

粗心了，抱歉

跳转到最佳答案楼层

suchocolate

加header

1. headers = {'user-agent': 'Mozilla'}
   
2. res = r.get('https://movie.douban.com/top250', headers=headers)
   
3. soup = b.BeautifulSoup(res.text, 'html.parser')
   
4. target = soup.find_all('div', class_='hd')
   
5. for each in target:    # 改成each
   
6.      print(each.a.span.text)
   
7. 
   

复制代码

青出于蓝

豆瓣有反爬虫系统，加上handers就好了

青出于蓝

以下为完整代码

1. import requests as r
   
2. import bs4 as b
   
3. headers={"User-Agent":
   
4.         "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0"}
   
5. res = r.get('https://movie.douban.com/top250', headers=headers)
   
6. soup = b.BeautifulSoup(res.text, 'html.parser',headers=headers)
   
7. target = soup.find_all('div', class_='hd')
   
8. for each in target:    # 改成each
   
9.      print(each.a.span.text)

复制代码

柿子饼同学

青出于蓝 发表于 2021-2-25 14:59
以下为完整代码

它出错了

1. ============== RESTART: C:\Users\86177\Desktop\python学习\豆瓣250爬虫.py =============
   
2. Traceback (most recent call last):
   
3.   File "C:\Users\86177\Desktop\python学习\豆瓣250爬虫.py", line 6, in <module>
   
4.     soup = b.BeautifulSoup(res.text, 'html.parser',headers=headers)
   
5.   File "C:\Users\86177\AppData\Local\Programs\Python\Python39-32\lib\site-packages\bs4\__init__.py", line 252, in __init__
   
6.     builder = builder_class(**kwargs)
   
7.   File "C:\Users\86177\AppData\Local\Programs\Python\Python39-32\lib\site-packages\bs4\builder\_htmlparser.py", line 325, in __init__
   
8.     super(HTMLParserTreeBuilder, self).__init__(**kwargs)
   
9. TypeError: __init__() got an unexpected keyword argument 'headers'
   
10. >>>
    

复制代码

青出于蓝

草率草率，没在编译器上运行

1. import requests as r
   
2. import bs4 as b
   
3. headers={"User-Agent":
   
4.         "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0"}
   
5. res = r.get('https://movie.douban.com/top250', headers=headers)
   
6. soup = b.BeautifulSoup(res.text, 'html.parser')
   
7. target = soup.find_all('div', class_='hd')
   
8. for each in target:    # 改成each
   
9.      print(each.a.span.text)
   

复制代码

粗心了，抱歉

青出于蓝

柿子饼同学 发表于 2021-2-25 15:02
它出错了

这样好了吗，环境不太方便，看不出有啥问题了。

柿子饼同学

谢谢谢谢，能用了