requests+beautifulsoup 使用select/ccs选择器爬取
本帖最后由 加冰青柠檬 于 2020-4-3 14:48 编辑新人第一次发帖，跟着甲鱼老师学习了这么久，刚开始只是为了参加大学生数学建模比赛学习python爬虫，入坑之后跟着甲鱼老师复习了一下C语言，后来又跟着小甲鱼学习数据结构与算法，然后又跟了零基础入门学习python，现在终于开始了我一开始的目的，爬取了妹子图的网站推荐图片，这也是第一次我自主的完成爬虫实战，妹子什么的不重要，主要就是和大家交流交流技术，有什么不到位还请大家指点指点
模块清单：
requests
BeautifulSoup
lxml
import requests
from bs4 import BeautifulSoup
import lxml
import os
def get_home_url(home_url):
headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.100 Safari/537.36"}
res=requests.get(home_url,headers=headers)
soup=BeautifulSoup(res.text,"lxml")
soup_mm_url=soup.select(".postlist #pins .lazy")
return soup_mm_url
def get_mm_url(soup_mm_url):
load_url = {}
for count in soup_mm_url:
# print(count.get("data-original"))
load_url=count.get("data-original")
# print(count.get("alt"))
save_img(load_url)
def os_op():
os.mkdir("XXOO_max")
os.chdir("XXOO_max")
# print(load_url)
def save_img(load_url):
for i in load_url:
url=load_url
headers_mm={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.100 Safari/537.36",
"referer": "https://www.mzitu.com/"}
resp=requests.get(url,headers=headers_mm)
filename=i+".jpg"
# print(filename)
with open(filename,"wb") as fp:
fp.write(resp.content)
def main():
os_op()
for page in range(2,10):
home_url = "https://www.mzitu.com/page/"
home_url=home_url+str(page)
print("开始爬取第%d页图片"%page)
print(home_url)
soup_mm_url=get_home_url(home_url)
get_mm_url(soup_mm_url)
if __name__ == "__main__":
main()
另外爬完之后我笔记本的网断了，？？？，不会是我家路由器担心的身体吧，，，现在我用的是手机热点，各位鱼油有什么好办法吗
页:
[1]