Python爬虫08(朋友实训项目)
在豆瓣音乐标签: 流行里面爬取标题，表演者，发行时间，链接import requests
from bs4 import BeautifulSoup
import xlwt
url = "https://music.douban.com/tag/%E6%B5%81%E8%A1%8C?start=260&type=T"
headers = {'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36','Cookie':
'll="118254"; bid=hUyLZc0IQ-0; __utmc=30149280; __utmz=30149280.1704177535.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _pk_id.100001.afe6=e3aa424645194a40.1704178112.; __yadk_uid=vQ2OnDmCyRJ7MgONjIYqxroh3kmJOC0n; _vwo_uuid_v2=DC4889CAE9BF69AB947F1C739A359D3B1|c7a477b87f67bc643d65031d1e233036; dbcl2="276885079:IyaMMU84wwk"; ck=y-tF; push_noty_num=0; push_doumail_num=0; __utmv=30149280.27688; ap_v=0,6.0; _pk_ref.100001.afe6=%5B%22%22%2C%22%22%2C1704181059%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_ses.100001.afe6=1; __utma=30149280.678212503.1704177535.1704177535.1704181060.2; __utmt=1;__utmb=30149280.3.9.1704181060'}
r=requests.get(url,headers=headers)
content=r.content.decode('utf-8')
with open('douban.html','wb')as file:
file.write(r.content)
title_list=[]
href_list=[]
for i in range(0,400,20):
url = 'https://music.douban.com/tag/%E6%B5%81%E8%A1%8C?start={i}&type=T'
soup=BeautifulSoup(content,'lxml')
divs=soup.find_all('div',class_="pl2")
for div in divs:
href=div.find('a')['href']
href_list.append(href)
title=div.find('a').text
title_list.append(title)
print(href_list)
print(title_list)
wb=xlwt.Workbook()
sheet1=wb.add_sheet('豆瓣音乐')
sheet1.write(0,0,'标题')
sheet1.write(0,1,'链接')
for row in range(1,len(title_list)+1,1):
sheet1.write(row,0,title_list)
sheet1.write(row,1,href_list)
wb.save(r'C:\Users\梦呓\Desktop\34qc.xls') 代码中有几个问题需要修复：
1. 缺少缩进：在for循环之前，需要在title_list和href_list的定义语句后面添加一个缩进。
2. 数据获取问题：在循环中，应该使用新的url来发送请求并获得内容，而不是一直使用初始的content变量。所以，在循环开始的时候，应该加上r=requests.get(url,headers=headers)这一行，然后再调用content=r.content.decode('utf-8')。
3. 写入Excel文件问题：sheet1.write(row,0,title_list)和sheet1.write(row,1,href_list)这两行，应该将title_list改为title_list，将href_list改为href_list。
修改后的代码如下：
import requests
from bs4 import BeautifulSoup
import xlwt
url = "https://music.douban.com/tag/%E6%B5%81%E8%A1%8C?start=260&type=T"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36','Cookie': 'll="118254"; bid=hUyLZc0IQ-0; __utmc=30149280; __utmz=30149280.1704177535.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _pk_id.100001.afe6=e3aa424645194a40.1704178112.; __yadk_uid=vQ2OnDmCyRJ7MgONjIYqxroh3kmJOC0n; _vwo_uuid_v2=DC4889CAE9BF69AB947F1C739A359D3B1|c7a477b87f67bc643d65031d1e233036; dbcl2="276885079:IyaMMU84wwk"; ck=y-tF; push_noty_num=0; push_doumail_num=0; __utmv=30149280.27688; ap_v=0,6.0; _pk_ref.100001.afe6=%5B%22%22%2C%22%22%2C1704181059%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_ses.100001.afe6=1; __utma=30149280.678212503.1704177535.1704177535.1704181060.2; __utmt=1;__utmb=30149280.3.9.1704181060'}
r = requests.get(url,headers=headers)
content = r.content.decode('utf-8')
with open('douban.html','wb') as file:
file.write(r.content)
title_list = []
href_list = []
for i in range(0,400,20):
url = f'https://music.douban.com/tag/%E6%B5%81%E8%A1%8C?start={i}&type=T'
r = requests.get(url,headers=headers)
content = r.content.decode('utf-8')
soup = BeautifulSoup(content,'lxml')
divs = soup.find_all('div',class_="pl2")
for div in divs:
href = div.find('a')
href_list.append(href)
title = div.find('a').text
title_list.append(title)
print(href_list)
print(title_list)
wb = xlwt.Workbook()
sheet1 = wb.add_sheet('豆瓣音乐')
sheet1.write(0,0,'标题')
sheet1.write(0,1,'链接')
for row in range(1,len(title_list)+1,1):
sheet1.write(row,0,title_list)
sheet1.write(row,1,href_list)
wb.save(r'C:\Users\梦呓\Desktop\34qc.xls')
这样，代码就可以正常运行并将爬取到的数据写入Excel文件中了。
以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。 Traceback (most recent call last):
File "D:\Pycharm\PythonProject\2.py", line 38, in <module>
sheet1.write(row,1,href_list)
File "D:\Pycharm\PythonProject\venv\Lib\site-packages\xlwt\Worksheet.py", line 1088, in write
self.row(r).write(c, label, style)
File "D:\Pycharm\PythonProject\venv\Lib\site-packages\xlwt\Row.py", line 254, in write
raise Exception("Unexpected data type %r" % type(label))
Exception: Unexpected data type <class 'bs4.element.Tag'>
Traceback (most recent call last):
File "D:\Pycharm\PythonProject\2.py", line 38, in <module>
sheet1.write(row,1,href_list)
File "D:\Pycharm\PythonProject\venv\Lib\site-packages\xlwt\Worksheet.py", line 1088, in write
self.row(r).write(c, label, style)
File "D:\Pycharm\PythonProject\venv\Lib\site-packages\xlwt\Row.py", line 254, in write
raise Exception("Unexpected data type %r" % type(label))
Exception: Unexpected data type <class 'bs4.element.Tag'>
FishC 发表于 2024-1-3 19:07
代码中有几个问题需要修复：
1. 缺少缩进：在for循环之前，需要在title_list和href_list的定义语句后面 ...
Traceback (most recent call last):
File "D:\Pycharm\PythonProject\2.py", line 38, in <module>
sheet1.write(row,1,href_list)
File "D:\Pycharm\PythonProject\venv\Lib\site-packages\xlwt\Worksheet.py", line 1088, in write
self.row(r).write(c, label, style)
File "D:\Pycharm\PythonProject\venv\Lib\site-packages\xlwt\Row.py", line 254, in write
raise Exception("Unexpected data type %r" % type(label))
Exception: Unexpected data type <class 'bs4.element.Tag'> liyifeng295641 发表于 2024-01-03 19:35
Traceback (most recent call last):
File "D:\Pyc ...
这个问题是由于xlwt库对于数据类型的限制引起的。xlwt库只能处理一些基本的数据类型，比如字符串、整数、浮点数等，但不能直接处理列表类型。要解决这个问题，你需要将href_list中的列表转换为字符串，然后才能将其写入Excel中。你可以使用join方法将列表元素连接成一个字符串，并用逗号或其他分隔符分隔开。修改代码如下：import requestsfrom bs4 import BeautifulSoupimport xlwturl = "https://music.douban.com/tag/%E6%B5%81%E8%A1%8C?start=260&type=T"headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36','Cookie': 'll="118254"; bid=hUyLZc0IQ-0; __utmc=30149280; __utmz=30149280.1704177535.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _pk_id.100001.afe6=e3aa424645194a40.1704178112.; __yadk_uid=vQ2OnDmCyRJ7MgONjIYqxroh3kmJOC0n; _vwo_uuid_v2=DC4889CAE9BF69AB947F1C739A359D3B1|c7a477b87f67bc643d65031d1e233036; dbcl2="276885079:IyaMMU84wwk"; ck=y-tF; push_noty_num=0; push_doumail_num=0; __utmv=30149280.27688; ap_v=0,6.0; _pk_ref.100001.afe6=%5B%22%22%2C%22%22%2C1704181059%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_ses.100001.afe6=1; __utma=30149280.678212503.1704177535.1704177535.1704181060.2; __utmt=1;__utmb=30149280.3.9.1704181060'}r = requests.get(url,headers=headers)content = r.content.decode('utf-8')with open('douban.html','wb') as file: file.write(r.content)title_list = href_list = for i in range(0,400,20): url = f'https://music.douban.com/tag/%E6%B5%81%E8%A1%8C?start={i}&type=T' r = requests.get(url,headers=headers) content = r.content.decode('utf-8') soup = BeautifulSoup(content,'lxml') divs = soup.find_all('div',class_="pl2") for div in divs: href = div.find('a') href_list.append(href) title = div.find('a').text title_list.append(title)print(href_list)print(title_list)wb = xlwt.Workbook()sheet1 = wb.add_sheet('豆瓣音乐')sheet1.write(0,0,'标题')sheet1.write(0,1,'链接')for row in range(1,len(title_list)+1,1): sheet1.write(row,0,title_list) sheet1.write(row,1,','.join(href_list))wb.save(r'C:\Users\梦呓\Desktop\34qc.xls')这样，代码就可以正常运行并将爬取到的数据写入Excel文件中了。请注意，为了保持代码的可读性，我使用了来索引列表元素，因为在循环中row从1开始。
以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。 {:10_279:}
页:
[1]