scrapy报错
import scrapyclass BpictureSpider(scrapy.Spider):
name = 'bpicture'
allowed_domains = ['www.qiushibaike.com']
start_urls = ['https://www.qiushibaike.com/text/']
def parse(self, response):
#解析
div_list = response.xpath('//*[@class="col1 old-style-col1"]/div/div')
for div in div_list:
author = div.xpath('./div/a/h2/text()').extract()
content = div.xpath('./a/div/span//text()').extract_first()
content = ''.join(content)
print(author, content)
break
错误结果如图：
尝试过将list.extract()改成list().extract_first()还是一样的结果
在15行后面加上print(content),看看content的内容，错误的提示是只能join可迭代的对象。 冬雪雪冬 发表于 2021-8-31 15:40
在15行后面加上print(content),看看content的内容，错误的提示是只能join可迭代的对象。
import scrapy
class BpictureSpider(scrapy.Spider):
name = 'bpicture'
allowed_domains = ['www.xxx.com']
start_urls = ['https://www.qiushibaike.com/text/']
def parse(self, response):
#解析
div_list = response.xpath('//*[@class="col1 old-style-col1"]/div')
#response.xpath('//div[@class="col1 old-style-col1"]')
for div in div_list:
author = div.xpath('./div/a/h2/text()').extract_first()
content = div.xpath('./a/div/span//text()').extract()
print(content)
content = ''.join(content)
print(author, content)
break
我先将author和content的extract改成了现在的样子，返回的结果是None
然后我有将div_list的路径修改呈现在的样子，成功了
我scrapy用的很少，用bs写了一个供参考。
import requests
from bs4 import BeautifulSoup
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36"}
res = requests.get('https://www.qiushibaike.com/text/', headers = headers)
soup = BeautifulSoup(res.text, 'html.parser')
targets = soup.find_all('div', class_ = 'content')
for each in targets:
print(each.span.text)
页:
[1]