spider [scrapy.core.scraper] ERROR: Spider error processing <GET https
from scrapy import Requestfrom scrapy.spiders import Spider
from lianjia_home.items import LianjiaHomeItem
class HomeSpider(Spider):
name = 'home'
current_page = 1
def start_requests(self):
url = "https://su.lianjia.com/ershoufang/"
yield Request(url)
def parse(self, response):
#1.提取主页面中的房屋信息
list_selector = response.xpath("//li/div[@class='info clear']")
for one_selector in list_selector:
try:
name = one_selector.xpath("div[@class='address']/"
"div[@class='houseInfo']"
"/a/text()").extract_first()
other = one_selector.xpath("div[@class='address']/"
"div[@class='houseInfo']"
"/text()").extract_first()
other_list = other.splite("|")
type = other_list.strip(" ")
area = other_list.strip(" ")
direction = other_list.strip(" ")
fitment = other_list.strip(" ")
elevator = other_list.strip(" ")
price_list = one_selector.xpath("div[@class='priceInfo']//span/text()")
total_price = price_list.extract()
unit_price = price_list.extract()
item = LianjiaHomeItem()
#将已经获取的字段保存再item对象中
item["name"] = name.strip(" ")
item["type"] = type
item["area"] = area
item["direction"] = direction
item["fitment"] = fitment
item["elevator"] = elevator
item["total_price"] = total_price
item["unit_price"] = unit_price
#2.获取详细页URL
url = one_selector.xpath("div[@class='title']/a/@href").extract_first()
#3.生成详情页的请求对象，参数meta保存房屋部分数据
yield Request(url,
meta={"item": item},
callback=self.property_parse)
except:
pass
#4.获取下一页URL，并生成Request请求
#（1）获取下一页URL。仅在解析第一页时获取总页数的值
if self.current_page == 1:
self.total_page = response.xpath("//div[@class='page-box house-1st-page-box']"
"//@page-data").re("\d+")
self.total_page = int(self.total_page)
self.current_page += 1
if self.current_page <= self.total_page:
next_url = "https://su.lianjia.com/ershoufang/pg%d"%(self.current_page)
#根据URL生成Req。使用yield提交给引擎
yield Request(next_url)
def property_parse(self,response):
property = response.xpath("//div[@class='base']/div[@class='content']/u1/li/text()").extract_first()
item = response.meta["item"]
item["property"] = property
yield item
2
021-05-06 15:17:21 INFO: Scrapy 2.4.1 started (bot: lianjia_home)
2021-05-06 15:17:21 INFO: Versions: lxml 4.6.3.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec7 2020, 17:08:21) , pyOpenSSL 20.0.1 (OpenSSL 1.1.1k25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19041-SP0
2021-05-06 15:17:21 DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-05-06 15:17:21 INFO: Overridden settings:
{'BOT_NAME': 'lianjia_home',
'NEWSPIDER_MODULE': 'lianjia_home.spiders',
'SPIDER_MODULES': ['lianjia_home.spiders']}
2021-05-06 15:17:21 INFO: Telnet Password: 735a85a99936aae5
2021-05-06 15:17:21 INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2021-05-06 15:17:21 INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-05-06 15:17:21 INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-05-06 15:17:21 INFO: Enabled item pipelines:
['lianjia_home.pipelines.FilterPipeline', 'lianjia_home.pipelines.CSVPipeline']
2021-05-06 15:17:21 INFO: Spider opened
2021-05-06 15:17:21 INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-05-06 15:17:21 INFO: Telnet console listening on 127.0.0.1:6023
找到错误啦
self.total_page = response.xpath("//div[@class='page-box house-1st-page-box']"
那个house-1st 改成 house-lst 就好了。是l不是1
页:
[1]