spider [scrapy.core.scraper] ERROR: Spider error processing <GET https

甲鱼飞鱼

[code][code]from scrapy import Request
from scrapy.spiders import Spider
from lianjia_home.items import LianjiaHomeItem

class HomeSpider(Spider):
    name = 'home'
    current_page = 1
    def start_requests(self):
        url = "https://su.lianjia.com/ershoufang/"
        yield Request(url)

    def parse(self, response):
        #1.提取主页面中的房屋信息
        list_selector = response.xpath("//li/div[@class='info clear']")
        for one_selector in list_selector:
            try:
                name = one_selector.xpath("div[@class='address']/"
                                          "div[@class='houseInfo']"
                                          "/a/text()").extract_first()
                other = one_selector.xpath("div[@class='address']/"
                                           "div[@class='houseInfo']"
                                           "/text()").extract_first()
                other_list = other.splite("|")
                type = other_list[1].strip(" ")
                area = other_list[2].strip(" ")
                direction = other_list[3].strip(" ")
                fitment = other_list[4].strip(" ")
                elevator = other_list[5].strip(" ")

                price_list = one_selector.xpath("div[@class='priceInfo']//span/text()")
                total_price = price_list[0].extract()
                unit_price = price_list[1].extract()
                item = LianjiaHomeItem()
                #将已经获取的字段保存再item对象中
                item["name"] = name.strip(" ")
                item["type"] = type
                item["area"] = area
                item["direction"] = direction
                item["fitment"] = fitment
                item["elevator"] = elevator
                item["total_price"] = total_price
                item["unit_price"] = unit_price
                #2.获取详细页URL
                url = one_selector.xpath("div[@class='title']/a/@href").extract_first()
                #3.生成详情页的请求对象，参数meta保存房屋部分数据
                yield Request(url,
                              meta={"item": item},
                              callback=self.property_parse)
            except:
                pass
        #4.获取下一页URL，并生成Request请求
        #（1）获取下一页URL。仅在解析第一页时获取总页数的值
        if self.current_page == 1:
            self.total_page = response.xpath("//div[@class='page-box house-1st-page-box']"
                                             "//@page-data").re("\d+")
            self.total_page = int(self.total_page[0])
        self.current_page += 1
        if self.current_page <= self.total_page:
            next_url = "https://su.lianjia.com/ershoufang/pg%d"%(self.current_page)
        #根据URL生成Req。使用yield提交给引擎
            yield Request(next_url)

    def property_parse(self,response):
        property = response.xpath("//div[@class='base']/div[@class='content']/u1/li[12]/text()").extract_first()
        item = response.meta["item"]
        item["property"] = property
        yield item

[/code][/code]

2
021-05-06 15:17:21 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: lianjia_home)
2021-05-06 15:17:21 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k  25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19041-SP0
2021-05-06 15:17:21 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-05-06 15:17:21 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'lianjia_home',
 'NEWSPIDER_MODULE': 'lianjia_home.spiders',
 'SPIDER_MODULES': ['lianjia_home.spiders']}
2021-05-06 15:17:21 [scrapy.extensions.telnet] INFO: Telnet Password: 735a85a99936aae5
2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled item pipelines:
['lianjia_home.pipelines.FilterPipeline', 'lianjia_home.pipelines.CSVPipeline']
2021-05-06 15:17:21 [scrapy.core.engine] INFO: Spider opened
2021-05-06 15:17:21 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-05-06 15:17:21 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023

甲鱼飞鱼

找到错误啦
self.total_page = response.xpath("//div[@class='page-box house-1st-page-box']"
那个house-1st 改成 house-lst 就好了。是l不是1