spider [scrapy.core.scraper] ERROR: Spider error processing <GET https

甲鱼飞鱼

1. [code][code]from scrapy import Request
   
2. from scrapy.spiders import Spider
   
3. from lianjia_home.items import LianjiaHomeItem
   
4. 
   
5. class HomeSpider(Spider):
   
6.     name = 'home'
   
7.     current_page = 1
   
8.     def start_requests(self):
   
9.         url = "https://su.lianjia.com/ershoufang/"
   
10.         yield Request(url)
    
11. 
    
12.     def parse(self, response):
    
13.         #1.提取主页面中的房屋信息
    
14.         list_selector = response.xpath("//li/div[@class='info clear']")
    
15.         for one_selector in list_selector:
    
16.             try:
    
17.                 name = one_selector.xpath("div[@class='address']/"
    
18.                                           "div[@class='houseInfo']"
    
19.                                           "/a/text()").extract_first()
    
20.                 other = one_selector.xpath("div[@class='address']/"
    
21.                                            "div[@class='houseInfo']"
    
22.                                            "/text()").extract_first()
    
23.                 other_list = other.splite("|")
    
24.                 type = other_list[1].strip(" ")
    
25.                 area = other_list[2].strip(" ")
    
26.                 direction = other_list[3].strip(" ")
    
27.                 fitment = other_list[4].strip(" ")
    
28.                 elevator = other_list[5].strip(" ")
    
29. 
    
30.                 price_list = one_selector.xpath("div[@class='priceInfo']//span/text()")
    
31.                 total_price = price_list[0].extract()
    
32.                 unit_price = price_list[1].extract()
    
33.                 item = LianjiaHomeItem()
    
34.                 #将已经获取的字段保存再item对象中
    
35.                 item["name"] = name.strip(" ")
    
36.                 item["type"] = type
    
37.                 item["area"] = area
    
38.                 item["direction"] = direction
    
39.                 item["fitment"] = fitment
    
40.                 item["elevator"] = elevator
    
41.                 item["total_price"] = total_price
    
42.                 item["unit_price"] = unit_price
    
43.                 #2.获取详细页URL
    
44.                 url = one_selector.xpath("div[@class='title']/a/@href").extract_first()
    
45.                 #3.生成详情页的请求对象，参数meta保存房屋部分数据
    
46.                 yield Request(url,
    
47.                               meta={"item": item},
    
48.                               callback=self.property_parse)
    
49.             except:
    
50.                 pass
    
51.         #4.获取下一页URL，并生成Request请求
    
52.         #（1）获取下一页URL。仅在解析第一页时获取总页数的值
    
53.         if self.current_page == 1:
    
54.             self.total_page = response.xpath("//div[@class='page-box house-1st-page-box']"
    
55.                                              "//@page-data").re("\d+")
    
56.             self.total_page = int(self.total_page[0])
    
57.         self.current_page += 1
    
58.         if self.current_page <= self.total_page:
    
59.             next_url = "https://su.lianjia.com/ershoufang/pg%d"%(self.current_page)
    
60.         #根据URL生成Req。使用yield提交给引擎
    
61.             yield Request(next_url)
    
62. 
    
63.     def property_parse(self,response):
    
64.         property = response.xpath("//div[@class='base']/div[@class='content']/u1/li[12]/text()").extract_first()
    
65.         item = response.meta["item"]
    
66.         item["property"] = property
    
67.         yield item
    
68. 
    
69. 
    
70. 
    
71. 
    

复制代码

[/code][/code]

1. 2
   
2. 021-05-06 15:17:21 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: lianjia_home)
   
3. 2021-05-06 15:17:21 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k  25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19041-SP0
   
4. 2021-05-06 15:17:21 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
   
5. 2021-05-06 15:17:21 [scrapy.crawler] INFO: Overridden settings:
   
6. {'BOT_NAME': 'lianjia_home',
   
7. 'NEWSPIDER_MODULE': 'lianjia_home.spiders',
   
8. 'SPIDER_MODULES': ['lianjia_home.spiders']}
   
9. 2021-05-06 15:17:21 [scrapy.extensions.telnet] INFO: Telnet Password: 735a85a99936aae5
   
10. 2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled extensions:
    
11. ['scrapy.extensions.corestats.CoreStats',
    
12. 'scrapy.extensions.telnet.TelnetConsole',
    
13. 'scrapy.extensions.logstats.LogStats']
    
14. 2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled downloader middlewares:
    
15. ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
    
16. 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
    
17. 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
    
18. 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
    
19. 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
    
20. 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
    
21. 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
    
22. 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
    
23. 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
    
24. 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
    
25. 'scrapy.downloadermiddlewares.stats.DownloaderStats']
    
26. 2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled spider middlewares:
    
27. ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
    
28. 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
    
29. 'scrapy.spidermiddlewares.referer.RefererMiddleware',
    
30. 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
    
31. 'scrapy.spidermiddlewares.depth.DepthMiddleware']
    
32. 2021-05-06 15:17:21 [scrapy.middleware] INFO: Enabled item pipelines:
    
33. ['lianjia_home.pipelines.FilterPipeline', 'lianjia_home.pipelines.CSVPipeline']
    
34. 2021-05-06 15:17:21 [scrapy.core.engine] INFO: Spider opened
    
35. 2021-05-06 15:17:21 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    
36. 2021-05-06 15:17:21 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
    

复制代码

甲鱼飞鱼

找到错误啦
self.total_page = response.xpath("//div[@class='page-box house-1st-page-box']"
那个house-1st 改成 house-lst 就好了。是l不是1