设为首页收藏本站
切换到窄版

鱼C论坛

 找回密码
 立即注册
鱼C论坛»论坛 › 编程语言专区 › Python交流 › scrapy爬取链家二手房信息返回403
返回列表 发新帖
查看: 1438|回复: 2

[已解决]scrapy爬取链家二手房信息返回403

[复制链接]
Jinvador
发表于 2022-3-3 21:05:10 | 显示全部楼层 |阅读模式

马上注册，结交更多好友，享用更多功能^_^

您需要 登录 才可以下载或查看，没有账号？立即注册

x
最近在看一本书，上面有一个爬虫案例始终完成不了，爬取的是链家网的上海黄埔二手房信息https://sh.lianjia.com/ershoufang/huangpu/

爬虫返回的是
  1. 2022-03-03 20:44:40 [houseSpider] INFO: Spider opened: houseSpider
  2. 2022-03-03 20:44:40 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
  3. 2022-03-03 20:44:40 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg1/> (referer: None)
  4. 2022-03-03 20:44:40 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg1/>: HTTP status code is not handled or not allowed
  5. 2022-03-03 20:44:44 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg2/> (referer: None)
  6. 2022-03-03 20:44:44 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg2/>: HTTP status code is not handled or not allowed
  7. 2022-03-03 20:44:48 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg3/> (referer: None)
  8. 2022-03-03 20:44:49 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg3/>: HTTP status code is not handled or not allowed
  9. 2022-03-03 20:44:51 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg4/> (referer: None)
  10. 2022-03-03 20:44:51 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg4/>: HTTP status code is not handled or not allowed
  11. 2022-03-03 20:44:55 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg5/> (referer: None)
  12. 2022-03-03 20:44:55 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg5/>: HTTP status code is not handled or not allowed
  13. 2022-03-03 20:44:59 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg6/> (referer: None)
  14. 2022-03-03 20:44:59 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg6/>: HTTP status code is not handled or not allowed
  15. 2022-03-03 20:45:04 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg7/> (referer: None)
  16. 2022-03-03 20:45:04 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg7/>: HTTP status code is not handled or not allowed
  17. 2022-03-03 20:45:07 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg8/> (referer: None)
  18. 2022-03-03 20:45:07 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg8/>: HTTP status code is not handled or not allowed
  19. 2022-03-03 20:45:10 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg9/> (referer: None)
  20. 2022-03-03 20:45:10 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg9/>: HTTP status code is not handled or not allowed
  21. 2022-03-03 20:45:15 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://https//sh,lianjia.com/ershoufang/huangpu/pg10/> (referer: None)
  22. 2022-03-03 20:45:15 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://https//sh,lianjia.com/ershoufang/huangpu/pg10/>: HTTP status code is not handled or not allowed
  23. 2022-03-03 20:45:15 [scrapy.core.engine] INFO: Closing spider (finished)
复制代码



我己经在setting.py里面加了UserAgent和关闭了ROBOTSTXT_OBEY和COOKIES_ENABLE,
还加上了延迟DOWNLOAD_DELAY=3,但是还是只返回403。

Spider.py, items.py, pipelines都是抄书本上的，应该是没问题的。

代码的缩进没问题的，只是复制上来的时候用编辑器它就自动没了缩进
setting部分:
  1. BOT_NAME = 'houseScrapy'

  3. SPIDER_MODULES = ['houseScrapy.spiders']
  4. NEWSPIDER_MODULE = 'houseScrapy.spiders'

  6. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36

  8. DOWNLOADER_MIDDLEWARES = {
  9.    'houseScrapy.middlewares.HousescrapyDownloaderMiddleware': 543,
  10. }
  11. ITEM_PIPELINES = {
  12.    'houseScrapy.pipelines.HousescrapyPipeline': 300,
  13. }
  14. COOKIES_ENABLED = False
  15. DOWNLOAD_DELAY = 3
复制代码

items.py:
  1. import scrapy


  4. class HousescrapyItem(scrapy.Item):
  5.     # define the fields for your item here like:
  6.     # name = scrapy.Field()
  7.     title = scrapy.Field()
  8.     price = scrapy.Field()
  9.     pricePerSqM2 = scrapy.Field()
  10.     community = scrapy.Field()
  11.     address = scrapy.Field()
  12.     houseType = scrapy.Field()
  13.     area = scrapy.Field()
  14.     orientation = scrapy.Field()
  15.     repairStatus = scrapy.Field()
  16.     floor = scrapy.Field()
  17.     noticeNum = scrapy.Field()
  18.     mortgage = scrapy.Field()
  19.     transProperty = scrapy.Field()
复制代码
houseSpider.py
  1. import scrapy
  2. from scrapy.spiders import CrawlSpider, Rule
  3. from scrapy.linkextractors import LinkExtractor
  4. from houseScrapy.items import HousescrapyItem

  6. class HousespiderSpider(scrapy.Spider):
  7.     name = 'houseSpider'
  8.     allowed_domains = ['sh,lianjia.com/ershoufang/']
  9.     start_urls = ['http://https://sh,lianjia.com/ershoufang/huangpu/pg{}/'.format(i) for i in range(1, 11)]

  11.     def parse(self, response):
  12.     houses = response.xpath("//ul[@class='sellListContent']/li/div[@class='info clear']")
  13.     for house in houses:
  14.         item = HousescrapyItem()
复制代码
pipelines.py:
  1. # Define your item pipelines here
  2. #
  3. # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  4. # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


  7. # useful for handling different item types with a single interface
  8. from itemadapter import ItemAdapter
  9. import codecs
  10. import json
  11. import os


  14. class HousescrapyPipeline(object):
  15.     def __init__(self):
  16.         filename = 'house.json'
  17.         if (os.path.exists(filename)):
  18.     os.remove(filename)
  19.     self.file = codecs.open(filename, 'w', encoding="utf-8")
  20.    
  21.     def process_item(self, item, spider):
  22.     lines = json.dumps(dict(item), ensure_ascii=False) + "\n"
  23.     self.file.write(lines)
  24.     return item

  26.     def spider_closed(self, spider):
  27.     self.file.close()
复制代码


我查了链家的robots.txt
  1. User-agent: Baiduspider
  2. Allow: *?utm_source=office*

  4. User-agent: *
  5. sitemap: https://sh.lianjia.com/sitemap/sh_index.xml
  6. Disallow: /rs
  7. Disallow: /huxingtu/p
  8. Disallow: /user
  9. Disallow: /login
  10. Disallow: /userinfo
  11. Disallow: *?project_name=*
  12. Disallow: *?id=*
  13. Disallow: *?house_codes=*
  14. Disallow: *?sug=*
复制代码
最后放上爬虫文件夹的图片

                               
登录/注册后可看大图


                               
登录/注册后可看大图



最佳答案
月排行榜 / 总排行榜
isdkz
2022-3-3 23:47:46
你的 url 写错了，

houseSpider.py
  1. import scrapy
  2. from scrapy.spiders import CrawlSpider, Rule
  3. from scrapy.linkextractors import LinkExtractor
  4. from houseScrapy.items import HousescrapyItem

  6. class HousespiderSpider(scrapy.Spider):
  7.     name = 'houseSpider'
  8.     allowed_domains = ['sh.lianjia.com/ershoufang/']                  # sh, 改成 sh.
  9.     start_urls = ['https://sh.lianjia.com/ershoufang/huangpu/pg{}/'.format(i) for i in range(1, 11)]       # http://https://sh, 改成 https://sh.

  11.     def parse(self, response):
  12.     houses = response.xpath("//ul[@class='sellListContent']/li/div[@class='info clear']")
  13.     for house in houses:
  14.         item = HousescrapyItem()
复制代码
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

isdkz
发表于 2022-3-3 23:47:46 | 显示全部楼层    本楼为最佳答案   
你的 url 写错了，

houseSpider.py
  1. import scrapy
  2. from scrapy.spiders import CrawlSpider, Rule
  3. from scrapy.linkextractors import LinkExtractor
  4. from houseScrapy.items import HousescrapyItem

  6. class HousespiderSpider(scrapy.Spider):
  7.     name = 'houseSpider'
  8.     allowed_domains = ['sh.lianjia.com/ershoufang/']                  # sh, 改成 sh.
  9.     start_urls = ['https://sh.lianjia.com/ershoufang/huangpu/pg{}/'.format(i) for i in range(1, 11)]       # http://https://sh, 改成 https://sh.

  11.     def parse(self, response):
  12.     houses = response.xpath("//ul[@class='sellListContent']/li/div[@class='info clear']")
  13.     for house in houses:
  14.         item = HousescrapyItem()
复制代码
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

Jinvador
 楼主| 发表于 2022-3-4 10:30:24 | 显示全部楼层
isdkz 发表于 2022-3-3 23:47
你的 url 写错了，

houseSpider.py

非常感谢！！！
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

返回列表 发新帖
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

小黑屋|手机版|Archiver|鱼C工作室 ( 粤ICP备18085999号-1 | 粤公网安备 44051102000585号)

GMT+8, 2025-4-30 05:26

Powered by Discuz! X3.4

© 2001-2023 Discuz! Team.

快速回复 返回顶部 返回列表