[已解决]scrapy

哈岁NB

大佬们，为什么这个一运行，用selenium启动浏览器是一闪而过呢
爬虫文件

import scrapy
from selenium import webdriver

class XinwenSpider(scrapy.Spider):
    name = 'xinwen'
    #allowed_domains = ['www.xxx.com']
    start_urls = ['https://news.163.com/']
    #创建浏览器对象，把浏览器作为类的成员
    bro = webdriver.Chrome('D:/技能/chromedriver.exe')
    model_urls =[]
    def parse(self, response):
        model_index = [2,3,5,6]
        li_list = response.xpath('//*[@id="index2016_wrap"]/div[3]/div[2]/div[2]/div[2]/div/ul/li')
        for index in model_index:
            model_url = li_list[index].xpath('./a/@href').extract_first()

            self.model_urls.append(model_url)
            print(self.model_urls)
            for url in self.model_urls:
                yield scrapy.Request(url=url,callback=self.detail_parse)
    #解析详情页的新闻标题和url
    def detail_parse(self,response):
        page = response.xpath('/html/body/div/div[3]/div[3]/div[1]/div[1]/div/ul/li/div/div[1]/div/div[1]/h3/a/@href').extract_first()
        print(page)

    def closed(self,spider):
        self.bro.quit()

中间件

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

#scrapy封装响应对象的类
from scrapy.http import HtmlResponse
from scrapy import signals
from time import sleep

# useful for handling different item types with a single interface
from itemadapter import is_item, ItemAdapter


class WanyiDownloaderMiddleware:


    def process_request(self, request, spider):

        return None

    def process_response(self, request, response, spider):
        #如何筛选指定的url
         #可以先找出指定四个板块的请求对象，然后根据请求对象定位
         #可以根据四个板块的url定位到四个板块的请求对象
        model_urls = spider.model_urls
        if request.url in model_urls:
            bro = spider.bro#从爬虫类获取创建好的浏览器对象
            bro.get(request.url)
            sleep(1)
            page_text = bro.page_source
            #说明该request就是指定响应对象的请求对象
            #此处的response就是指定板块的响应对象
            response = HtmlResponse(url=request.url,request=request,
                                    encoding='utf-8',body=page_text)#body就是响应对象的响应对象，就是要篡改的数据
            return response
        else:
            return response

    def process_exception(self, request, exception, spider):
        pass

selenium不用在scrapy就可以正常操作，一和scrapy结合就闪退，也没有发起请求

最佳答案

月排行榜 / 总排行榜

isdkz

2023-3-1 15:28:43

哈岁NB 发表于 2023-3-1 15:26
这是报错信息

没有报错呀，你说的闪退可能是因为它运行结束了

跳转到最佳答案楼层

isdkz

可能是哪里出错了，看一下报错信息

哈岁NB

isdkz 发表于 2023-3-1 14:37
可能是哪里出错了，看一下报错信息

DevTools listening on ws://127.0.0.1:1286/devtools/browser/f22e9ec7-e022-4014-9d24-354ea870b413
2023-03-01 15:23:15 [scrapy.utils.log] INFO: Scrapy 2.7.1 started (bot: wanyi)
2023-03-01 15:23:15 [scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.12, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 20.3.0, Python 3.8.1 (tags/
v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 22.1.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 38.0.4, Platform Windows-10-10.0.19041
-SP0
2023-03-01 15:23:15 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'wanyi',
 'NEWSPIDER_MODULE': 'wanyi.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['wanyi.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
               'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.109 '
               'Safari/537.36'}
2023-03-01 15:23:15 [asyncio] DEBUG: Using selector: SelectSelector
2023-03-01 15:23:15 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2023-03-01 15:23:15 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
2023-03-01 15:23:15 [scrapy.extensions.telnet] INFO: Telnet Password: 3363e4532231a8d7
2023-03-01 15:23:16 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
[1536:4164:0301/152317.197:ERROR:device_event_log_impl.cc(215)] [15:23:17.185] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: 连
到系统上的设备没有发挥作用。 (0x1F)
[1536:4164:0301/152317.229:ERROR:device_event_log_impl.cc(215)] [15:23:17.222] Bluetooth: bluetooth_adapter_winrt.cc:1205 Getting Radio failed. Chrome will be unabl
e to change the power state by itself.
[1536:4164:0301/152317.260:ERROR:device_event_log_impl.cc(215)] [15:23:17.247] Bluetooth: bluetooth_adapter_winrt.cc:1283 OnPoweredRadioAdded(), Number of Powered R
adios: 1
[1536:4164:0301/152317.260:ERROR:device_event_log_impl.cc(215)] [15:23:17.248] Bluetooth: bluetooth_adapter_winrt.cc:1298 OnPoweredRadiosEnumerated(), Number of Pow
ered Radios: 1
2023-03-01 15:23:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-03-01 15:23:18 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-03-01 15:23:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-03-01 15:23:18 [scrapy.core.engine] INFO: Spider opened
2023-03-01 15:23:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-03-01 15:23:18 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-03-01 15:23:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.163.com/> (referer: None)
['https://news.163.com/world/']
['https://news.163.com/world/', 'https://data.163.com/special/datablog/']
2023-03-01 15:23:19 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://news.163.com/world/> - no more duplicates will be shown (see DUPEFILTER_DEB
UG to show all duplicates)
['https://news.163.com/world/', 'https://data.163.com/special/datablog/', 'https://news.163.com/air/']
['https://news.163.com/world/', 'https://data.163.com/special/datablog/', 'https://news.163.com/air/', 'https://news.163.com/college']
2023-03-01 15:23:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.163.com/world/> (referer: https://news.163.com/)
2023-03-01 15:23:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.163.com/college> (referer: https://news.163.com/)
2023-03-01 15:23:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://news.163.com/air/> (referer: https://news.163.com/)
2023-03-01 15:23:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://data.163.com/special/datablog/> (referer: https://news.163.com/)
None
None
None
None
2023-03-01 15:23:19 [scrapy.core.engine] INFO: Closing spider (finished)
2023-03-01 15:23:19 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://localhost:1282/session/9842f6fffdfdd79191970c39703d24ea {}
2023-03-01 15:23:19 [urllib3.connectionpool] DEBUG: http://localhost:1282 "DELETE /session/9842f6fffdfdd79191970c39703d24ea HTTP/1.1" 200 14
 'scheduler/enqueued': 5,
 'scheduler/enqueued/memory': 5,
 'start_time': datetime.datetime(2023, 3, 1, 7, 23, 18, 813688)}
2023-03-01 15:23:21 [scrapy.core.engine] INFO: Spider closed (finished)

哈岁NB

isdkz 发表于 2023-3-1 14:37
可能是哪里出错了，看一下报错信息

这是报错信息

wiselin

是不是你driver那个目录有中文名？

isdkz

哈岁NB 发表于 2023-3-1 15:26
这是报错信息

没有报错呀，你说的闪退可能是因为它运行结束了

哈岁NB

wiselin 发表于 2023-3-1 15:27
是不是你driver那个目录有中文名？

不是呀，同样的路径不用scrapy就能运行

哈岁NB

isdkz 发表于 2023-3-1 15:28
没有报错呀，你说的闪退可能是因为它运行结束了

那他也没有怎么没对网页发起请求，返回的直接是none呢

哈岁NB

isdkz 发表于 2023-3-1 15:28
没有报错呀，你说的闪退可能是因为它运行结束了

就显示个这，就退了

哈岁NB

isdkz 发表于 2023-3-1 15:28
没有报错呀，你说的闪退可能是因为它运行结束了

找到原因了，没开启中间件 感谢感谢

哈岁NB

wiselin 发表于 2023-3-1 15:27
是不是你driver那个目录有中文名？

找到原因了，没开启中间件，感谢感谢