Xpath抓取拉勾网信息结果返回空列表问题

zoe_wucs

大四女生自学python，这两天怎么都没解决这个问题，还希望各位坛友能帮我看看
想要抓取拉勾网静态网页每个职位的元素 拉勾网网址

                               
登录/注册后可看大图

上面这一步我搞定了，但是当收集每一个点进去的链接里的“岗位职责”时就总是返回空列表
[url=https://www.lagou.com/jobs/5549207.html]职位的详细链接

1. import requests
   
2. from lxml import etree
   
3. import random
   
4. from time import sleep
   
5. 
   
6. headers={
   
7.         'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
   
8.         'cookie':'WEBTJ-ID=2019-3-12113438-1696ff7666d357-0d40d7cbc13113-4d045769-921600-1696ff7666e5a; user_trace_token=20190312113438-c351e464-4477-11e9-942c-5254005c3644; LGUID=20190312113438-c351e89d-4477-11e9-942c-5254005c3644; TG-TRACK-CODE=index_navigation; index_location_city=%E5%B9%BF%E5%B7%9E; JSESSIONID=ABAAABAAAGGABCB10E773DE2B9DE50F5F48FE257558B7D5; SEARCH_ID=212cb3d9a0784183b3903755db814d0c; _gid=GA1.2.1076558024.1552361681; _gat=1; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1552361680; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1552371832; _ga=GA1.2.1361119337.1552361680; LGSID=20190312142349-663a9322-448f-11e9-943b-5254005c3644; PRE_UTM=; PRE_HOST=; PRE_SITE=https%3A%2F%2Fwww.lagou.com%2F; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fzhaopin%2Ffengkong%2F%3FlabelWords%3Dlabel; LGRID=20190312142349-663a9519-448f-11e9-943b-5254005c3644; _putrc=B50DB7FF004C30B8123F89F2B170EADC; login=true; unick=%E6%8B%89%E5%8B%BE%E7%94%A8%E6%88%B77929; showExpriedIndex=1; showExpriedCompanyHome=1; showExpriedMyPublish=1; hasDeliver=0; gate_login_token=4638d021a035b93a342748ffef9cf3919cd4ddea8b479e78e3848c6514fe7742'
   
9.         }
   
10. 
    
11. for i in range(1,3):
    
12.     sleep(random.randint(3,10))
    
13.     url='https://www.lagou.com/zhaopin/shichangshichangyingxiao1/{}'.format(i)
    
14.     html=requests.get(url,headers=headers)
    
15.     con=etree.HTML(html.text)
    
16.     #抓取每个岗位的详细链接
    
17.     job_link=[i for i in con.xpath('//div[@class="p_top"]/a/@href')]
    
18.     for detaillink in job_link:
    
19.         url1=detaillink
    
20.         html1=requests.get(url=url1,headers=headers)
    
21.         con1=etree.HTML(html1.text)
    
22.         job_name=[qq for qq in con.xpath('//div[@class="job-detail"]/p[1]/text()')]
    
23.         print(job_name)

复制代码

运行出的结果就是一连串的[][][]，希望有大神能帮我看一看

wiselin

可能那些数据都是js生成的，所以我用的是splash渲染后，再用scrapy抓取的方式，具体流程如下：
1.安装scrapy跟splash，教程论坛可以找到
2.建好scrapy项目，然后在setting添加

1. # 渲染服务的url
   
2. SPLASH_URL='http://192.168.99.100:8050'
   
3. #下载器中间件
   
4. DOWNLOADER_MIDDLEWARES = {
   
5.         'scrapy_splash.SplashCookiesMiddleware':723,
   
6.         'scrapy_splash.SplashMiddleware':725,
   
7.         'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware':800
   
8. }
   
9. 
   
10. SPIDER_MIDDLEWARES ={
    
11.         'scrapy_splash.SplashDeduplicateArgsMiddleware':100
    
12. }
    
13. # 去重过滤器
    
14. DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
    
15. 
    
16. # 使用Splash的Http缓存
    
17. HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

复制代码

然后spider的代码如下：

1. # -*- coding: utf-8 -*-
   
2. 
   
3. import scrapy
   
4. from scrapy_splash import SplashRequest
   
5. from scrapy_splash import SplashMiddleware
   
6. from scrapy.selector import Selector
   
7. 
   
8. class LagouJobSpider(scrapy.Spider):
   
9.     name = 'lagou_job'
   
10.     allowed_domains = ['www.lagou.com']
    
11.     urls = ['https://www.lagou.com/jobs/5549207.html']
    
12.     header = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER"}
    
13. 
    
14.     def start_requests(self):
    
15.         yield SplashRequest(self.urls[0],headers=self.header,callback=self.parse)
    
16. 
    
17.     def parse(self, response):
    
18.         print(response.xpath('//*[@id="job_detail"]/dd[2]/div/p[1]').extract_first())
    

复制代码

抓取完的结果是这样的，你再去掉那些标签就行