设为首页收藏本站
切换到窄版

鱼C论坛

 找回密码
 立即注册
鱼C论坛»论坛 › 编程语言专区 › Python交流 › 爬取json
返回列表 发新帖
查看: 2642|回复: 10

[已解决]爬取json

[复制链接]
哈岁NB
发表于 2023-2-11 11:21:06 | 显示全部楼层 |阅读模式

马上注册，结交更多好友，享用更多功能^_^

您需要 登录 才可以下载或查看，没有账号？立即注册

x
xpath怎样写可以把__INITIAL_STATE__后面的json字符串提取出来呢
  1. from selenium import webdriver
  2. from time import sleep
  3. import requests
  4. from lxml import html
  5. etree = html.etree

  7. import json
  8. from selenium.webdriver.common.by import By
  9. bro = webdriver.Chrome('D:/技能/chromedriver.exe')

  11. bro.get('https://xiaoyuan.zhaopin.com/job/CC407288330J40383568908')
  12. sleep(2)

  14. cookies = bro.get_cookies()
  15. print(cookies)
  16. bro.quit()

  18. #解析cookie
  19. dic = {}
  20. for cookie in cookies:
  21.     dic[cookie['name']] = cookie['value']


  24. #解析页面源码
  25. url = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=12&pageSize=30&_v=0.43957010&at=8d9987f50aed40bc8d1362e9c44a7fba&rt=ed9c026545294384a20a4473e1e2ecd3&x-zp-page-request-id=0933f66d64684fd6b0bc0756ed6791b6-1675650906506-860845&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'

  27. header = {
  28.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
  29. }
  30. url1 = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=&pageSize=30&_v=0.06047107&at=9790578095794e1b9cc693485ef05237&rt=6448b0d50c2d460eb823575593f5a909&cityId=&jobTypeId=&jobSource=&industryId=&companyTypeId=&dateSearchTypeId=&x-zp-page-request-id=fcf1dcda72444dc6b8a17609bdb3a02f-1676083831807-687308&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'
  31. response = requests.get(url=url1,headers=header,cookies=dic).json()#json后要加括号
  32. list = response['data']['data']['list']
  33. for i in list:
  34.     name = i['name']
  35.     number = i['number']
  36.     job_url = 'https://xiaoyuan.zhaopin.com/job/' + number

  38.     page = requests.get(url=job_url, headers=header,cookies=dic).text
  39.     tree = etree.HTML(page)
  40.     #解析详情页数据
  41.     imformation = tree.xpath('/html/body/script[8]/text()')
  42.     print(imformation)
  43.     break
复制代码
我是这样写的，但输出的是一个空列表
最佳答案
月排行榜 / 总排行榜
isdkz
2023-2-11 12:38:22
哈岁NB 发表于 2023-2-11 12:31
那该怎么才能把__INITIAL_STATE__后面的json字符串单独取出来呢，就是只要json字符串，不要__INITIAL_STA ...


  1. from selenium import webdriver
  2. from time import sleep
  3. import requests
  4. from lxml import html
  5. etree = html.etree

  7. import json
  8. from selenium.webdriver.common.by import By
  9. bro = webdriver.Chrome('D:/技能/chromedriver.exe')

  11. bro.get('https://xiaoyuan.zhaopin.com/job/CC407288330J40383568908')
  12. sleep(2)

  14. cookies = bro.get_cookies()
  15. print(cookies)
  16. bro.quit()

  18. #解析cookie
  19. dic = {}
  20. for cookie in cookies:
  21.     dic[cookie['name']] = cookie['value']


  24. #解析页面源码
  25. url = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=12&pageSize=30&_v=0.43957010&at=8d9987f50aed40bc8d1362e9c44a7fba&rt=ed9c026545294384a20a4473e1e2ecd3&x-zp-page-request-id=0933f66d64684fd6b0bc0756ed6791b6-1675650906506-860845&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'

  27. header = {
  28.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
  29. }
  30. url1 = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=&pageSize=30&_v=0.06047107&at=9790578095794e1b9cc693485ef05237&rt=6448b0d50c2d460eb823575593f5a909&cityId=&jobTypeId=&jobSource=&industryId=&companyTypeId=&dateSearchTypeId=&x-zp-page-request-id=fcf1dcda72444dc6b8a17609bdb3a02f-1676083831807-687308&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'
  31. response = requests.get(url=url1,headers=header,cookies=dic).json()#json后要加括号
  32. list = response['data']['data']['list']
  33. for i in list:
  34.     name = i['name']
  35.     number = i['number']
  36.     job_url = 'https://xiaoyuan.zhaopin.com/job/' + number

  38.     page = requests.get(url=job_url, headers=header,cookies=dic).text
  39.     tree = etree.HTML(page)
  40.     #解析详情页数据
  41.     information = tree.xpath('/html/body/script[6]/text()')[0].removeprefix('__INITIAL_STATE__=')                                                      # 改了这里
  42.     print(information)
复制代码
屏幕截图(18)_LI.jpg
小甲鱼最新课程 -> https://ilovefishc.com
回复

使用道具 举报

isdkz
发表于 2023-2-11 11:39:11 | 显示全部楼层
  1. from selenium import webdriver
  2. from time import sleep
  3. import requests
  4. from lxml import html
  5. etree = html.etree

  7. import json
  8. from selenium.webdriver.common.by import By
  9. bro = webdriver.Chrome('D:/技能/chromedriver.exe')

  11. bro.get('https://xiaoyuan.zhaopin.com/job/CC407288330J40383568908')
  12. sleep(2)

  14. cookies = bro.get_cookies()
  15. print(cookies)
  16. bro.quit()

  18. #解析cookie
  19. dic = {}
  20. for cookie in cookies:
  21.     dic[cookie['name']] = cookie['value']


  24. #解析页面源码
  25. url = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=12&pageSize=30&_v=0.43957010&at=8d9987f50aed40bc8d1362e9c44a7fba&rt=ed9c026545294384a20a4473e1e2ecd3&x-zp-page-request-id=0933f66d64684fd6b0bc0756ed6791b6-1675650906506-860845&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'

  27. header = {
  28.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
  29. }
  30. url1 = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=&pageSize=30&_v=0.06047107&at=9790578095794e1b9cc693485ef05237&rt=6448b0d50c2d460eb823575593f5a909&cityId=&jobTypeId=&jobSource=&industryId=&companyTypeId=&dateSearchTypeId=&x-zp-page-request-id=fcf1dcda72444dc6b8a17609bdb3a02f-1676083831807-687308&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'
  31. response = requests.get(url=url1,headers=header,cookies=dic).json()#json后要加括号
  32. list = response['data']['data']['list']
  33. for i in list:
  34.     name = i['name']
  35.     number = i['number']
  36.     job_url = 'https://xiaoyuan.zhaopin.com/job/' + number

  38.     page = requests.get(url=job_url, headers=header,cookies=dic).text
  39.     tree = etree.HTML(page)
  40.     #解析详情页数据
  41.     imformation = tree.xpath('/html/body/script[6]/text()')         # 这里应该是 6
  42.     print(imformation)
  43.     break
复制代码
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

哈岁NB
 楼主| 发表于 2023-2-11 12:31:39 | 显示全部楼层
isdkz 发表于 2023-2-11 11:39

那该怎么才能把__INITIAL_STATE__后面的json字符串单独取出来呢，就是只要json字符串，不要__INITIAL_STATE__=
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

isdkz
发表于 2023-2-11 12:38:22 | 显示全部楼层    本楼为最佳答案   
哈岁NB 发表于 2023-2-11 12:31
那该怎么才能把__INITIAL_STATE__后面的json字符串单独取出来呢，就是只要json字符串，不要__INITIAL_STA ...


  1. from selenium import webdriver
  2. from time import sleep
  3. import requests
  4. from lxml import html
  5. etree = html.etree

  7. import json
  8. from selenium.webdriver.common.by import By
  9. bro = webdriver.Chrome('D:/技能/chromedriver.exe')

  11. bro.get('https://xiaoyuan.zhaopin.com/job/CC407288330J40383568908')
  12. sleep(2)

  14. cookies = bro.get_cookies()
  15. print(cookies)
  16. bro.quit()

  18. #解析cookie
  19. dic = {}
  20. for cookie in cookies:
  21.     dic[cookie['name']] = cookie['value']


  24. #解析页面源码
  25. url = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=12&pageSize=30&_v=0.43957010&at=8d9987f50aed40bc8d1362e9c44a7fba&rt=ed9c026545294384a20a4473e1e2ecd3&x-zp-page-request-id=0933f66d64684fd6b0bc0756ed6791b6-1675650906506-860845&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'

  27. header = {
  28.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
  29. }
  30. url1 = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=&pageSize=30&_v=0.06047107&at=9790578095794e1b9cc693485ef05237&rt=6448b0d50c2d460eb823575593f5a909&cityId=&jobTypeId=&jobSource=&industryId=&companyTypeId=&dateSearchTypeId=&x-zp-page-request-id=fcf1dcda72444dc6b8a17609bdb3a02f-1676083831807-687308&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'
  31. response = requests.get(url=url1,headers=header,cookies=dic).json()#json后要加括号
  32. list = response['data']['data']['list']
  33. for i in list:
  34.     name = i['name']
  35.     number = i['number']
  36.     job_url = 'https://xiaoyuan.zhaopin.com/job/' + number

  38.     page = requests.get(url=job_url, headers=header,cookies=dic).text
  39.     tree = etree.HTML(page)
  40.     #解析详情页数据
  41.     information = tree.xpath('/html/body/script[6]/text()')[0].removeprefix('__INITIAL_STATE__=')                                                      # 改了这里
  42.     print(information)
复制代码
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

哈岁NB
 楼主| 发表于 2023-2-11 12:43:17 | 显示全部楼层
isdkz 发表于 2023-2-11 12:38

好的，感谢感谢
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

哈岁NB
 楼主| 发表于 2023-2-11 15:57:03 | 显示全部楼层
isdkz 发表于 2023-2-11 12:38

运行怎么报了'lxml.etree._ElementUnicodeResult' object has no attribute 'removeprefix'这个错误，这是为什么 呀，上网查也没有查到
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

isdkz
发表于 2023-2-11 16:16:09 | 显示全部楼层
哈岁NB 发表于 2023-2-11 15:57
运行怎么报了'lxml.etree._ElementUnicodeResult' object has no attribute 'removeprefix'这个错误，这 ...

代码有改过吗？我这里运行没有问题，你那个报错是因为只有字符串才有 removeprefix 方法，这说明你获取到的不是字符串
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

哈岁NB
 楼主| 发表于 2023-2-11 16:26:11 | 显示全部楼层
本帖最后由 哈岁NB 于 2023-2-11 16:29 编辑
  1. from selenium import webdriver
  2. from time import sleep
  3. import requests
  4. from lxml import html
  5. etree = html.etree

  7. bro = webdriver.Chrome('D:/技能/chromedriver.exe')

  9. bro.get('https://xiaoyuan.zhaopin.com/job/CC407288330J40383568908')
  10. sleep(2)

  12. cookies = bro.get_cookies()
  13. print(cookies)
  14. bro.quit()

  16. #解析cookie
  17. dic = {}
  18. for cookie in cookies:
  19.     dic[cookie['name']] = cookie['value']


  22. #解析页面源码
  23. url = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=12&pageSize=30&_v=0.43957010&at=8d9987f50aed40bc8d1362e9c44a7fba&rt=ed9c026545294384a20a4473e1e2ecd3&x-zp-page-request-id=0933f66d64684fd6b0bc0756ed6791b6-1675650906506-860845&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'

  25. header = {
  26.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
  27. }
  28. url1 = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=&pageSize=30&_v=0.06047107&at=9790578095794e1b9cc693485ef05237&rt=6448b0d50c2d460eb823575593f5a909&cityId=&jobTypeId=&jobSource=&industryId=&companyTypeId=&dateSearchTypeId=&x-zp-page-request-id=fcf1dcda72444dc6b8a17609bdb3a02f-1676083831807-687308&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'
  29. response = requests.get(url=url1,headers=header,cookies=dic).json()#json后要加括号
  30. list = response['data']['data']['list']
  31. for i in list:
  32.     name = i['name']
  33.     number = i['number']
  34.     job_url = 'https://xiaoyuan.zhaopin.com/job/' + number

  36.     page = requests.get(url=job_url, headers=header,cookies=dic).text
  37.     tree = etree.HTML(page)
  38.     #解析详情页数据
  39.     information = tree.xpath('/html/body/script[6]/text()')[0].removeprefix('__INITIAL_STATE__=')
  40.     print(information)
复制代码
isdkz 发表于 2023-2-11 16:16
代码有改过吗？我这里运行没有问题，你那个报错是因为只有字符串才有 removeprefix 方法，这说明你获取到 ...


没改过，返回的类型是<class 'lxml.etree._ElementUnicodeResult'>这个
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

哈岁NB
 楼主| 发表于 2023-2-11 16:36:37 | 显示全部楼层
isdkz 发表于 2023-2-11 16:16
代码有改过吗？我这里运行没有问题，你那个报错是因为只有字符串才有 removeprefix 方法，这说明你获取到 ...

好像是我的python3.9才有，我用的3.8
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

isdkz
发表于 2023-2-11 17:21:14 | 显示全部楼层
哈岁NB 发表于 2023-2-11 16:36
好像是我的python3.9才有，我用的3.8

倒是忘了这茬了，removeprefix 这个方法是后面的版本才有的，如果不能用这个方法就直接用索引吧
  1. from selenium import webdriver
  2. from time import sleep
  3. import requests
  4. from lxml import html
  5. etree = html.etree

  7. bro = webdriver.Chrome('D:/技能/chromedriver.exe')

  9. bro.get('https://xiaoyuan.zhaopin.com/job/CC407288330J40383568908')
  10. sleep(2)

  12. cookies = bro.get_cookies()
  13. print(cookies)
  14. bro.quit()

  16. #解析cookie
  17. dic = {}
  18. for cookie in cookies:
  19.     dic[cookie['name']] = cookie['value']


  22. #解析页面源码
  23. url = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=12&pageSize=30&_v=0.43957010&at=8d9987f50aed40bc8d1362e9c44a7fba&rt=ed9c026545294384a20a4473e1e2ecd3&x-zp-page-request-id=0933f66d64684fd6b0bc0756ed6791b6-1675650906506-860845&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'

  25. header = {
  26.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
  27. }
  28. url1 = 'https://xiaoyuan.zhaopin.com/api/sou?S_SOU_FULL_INDEX=java&S_SOU_POSITION_SOURCE_TYPE=&pageIndex=1&S_SOU_POSITION_TYPE=2&S_SOU_WORK_CITY=&S_SOU_JD_INDUSTRY_LEVEL=&S_SOU_COMPANY_TYPE=&S_SOU_REFRESH_DATE=&order=&pageSize=30&_v=0.06047107&at=9790578095794e1b9cc693485ef05237&rt=6448b0d50c2d460eb823575593f5a909&cityId=&jobTypeId=&jobSource=&industryId=&companyTypeId=&dateSearchTypeId=&x-zp-page-request-id=fcf1dcda72444dc6b8a17609bdb3a02f-1676083831807-687308&x-zp-client-id=b242c663-f23a-4571-aca7-de919a057afe'
  29. response = requests.get(url=url1,headers=header,cookies=dic).json()#json后要加括号
  30. list = response['data']['data']['list']
  31. for i in list:
  32.     name = i['name']
  33.     number = i['number']
  34.     job_url = 'https://xiaoyuan.zhaopin.com/job/' + number

  36.     page = requests.get(url=job_url, headers=header,cookies=dic).text
  37.     tree = etree.HTML(page)
  38.     #解析详情页数据
  39.     information = tree.xpath('/html/body/script[6]/text()')[0][18:]
  40.     print(information)
复制代码
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

哈岁NB
 楼主| 发表于 2023-2-11 19:10:41 | 显示全部楼层
isdkz 发表于 2023-2-11 17:21
倒是忘了这茬了，removeprefix 这个方法是后面的版本才有的，如果不能用这个方法就直接用索引吧

好的，感谢感谢
小甲鱼最新课程 -> https://ilovefishc.com
回复 支持 反对

使用道具 举报

返回列表 发新帖
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

小黑屋|手机版|Archiver|鱼C工作室 ( 粤ICP备18085999号-1 | 粤公网安备 44051102000585号)

GMT+8, 2025-4-25 01:39

Powered by Discuz! X3.4

© 2001-2023 Discuz! Team.

快速回复 返回顶部 返回列表