爬虫中网页

贪睡的苏简 · 发表于 2021-4-9 13:09:27

马上注册，结交更多好友，享用更多功能^_^

您需要登录才可以下载或查看，没有账号？立即注册

x

当一个网页的源代码被隐藏了，
url=“百度的网址”
req=urllib.request.Request(url,headers=henders)
response=urllib.request.openurl(req)
print(response.read())
代码大致好像是这样（手机敲的，有些单词可能是错的，但首字母一定对）
打印出来的结果，是body内容好像被隐藏了，想问问怎么破解。
就是打印出网页内的源代码

贪睡的苏简 · 发表于 2021-4-9 16:16:41

import urllib.request
url = r"http://www.baidu.com"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0'}
req = urllib.request.Request(url,headers=headers)
response = urllib.request.urlopen(req)
print(response.read().decode("utf-8"))

这个代码，为什么会得到这个结果
D:\BianCheng\Python\python.exe D:/BianCheng/Pythonwork/NextPython/重塑爬虫/Text/text.py
<html>
<head>
<script language="javascript">setTimeout("location.replace(location.href.split(\"#\")[0])",1000);</script>
</head>
<iframe src="http://10.30.1.30:89/flashredir.html" frameborder=0></iframe>
</html>
怎么才可以得到真正的源代码啊

suchocolate · 发表于 2021-4-10 13:19:33

贪睡的苏简发表于 2021-4-9 16:16
这个代码，为什么会得到这个结果
D:\BianCheng\Python\python.exe D:/BianCheng/Pythonwork/NextPython/ ...

用你的代码能拿到html。
你想爬什么具体的数据？

贪睡的苏简 · 发表于 2021-4-10 16:09:55

suchocolate 发表于 2021-4-10 13:19
用你的代码能拿到html。
你想爬什么具体的数据？

这个难说，例如这里的城市和工资被隐藏了，我就爬不到

https://averagewage.xinchou.com/Ranking/cities/2020

贪睡的苏简 · 发表于 2021-4-10 16:11:15

本帖最后由贪睡的苏简于 2021-4-10 16:16 编辑

工资

<g data-z-index="6" class="highcharts-data-labels highcharts-series-0 highcharts-bar-series highcharts-tracker" transform="translate(80,71) scale(1 1)" opacity="1"><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(741,-3)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round" style="">9603.0 cny</tspan><tspan x="5" y="18">9603.0 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(666,17)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">8633.8 cny</tspan><tspan x="5" y="18">8633.8 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(594,36)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">7699.5 cny</tspan><tspan x="5" y="18">7699.5 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(592,55)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">7671.5 cny</tspan><tspan x="5" y="18">7671.5 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(536,75)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6939.7 cny</tspan><tspan x="5" y="18">6939.7 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(532,94)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6891.2 cny</tspan><tspan x="5" y="18">6891.2 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(531,114)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6885.0 cny</tspan><tspan x="5" y="18">6885.0 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(523,133)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6771.1 cny</tspan><tspan x="5" y="18">6771.1 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(519,152)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6723.2 cny</tspan><tspan x="5" y="18">6723.2 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(514,172)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6658.1 cny</tspan><tspan x="5" y="18">6658.1 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(509,191)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6599.2 cny</tspan><tspan x="5" y="18">6599.2 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(507,211)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6562.8 cny</tspan><tspan x="5" y="18">6562.8 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(503,230)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6522.7 cny</tspan><tspan x="5" y="18">6522.7 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(502,249)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6509.5 cny</tspan><tspan x="5" y="18">6509.5 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(495,269)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6414.3 cny</tspan><tspan x="5" y="18">6414.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(494,288)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6396.9 cny</tspan><tspan x="5" y="18">6396.9 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(492,307)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6372.3 cny</tspan><tspan x="5" y="18">6372.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(491,327)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6360.3 cny</tspan><tspan x="5" y="18">6360.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(488,346)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6319.3 cny</tspan><tspan x="5" y="18">6319.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(481,366)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6233.9 cny</tspan><tspan x="5" y="18">6233.9 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(480,385)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6217.2 cny</tspan><tspan x="5" y="18">6217.2 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(480,404)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6216.1 cny</tspan><tspan x="5" y="18">6216.1 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(477,424)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6174.0 cny</tspan><tspan x="5" y="18">6174.0 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(474,443)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6145.2 cny</tspan><tspan x="5" y="18">6145.2 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(471,462)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6106.5 cny</tspan><tspan x="5" y="18">6106.5 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(470,482)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6084.7 cny</tspan><tspan x="5" y="18">6084.7 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(467,501)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6050.8 cny</tspan><tspan x="5" y="18">6050.8 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(466,521)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6038.1 cny</tspan><tspan x="5" y="18">6038.1 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(465,540)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6022.3 cny</tspan><tspan x="5" y="18">6022.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(463,559)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">6002.8 cny</tspan><tspan x="5" y="18">6002.8 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(463,579)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5993.3 cny</tspan><tspan x="5" y="18">5993.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(462,598)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5988.9 cny</tspan><tspan x="5" y="18">5988.9 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(461,618)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5968.9 cny</tspan><tspan x="5" y="18">5968.9 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(458,637)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5937.5 cny</tspan><tspan x="5" y="18">5937.5 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(456,656)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5912.4 cny</tspan><tspan x="5" y="18">5912.4 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(454,676)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5888.2 cny</tspan><tspan x="5" y="18">5888.2 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(454,695)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5876.9 cny</tspan><tspan x="5" y="18">5876.9 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(452,714)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5856.5 cny</tspan><tspan x="5" y="18">5856.5 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(450,734)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5829.4 cny</tspan><tspan x="5" y="18">5829.4 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(450,753)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5824.6 cny</tspan><tspan x="5" y="18">5824.6 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(445,773)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5762.3 cny</tspan><tspan x="5" y="18">5762.3 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(439,792)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5686.1 cny</tspan><tspan x="5" y="18">5686.1 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(437,811)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5666.8 cny</tspan><tspan x="5" y="18">5666.8 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(437,831)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5663.7 cny</tspan><tspan x="5" y="18">5663.7 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(432,850)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5599.9 cny</tspan><tspan x="5" y="18">5599.9 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(429,869)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5557.6 cny</tspan><tspan x="5" y="18">5557.6 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(418,889)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5410.0 cny</tspan><tspan x="5" y="18">5410.0 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(404,908)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">5230.7 cny</tspan><tspan x="5" y="18">5230.7 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(380,928)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">4920.7 cny</tspan><tspan x="5" y="18">4920.7 cny</tspan></text></g><g class="highcharts-label highcharts-data-label highcharts-data-label-color-undefined" data-z-index="1" transform="translate(374,947)"><text x="5" data-z-index="1" style="font-size:13px;font-weight:bold;color:#000000;fill:#000000;" y="18"><tspan x="5" y="18" class="highcharts-text-outline" fill="#FFFFFF" stroke="#FFFFFF" stroke-width="2px" stroke-linejoin="round">4839.4 cny</tspan><tspan x="5" y="18">4839.4 cny</tspan></text></g></g>

贪睡的苏简 · 发表于 2021-4-10 16:12:14

城市

<g class="highcharts-axis-labels highcharts-xaxis-labels" data-z-index="7"><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="84" opacity="1">北京市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="103" opacity="1">上海市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="123" opacity="1">杭州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="142" opacity="1">深圳市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="162" opacity="1">南京市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="181" opacity="1">台州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="200" opacity="1">绍兴市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="220" opacity="1">珠海市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="239" opacity="1">厦门市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="259" opacity="1">广州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="278" opacity="1">温州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="297" opacity="1">苏州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="317" opacity="1">金华市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="336" opacity="1">嘉兴市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="355" opacity="1">成都市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="375" opacity="1">乌鲁木齐市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="394" opacity="1">武汉市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="414" opacity="1">东莞市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="433" opacity="1">佛山市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="452" opacity="1">重庆市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="472" opacity="1">宁波市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="491" opacity="1">长沙市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="510" opacity="1">惠州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="530" opacity="1">福州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="549" opacity="1">济南市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="569" opacity="1">西安市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="588" opacity="1">贵阳市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="607" opacity="1">太原市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="627" opacity="1">无锡市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="646" opacity="1">中山市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="665" opacity="1">郑州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="685" opacity="1">徐州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="704" opacity="1">石家庄市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="724" opacity="1">合肥市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="743" opacity="1">南通市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="762" opacity="1">常州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="782" opacity="1">南昌市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="801" opacity="1">天津市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="821" opacity="1">兰州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="840" opacity="1">泉州市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="859" opacity="1">烟台市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="879" opacity="1">南宁市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="898" opacity="1">呼和浩特市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="917" opacity="1">海口市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="937" opacity="1">昆明市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="956" opacity="1">青岛市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="976" opacity="1">大连市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="995" opacity="1">长春市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="1014" opacity="1">哈尔滨市</text><text x="65" style="color:#666666;cursor:default;font-size:11px;fill:#666666;" text-anchor="end" transform="translate(0,0)" y="1034" opacity="1">沈阳市</text></g>

贪睡的苏简 · 发表于 2021-4-10 16:15:47

import xlwt
import re
import os
os.chdir(r"D:\BianCheng\Pythonwork\NextPython\重塑爬虫\TXT文件")
workbook = xlwt.Workbook(encoding="utf-8")
worksheet = workbook.add_sheet("sheet1")
worksheet.write(0,0,"城市")
worksheet.write(0,1,"工资")
with open("城市.html","r",encoding="utf-8") as f:
    fstr = str(f.read())
    FindName = re.compile(r'<text x=.*?>(.*?)</text>')
    Name = re.findall(FindName,fstr)
    for i in range(0,len(Name)):
        worksheet.write(i+1,0,Name[i])

with open("工资.html","r",encoding="utf-8") as f:
    fstr = str(f.read())
    Findgz = re.compile(r'<tspan x=.*?>(.*?)</tspan>')
    gz = re.findall(Findgz,fstr)
    j = 0
    for i in range(0,len(gz),2):
        worksheet.write(j+1,1,gz[i])
        j += 1


os.chdir(r"D:\BianCheng\Pythonwork\NextPython\重塑爬虫\比赛表格")
workbook.save("2020年各大城市工资排名表.xls")

我自己一种新的思路：
即——>你不给我用爬虫爬到你隐藏的html，那么我就直接去网页找到我要的元素的内容
然后我不访问，我直接复制你给我看到的html数据
然后弄到我的文件夹里，直接打开，利用正则表达式，来爬。

可见即可爬，不给我爬？不可能
（PS:只用过一次，就是这次，爬的也不是实时的，只能说是静态的，但对于初学者来说，可以开始爬就已经很不错了）

Daniel_Zhang · 发表于 2021-4-10 16:24:41

贪睡的苏简发表于 2021-4-10 16:15
我自己一种新的思路：
即——>你不给我用爬虫爬到你隐藏的html，那么我就直接去网页找到我要的元素的 ...

你咋还在搞这个

前几天就看你搞了,应该不是太难吧

suchocolate · 发表于 2021-4-10 20:05:34

贪睡的苏简发表于 2021-4-10 16:09
这个难说，例如这里的城市和工资被隐藏了，我就爬不到

不同的网站方式不同，有的数据在html里，有的得通过ajax获取。
刚刚贴的这个网站我登陆不上。

贪睡的苏简 · 发表于 2021-4-10 20:57:42

suchocolate 发表于 2021-4-10 20:05
不同的网站方式不同，有的数据在html里，有的得通过ajax获取。
刚刚贴的这个网站我登陆不上。

好吧

贪睡的苏简 · 发表于 2021-4-10 20:58:58

Daniel_Zhang 发表于 2021-4-10 16:24
你咋还在搞这个

前几天就看你搞了,应该不是太难吧

没办法，我理解的不是很清楚，所以就尝试一直搞下去，并且换几个思路去搞，发现些不一样的东西，现在我基本会用了

591821661 · 发表于 2021-4-13 19:28:26

贪睡的苏简发表于 2021-4-9 16:16
这个代码，为什么会得到这个结果
D:\BianCheng\Python\python.exe D:/BianCheng/Pythonwork/NextPython/ ...

import urllib.request,urllib.parse
#from lxml import etree
proxy = '127.0.0.1:80'
proxy_support = urllib.request.ProxyHandler({'http':proxy})
opener = urllib.request.build_opener(proxy_support)
opener.addheaders = [('User-Agent','Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36')]
urllib.request.install_opener(opener)
url = 'https://www.baidu.com'
#wrong_url = 'http://www.baidu.com' # Error code 10054
req         = urllib.request.Request(url)
res         = urllib.request.urlopen(req)
html_text   = res.read().decode('utf-8')
print(html_text)

账号		自动登录	找回密码
密码			立即注册