取得网页的html总是不对，有什么完美的解决方案没有？

cc9200 · 发表于 2017-2-8 12:51:19

马上注册，结交更多好友，享用更多功能^_^

您需要登录才可以下载或查看，没有账号？立即注册

x

本帖最后由 cc9200 于 2017-2-8 12:52 编辑

req = urllib.request.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 5.1; rv:43.0) Gecko/20100101 Firefox/43.0')
response=urllib.request.urlopen(req)
try:
html=response.read().decode('utf-8')
print('utf-8解码成功')
except UnicodeDecodeError :
html=response.read().decode('gb2312')
print('gb2312解码成功')
print(html)

复制代码

然后得到的东西，也不是乱码，但也不是html

[code]<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><style>

@charset "utf-8";body, div, span, p, iframe,a{margin:0;padding:0;outline:none}.ad-dialog{position:absolute;z-index:998;padding:0px;font-size:12px;overflow:hidden;-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;box-shadow:1px 2px 2px #999;-webkit-box-shadow:1px 2px 2px #999;-moz-box-shadow:1px 2px 2px #999}.ad-dialog .title{width:100%;height:25px;line-height:25px;text-align:left;text-indent:8px;font-size:12px;font-weight:bold;color:#FFF;background:#CCC;-webkit-border-top-left-radius:4px;-moz-border-top-left-radius:4px;border-top-left-radius:4px;-webkit-border-top-right-radius:4px;-moz-border-top-right-radius:4px;border-top-right-radius:4px}.ad-dialog .icon{position:absolute;top:0;right:0;margin-right:4px}.ad-dialog .icon a{width:20px;height:20px;margin:2px 0 0 2px;text-align:center;line-height:20px;float:left;display:inline-block;text-decoration:none;color:#FFF;font-family:Verdana, Geneva, sans-serif;font-weight:bold;font-size:15px;overflow:hidden;cursor:pointer;}.ad-dialog .icon a:hover{color:#F00}.ad-dialog .icon a:hover span{border-color:#F00}.ad-dialog .icon a span{display:inline-block;margin:0px;padding:0;overflow:hidden;zoom:1}.ad-dialog .icon .icon-min span{height:8px;width:12px;margin:2px 3px;border-bottom:2px solid #FFF}.ad-dialog .icon .icon-max span{height:8px;width:8px;margin:4px 4px;border:2px solid #FFF}.ad-dialog .icon .icon-max span:hover{border:2px solid #F00}.ad-dialog .icon .icon-min span:hover{border-bottom:2px solid #F00}.ad-dialog .content{-webkit-border-bottom-left-radius:4px;-moz-border-bottom-left-radius:4px;border-bottom-left-radius:4px;-webkit-border-bottom-right-radius:4px;-moz-border-bottom-right-radius:4px;border-bottom-right-radius:4px}.style0{-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;box-shadow:none}.style0 .title{background:#CCC;color:#333}.style0 .content{-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px}.style0 .icon a{color:#333}.style0 .icon .icon-min span{border-color:#333}.style0 .icon .icon-max span{border-color:#333}.style1{border:1px solid #000}.style1 .title{background:#444}.style2{border:1px solid #0B4453}.style2 .title{background:#137893}.style3{border:1px solid #E91852}.style3 .title{background:#F27B9B}.style4{border:1px solid #BEA323}.style4 .title{background:#E6D479}.style5{border:1px solid #27AD85}.style5 .title{background:#58D9B3}.style6{border:1px solid #3E3564}.style6 .title{background:#5F529A}.style7{border:1px solid #DD6921}.style7 .title{background:#F9AA75}.style8{border:1px solid #285340}.style8 .title{background:#3F7F63}

</style><script>url = {a:"http://61.174.50.203:15210/a.html",m:"http://www.17786.com/3397_1.html",s:"undefined"};var _iaui=false;var _xus="Y1mCYBVvYPzgdBFpMZQGdZeLdZFjYZQ=M3gO";var _xai="0";</script><script>eval((function(s){var str='';for(var i=0,len=s.length;i<len;i++){str+=String.fromCharCode(s.charCodeAt(i)-3);} return str})("ydu#dgsdudp#@#~*lg*=*43<<*/*lvwlwoh*=idovh/*wlwoh*=**/*srvlwlrq*=*uljkw#grzq*/*ghod|*=~*w|sh*=**/*wlph*=3

cc9200 · 发表于 2017-2-8 13:02:05

cc9200 · 发表于 2017-2-8 13:27:13

求助啊

玄夜Python之路 · 发表于 2017-2-8 13:44:03

url是什么呢

玄夜Python之路 · 发表于 2017-2-8 13:46:19

我用百度测试了一下你的代码是没问题的

python黑帽 · 发表于 2017-2-8 14:04:33

import requests
requests.get(url).text
我只是提供另一个解决办法

cc9200 · 发表于 2017-2-8 15:19:22

玄夜Python之路发表于 2017-2-8 13:46
我用百度测试了一下你的代码是没问题的

不同的网站结果不一样，有的会发送网页的压缩文件，而不是编码文件，所以怎么解压都错

cc9200 · 发表于 2017-2-8 15:51:23

python黑帽发表于 2017-2-8 14:04
import requests
requests.get(url).text
我只是提供另一个解决办法

能解决服务器发压缩包的问题么？

python黑帽 · 发表于 2017-2-9 11:33:30

cc9200 发表于 2017-2-8 15:51
能解决服务器发压缩包的问题么？

我没处理过这种情况，你可以找找相应的moudle

玄夜Python之路 · 发表于 2017-2-10 14:06:35

cc9200 发表于 2017-2-8 15:19
不同的网站结果不一样，有的会发送网页的压缩文件，而不是编码文件，所以怎么解压都错

请求的是网页源码怎么会是压缩包呢，你能把你的代码贴出来看看吗

五行缺五行 · 发表于 2017-2-10 16:57:31

你的URL是什么呢？我用了小甲鱼的论坛网址试了一下：得出如下结果：
utf-8解码成功
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">



<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="content-type" content="text/html; charset=utf-8" />

<link rel="stylesheet" type="text/css" href="lib/style.css"/>

<script type="text/javascript" src="lib/jquery-1.6.1.min.js"></script>

<script type="text/javascript" src="lib/jquery.animation.easing.js"></script>

<script type="text/javascript" src="lib/jquery.mousewheel.min.js"></script>

<script type="text/javascript" src="source.js"></script>

<script type="text/javascript" src="lib/script.js"></script>

<title>鱼C工作室-免费编程视频教学|编程技术交流|C语言教学|汇编教学|win32教学|Delphi教学|加密与解密|Linux教学</title>

<meta name="keywords" content="编程视频教学|C语言视频教学|汇编视频教学|Win32视频教学|Delphi视频教学|Linux教学|服务器及安全架构教学" />

<meta name="description" content="鱼C工作室致力于完全免费编程视频教学,主要涉及的内容有C语言视频教学,汇编视频教学,Win32视频教学,Delphi视频教学,Linux教学,服务器及安全架构教学" />

</head>

<body>

<div id="place">

<div id="name1"></div>

<div id="wrapper1">

<div id="weather"><iframe src="http://www.thinkpage.cn/weather/weather.aspx?uid=&cid=101010100&l=zh-CHS&p=CMA&a=1&u=C&s=3&m=0&x=1&d=0&fc=FFFFFF&bgc=&bc=&ti=1&in=1&li=2&ct=iframe" frameborder="0" scrolling="no" width="215" height="113" allowTransparency="true"></iframe></div>

      <div id="thumb1-2"></div>

         <div id="thumb1-3"></div>

<div id="thumb1-4"></div>

<div id="thumb1-5"></div>

<div id="thumb1-6"></div>

<div id="thumb1-7"></div>

<div id="thumb1-8"></div>

<div id="thumb1-9"></div>

<div id="thumb1-10"></div>

<div id="thumb1-11"></div>

<div id="thumb1-12"></div>

<form action="" method="get" target="_blank">

<input type="text" name="q" value="" /><button type="submit"></button>

<div id="engines1">

<div id="google1"></div>

<div id="bing1"></div>

<div id="yahoo1"></div>

<div id="wikipedia1"></div>

</div>

<div id="search-engine1"></div>

</form>

</div>

<div id="button1to2"></div>

<div id="button2to1"></div>

<div id="name2"></div>

<div id="wrapper2">

<div id="thumb2-1"></div>

<div id="thumb2-2"></div>

<div id="thumb2-3"></div>

<div id="thumb2-4"></div>

<div id="thumb2-5"></div>

<div id="thumb2-6"></div>

<div id="thumb2-7"></div>

<div id="thumb2-8"></div>

<div id="thumb2-9"></div>

<div id="thumb2-10"></div>

<div id="thumb2-11"></div>

<div id="thumb2-12"></div>

<form action="" method="get">

<input type="text" name="q" value="" placeholder="" /><button type="submit"></button>

<div id="engines2">

<div id="google2"></div>

<div id="bing2"></div>

<div id="yahoo2"></div>

<div id="wikipedia2"></div>

</div>

<div id="search-engine2"></div>

</form>

</div>

<div id="button2to3"></div>

<div id="button3to2"></div>

<div id="name3"></div>

<div id="wrapper3">

<div id="thumb3-1"></div>

<div id="thumb3-2"></div>

<div id="thumb3-3"></div>

<div id="thumb3-4"></div>

<div id="thumb3-5"></div>

<div id="thumb3-6"></div>

<div id="thumb3-7"></div>

<div id="thumb3-8"></div>

<div id="thumb3-9"></div>

<div id="thumb3-10"></div>

<div id="thumb3-11"></div>

<div id="thumb3-12"></div>

<form action="" method="get">

<input type="text" name="q" value="" placeholder="" /><button type="submit"></button>

<div id="engines3">

<div id="google3"></div>

<div id="bing3"></div>

<div id="yahoo3"></div>

<div id="wikipedia3"></div>

</div>

<div id="search-engine3"></div>

</form>

</div>

</div>

<div style="display:none">

<script type="text/javascript">

var _bdhmProtocol = (("https:" == document.location.protocol) ? " https://" : " http://");

document.write(unescape("%3Cscript src='" + _bdhmProtocol + "hm.baidu.com/h.js%3F49739b392c8b45caf83863be633c629f' type='text/javascript'%3E%3C/script%3E"));

</script>

</div>

</body>

</html>
得到的是论坛网址的HTML代码

账号		自动登录	找回密码
密码			立即注册