python

amoschen · 发表于 2019-9-27 17:45:28

马上注册，结交更多好友，享用更多功能^_^

您需要登录才可以下载或查看，没有账号？立即注册

x

一个爬虫的例子
网址：http://n.jgtong.com.cn/fjgs/BAlist.aspx?dq=1&pageIndex=2
问题：1、代码

import urllib.request
from bs4 import BeautifulSoup
url = "http://n.jgtong.com.cn/fjgs/BAlist.aspx?dq=1&pageIndex=2"
response = urllib.request.urlopen(url)
html = response.read().decode("utf-8")
soup = BeautifulSoup(html,"html.parser")
print(soup)

复制代码

   以上代码爬取的网站代码与在浏览器中f12看到的代码是不一样的，为什么？

      2、

zltzlt · 发表于 2019-9-27 20:55:47

哪里不一样？

amoschen · 发表于 2019-9-29 14:18:54

zltzlt 发表于 2019-9-27 20:55
哪里不一样？

<tbody><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=567db1c2-6b4a-4180-977c-6c152d689f34" class="zi1">安信颐和公寓(1号车库)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-13</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=9bd369f2-4405-456e-b96a-e4140dc926ae" class="zi1">学仕花园(16幢,19幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-12</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=42764db8-b8d0-4243-aa64-60c10c432950" class="zi1">豪景家园(1幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-12</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=a2b8ddfd-197a-461b-a5c0-094b427bcbc7" class="zi1">君宸苑(6幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-05</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=f52d8fd3-b70a-4ed3-bcfa-c7ccd44559d2" class="zi1">雅乐花园(13幢,18幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-05</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=d52c9799-ea9d-4c7c-b173-bd1f6daf9e79" class="zi1">文锦园(1幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=8D488EAE-58EC-4C2E-8886-48C6B93AA7B0" class="zi1">蓝天新苑重新备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=15cc3f6a-c995-472d-a959-84e49ee4dab6" class="zi1">思贤雅苑(11幢,14幢,7幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=21c934b7-6c20-47cc-9f1b-f6086ab83c9d" class="zi1">星韵城(A17幢,A9幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=347b5990-b8c5-4651-aee9-9ced867925b1" class="zi1">怡安家园(1幢,2幢,3幢,4幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=1B17A79C-7800-4AD1-8A48-C5837875D82E" class="zi1">御水华庭重新备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=5b8fb22c-49f0-4cda-bee2-83e375873c2c" class="zi1">泰和之春苑续建(1号车库)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-02</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=7407e206-2cd5-4c2e-85b1-987b266107b4" class="zi1">衡麓花园(1幢,2幢,4幢,5幢,6幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-01</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=1debbcbd-9e2d-4868-bea6-5f4fd6a83251" class="zi1">香荷名居(11幢,3幢,7幢)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-01</td> </tr><tr style="border-bottom:1px solid #fefefe;"> <td width="4%" valign="bottom"> <img src="images/a_r20_c15.jpg" width="13" height="13"></td> <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id=76d66757-9b67-4bf5-9bf0-bfb991322e7d" class="zi1">邹区时代广场(1号车库,3号车库、4号车库,6号车库)首次备案公示 </a></td> <td width="150" valign="bottom" class="zi_hun">2019-08-01</td> </tr></tbody>

复制代码

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312" />
<title>常州市商品房备案公示</title>
<link href="../css.css" rel="stylesheet" type="text/css" />
<style type="text/css">
body
{
margin-left: 0px;
margin-top: 0px;
margin-right: 0px;
margin-bottom: 0px;
}
body, td, th
{
font-size: 14px;
}
.STYLE2
{
font-size: 12px;
}
</style>
<script src="../js/jquery-1.7.1.min.js" type="text/javascript"></script>
</head>
<script>
//table分页
var pageSize=15; //每页显示的记录条数
var curPage='2'; //显示第curPage页
var len; //总行数
var page; //总页数
$(function(){
len =$("#show_tab tr").length-1; //去掉表头
page=len % pageSize==0 ? len/pageSize : Math.floor(len/pageSize)+1;//根据记录条数，计算页数
console.log("len:"+len+"page:"+page);
document.getElementById("page").value=page;
alert(curPage)
displayPage();//显示第一页
$("#nextpage").click(function(){//下一页
if(curPage<page){
curPage+=1;
}
else{
// alert("yishizuihouyiye");
}
displayPage();
});
$("#lastpage").click(function(){//上一页
if(curPage>1){
curPage-=1;
}
else{
// alert("yishidiyiye");
}
displayPage();
});
$("#npage").click(function(){//跳到固定某一页
var npage=parseInt(document.getElementById("curPage").value);
if(npage>page||npage<1){
// alert("gaiyebucunzai");
}
else{
curPage=npage;
}
displayPage();
});
});
function displayPage(){
var begin=(curPage-1)*pageSize;//起始记录号
var end = begin + pageSize;
console.log(" begin:"+len+" end:"+end);
if(end > len ) end=len;
$("#show_tab tr").hide();
$("#show_tab tr").each(function(i){
if(i-1>=begin && i-1<end)//显示第page页的记录
{
$("#show_tab_one").show();
$(this).show();
document.getElementById("curPage").value=curPage;
}
});
}
function pageSize2(){
curPage=0; //显示第curPage页
pageSize=parseInt(document.getElementById("pageSize").value);
len =$("#show_tab tr").length-1; //去掉表头
page=len % pageSize==0 ? len/pageSize : Math.floor(len/pageSize)+1;//根据记录条数，计算页数
console.log("len:"+len+" page:"+page);
document.getElementById("page").value=page;
curPage=1;
displayPage();//显示第一页
}
var strorl='{"data":[{"BA_ID":"567db1c2-6b4a-4180-977c-6c152d689f34","XM_MC":"安信颐和公寓", "DH":"1号车库","BA_LX":"0","BA_BJSJ_SJ":"2019-08-13"},{"BA_ID":"9bd369f2-4405-456e-b96a-e4140dc926ae","XM_MC":"学仕花园", "DH":"16幢,19幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-12"},{"BA_ID":"42764db8-b8d0-4243-aa64-60c10c432950","XM_MC":"豪景家园", "DH":"1幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-12"},{"BA_ID":"a2b8ddfd-197a-461b-a5c0-094b427bcbc7","XM_MC":"君宸苑", "DH":"6幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-05"},{"BA_ID":"f52d8fd3-b70a-4ed3-bcfa-c7ccd44559d2","XM_MC":"雅乐花园", "DH":"13幢,18幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-05"},{"BA_ID":"d52c9799-ea9d-4c7c-b173-bd1f6daf9e79","XM_MC":"文锦园", "DH":"1幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"8D488EAE-58EC-4C2E-8886-48C6B93AA7B0","XM_MC":"蓝天新苑", "DH":"","BA_LX":"1","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"15cc3f6a-c995-472d-a959-84e49ee4dab6","XM_MC":"思贤雅苑", "DH":"11幢,14幢,7幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"21c934b7-6c20-47cc-9f1b-f6086ab83c9d","XM_MC":"星韵城", "DH":"A17幢,A9幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"347b5990-b8c5-4651-aee9-9ced867925b1","XM_MC":"怡安家园", "DH":"1幢,2幢,3幢,4幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"1B17A79C-7800-4AD1-8A48-C5837875D82E","XM_MC":"御水华庭", "DH":"","BA_LX":"1","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"5b8fb22c-49f0-4cda-bee2-83e375873c2c","XM_MC":"泰和之春苑续建", "DH":"1号车库","BA_LX":"0","BA_BJSJ_SJ":"2019-08-02"},{"BA_ID":"7407e206-2cd5-4c2e-85b1-987b266107b4","XM_MC":"衡麓花园", "DH":"1幢,2幢,4幢,5幢,6幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-01"},{"BA_ID":"1debbcbd-9e2d-4868-bea6-5f4fd6a83251","XM_MC":"香荷名居", "DH":"11幢,3幢,7幢","BA_LX":"0","BA_BJSJ_SJ":"2019-08-01"},{"BA_ID":"76d66757-9b67-4bf5-9bf0-bfb991322e7d","XM_MC":"邹区时代广场", "DH":"1号车库,3号车库、4号车库,6号车库","BA_LX":"0","BA_BJSJ_SJ":"2019-08-01"}]}';
function ConvertToJsonForJs() {
var testJson = jQuery.parseJSON(strorl);
var html="";
var dts=testJson.data;
if(dts.length>0)
{
for(var i=0;i<dts.length;i++)
{
html +='<tr style="border-bottom:1px solid #fefefe;">';
html +=' <td width="4%" valign="bottom">';
html +=' <img src="images/a_r20_c15.jpg" width="13" height="13" /></td>';
var DH="";
if(dts[i].DH!=null && dts[i].DH!=""){
if(dts[i].DH.length>25)
{
DH="("+dts[i].DH.substr(0,25)+"...)";
}else{
DH="("+ dts[i].DH +")";
}
}
html +=' <td width="84%" height="35" valign="bottom"><a target="_blank" href="BAdetail.aspx?id='+ dts[i].BA_ID +'" class="zi1">'+dts[i].XM_MC+DH;
if(dts[i].BA_LX=="0"){
html+="首次备案";
}else{
html+="重新备案";
}
html +=' 公示';
html +=' </a></td>';
html +=' <td width="150" valign="bottom" class="zi_hun">'+ dts[i].BA_BJSJ_SJ +'</td>';
html +=' </tr>';
}
$("#show_tab").html(html);
}
}
</script>
<body>
<table width="1000" border="0" cellspacing="0" cellpadding="0" align="center">
<tr>
<td><img src="images/ss2.jpg"></td>
</tr>
</table>
<table width="1000" height="289" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td valign="top">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="259" height="657" valign="top">
<table width="100%" height="520" border="0" cellpadding="0" cellspacing="0">
<tr>
<td valign="top" background="images/main_r1_c1.jpg">
<table width="192" height="74" border="0" cellpadding="0" cellspacing="0">
<tr>
<td background="images/m3.png">
<table width="81%" height="34" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td class="zi_bai22">
房价公示
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td>
</td>
</tr>
</table>
<table width="100%" height="55" border="0" cellpadding="0" cellspacing="0">
<tr>
<td height="55" valign="top">
<table width="222" border="0" align="right" cellpadding="0" cellspacing="0">
<tr>
<td width="211" height="51" valign="top" background="images/m2.png">
<table width="85%" height="40" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td>
<a href="?dq=1" class="zi_hei20">中心城区</a>
</td>
</tr>
</table>
</td>
<td width="9">
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" height="55" border="0" cellpadding="0" cellspacing="0">
<tr>
<td height="55" valign="top">
<table width="222" border="0" align="right" cellpadding="0" cellspacing="0">
<tr>
<td width="211" height="51" valign="top" background="images/m2.png">
<table width="85%" height="40" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td>
<a href="?dq=2" class="zi_hei20">新北区</a>
</td>
</tr>
</table>
</td>
<td width="9">
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" height="55" border="0" cellpadding="0" cellspacing="0">
<tr>
<td height="55" valign="top">
<table width="222" border="0" align="right" cellpadding="0" cellspacing="0">
<tr>
<td width="211" height="51" valign="top" background="images/m2.png">
<table width="85%" height="40" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td>
<a href="?dq=3" class="zi_hei20">武进区</a>
</td>
</tr>
</table>
</td>
<td width="9">
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" height="55" border="0" cellpadding="0" cellspacing="0">
<tr>
<td height="55" valign="top">
<table width="222" border="0" align="right" cellpadding="0" cellspacing="0">
<tr>
<td width="211" height="51" valign="top" background="images/m2.png">
<table width="85%" height="40" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td>
<a href="?dq=4" class="zi_hei20">金坛市</a>
</td>
</tr>
</table>
</td>
<td width="9">
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" height="55" border="0" cellpadding="0" cellspacing="0">
<tr>
<td height="55" valign="top">
<table width="222" border="0" align="right" cellpadding="0" cellspacing="0">
<tr>
<td width="211" height="51" valign="top" background="images/m2.png">
<table width="85%" height="40" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td>
<a href="?dq=5" class="zi_hei20">溧阳市</a>
</td>
</tr>
</table>
</td>
<td width="9">
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- <table width="100%" height="55" border="0" cellpadding="0" cellspacing="0">
<tr>
<td height="55" valign="top">
<table width="222" border="0" align="right" cellpadding="0" cellspacing="0">
<tr>
<td width="211" height="51" valign="top" background="images/m2.png">
<table width="85%" height="40" border="0" align="center" cellpadding="0" cellspacing="0">
<tr>
<td>
<a href="newslist.aspx?tid=001012006" class="zi_hei20">表格下载</a>
</td>
</tr>
</table>
</td>
<td width="9">
</td>
</tr>
</table>
</td>
</tr>
</table>-->
</td>
</tr>
</table>
</td>
<td valign="top">
<table width="732" height="729" border="0" align="center" cellpadding="0" cellspacing="0"
class="table2">
<tr>
<td valign="top" align="center">
<table width="100%" height="37" border="0" cellpadding="0" cellspacing="0">
<tr>
<td background="images/main_bg.jpg">
<table width="229" border="0" cellspacing="0" cellpadding="0">
<tr>
<td height="37" background="images/m1.png" class="zi_bai22">
<table width="90%" height="27" border="0" cellpadding="0" cellspacing="0">
<tr>
<td width="9%">
</td>
<td width="91%" class="zi_bai22">
房价备案公示
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td>
</td>
</tr>
</table>
<table id="show_tab" width="100%">
</table>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td height="10" align=center><a href="BAlist.aspx?dq=1&pageIndex=1">上一页</a><a href="BAlist.aspx?dq=1&pageIndex=3"> 下一页 </a>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td height="10">
</td>
</tr>
</table>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td height="1">
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
<script>
ConvertToJsonForJs();
</script>
</html>

复制代码

第一段代码是在查看元素时是可以看到的，在tbody这个标签中，我要爬取的目标数据就在里面；第二段代码，是爬取的代码，在tbody中就没有我要数据，变成了是在script中的var strorl；而且还有一个地方就是，含有var strorl的script标签，在查看元素的代码中，它是在head中的，但是爬取的代码中，它又不在head中了，这我就迷茫了，怎么也爬取不到这段数据，麻烦您给看看，问题出在哪里了，

amoschen · 发表于 2019-9-29 14:36:14

zltzlt 发表于 2019-9-27 20:55
哪里不一样？

我没法发图片，代码太多，简单说吧，就是我要的目标数据var strorl，f12查看的网站源代码时，它所处的位置如下：

<head>
...
...
<script src="../js/jquery-1.7.1.min.js" type="text/javascript"></script>
<script>
..
var strorl='{"data":[....]}';
...
</script>
</head>

复制代码

但是我爬取的代码就变成了下面这种了：

<head>
...
...
<script src="../js/jquery-1.7.1.min.js" type="text/javascript"></script>
</head>
<script>
...
var strorl='{"data":[...]}';
...
</script>

复制代码

而且在查看元素的代码中，var strorl这部分数据可以在body标签中看到，在爬取的代码中就没有了。然后我就迷茫了，网上查了很久也没有查到怎么解决，所以就来求助了！

amoschen · 发表于 2019-9-29 16:22:11

额。。。。。已经解决了，原来一个正则表达式就能解决的问题！

账号		自动登录	找回密码
密码			立即注册

python

马上注册，结交更多好友，享用更多功能^_^

浏览过的版块