python url

little_chip

运行下面的代码得到
https://beses.cn/app/index.php?i ... _tbk&shopid=443
但是通过网页检查元素我的目标字符串是
href="https://beses.cn/app/index.php?i=4&c=entry&do=show_api&m=bsht_tbk&iid=550546081709&yj=2.78&tit=5aSP5aSp5a2j55S15Yqo6L2m6Ziy5pmS5oqr6IKp5aWz6aqR6L2m5aSW5aWX5rKZ5rup5be_b5aWz5aOr6Ziy57Sr5aSW57q_a5byA6L2m5oqr6aOO5paX56_b3&wenan=&pic=http://img.alicdn.com/bao/uploaded/i4/1766677855/TB2TtmliH_0UKFjy1XaXXbKfXXa_!!1766677855.jpg&fee1=13.9&fee2=13.9&qfee=0&fl=18&isdx=&shopid=443&isnb=1&maijia_id=1766677855&isnice=2&stype=5&shopyuji=0.9&hdk=1&actid=Array&vid=&q_yl=0&q_zl=0&qcid="
请问为什么怎么才能得到我想要的网址字符串


import requests
import base64
from bs4 import  BeautifulSoup

def get_url(name):
    url1='https://beses.cn/app/index.php?i=4&c=entry&do=index&m=bsht_tbk&shopid=443'
    url1='https://beses.cn/app/index.php?c=entry&i=4&m=bsht_tbkquan&do=so&itemid=&sid=&itemfee=&itempic=&nb_shopid=&keyword='
    url2='&shopid=443&actid=&t=1523351145&nbnice=1&openid='
    url=url1+base64.b64encode(base64.b64encode(name.encode())).decode()+url2
    aa = requests.get(url).text
    return aa

if __name__ == '__main__':
    html_doc = get_url('夏天季电动车防晒披肩女骑车外套沙滩巾女士防紫外线开车披风斗篷')
    soup = BeautifulSoup(html_doc, features="lxml")
    tag_all = soup.find(name='li').find(name='a').get('href')
    print(tag_all)

ABC23

正则表达式？

little_chip

ABC23 发表于 2018-4-11 17:46
正则表达式？

这个无关正则表达式，主要是request.get().text得到的字符串跟我直接从网页检查元素看到的不一样，查找资料发现应该是动态网页，不能用抓取静态网页的方法

wongyusing

这是手机端网页？？
如果是的话，要通过抓包获取内容，
或者是通过模拟操作

little_chip

wongyusing 发表于 2018-4-12 14:41
这是手机端网页？？
如果是的话，要通过抓包获取内容，
或者是通过模拟操作

大神请帮我看看，您所说的抓包是不是网页的network>XHR>name,里面的文件链接
Request URL:https://beses.cn/app/index.php?i ... &m=bsht_tbkquan
Request Method:POST
Status Code:200
Remote Address:47.92.118.178:443
Referrer Policy:no-referrer-when-downgrade

我打开https://beses.cn/app/index.php?i ... &m=bsht_tbkquan
还是看不明白，里面是类似
{"info":"ok","num":30,"nb_qlist":[{"issiteq":0,"title":"1\u7537\u7ae53\u5e7c\u513f\u5b9d\u5b9d4\u513f\u7ae5\u80cc\u5fc3\u7eaf\u68c9\u590f\u5b638\u5c815\u7537\u5b696\u5c0f\u5b69\u8584\u6b3e\u65e0\u8896\u5de5\u5b57\u7ae5\u88c5","wenan":null,"itemid":"533881470073","itempic":"http:\/\/img.alicdn.com\/bao\/uploaded\/i3\/2371097880\/TB2FvVqqFXXXXXLXXXXXXXXXXXX_!!2371097880.gif","picarr":"","itemmsell":"75","istmall":"0","maijia_id":"2371097880","shouru_bili":3,"itemfee":"8.9","qfee":0,"itemfee2":8.9,"itemyhj_zl":"0","itemyhj_yl":"0","itemyhj_stime":null,"itemyh{"issiteq":0,"title":"2018\u590f\u5b63\u5973\u7ae5\u7ae5\u88c5\u6253\u5e95\u88e4\u6b3e \u5b9d\u5b9d\u513f\u7ae5\u4e2d\u88e4\u4e94\u5206\u88e4\u7ae5\u88e4 \u5973kz-
这是什么编码

wongyusing

看这里：

1. \u590f\u5b63\u5973\u7ae5\u7ae5\u88c5\u6253\u5e95\u88e4\u6b3e \u5b9d\u5b9d\u513f\u7ae5\u4e2d\u88e4\u4e94\u5206\u88e4\u7ae5\u88e4 \u5973kz-

复制代码

如果我没猜错的话，要么decode一下编码，
要么这是json格式的内容，
我也是新手，最近在看爬虫的视频好像见过这种类型的内容，
再具体点我也说不上来