[已解决]爬虫问题

元豪 · 发表于 2022-9-24 20:58:12

跟着小甲鱼写程序但运行不起来
https://www.bilibili.com/video/BV1Fs411A7HZ/?p=55&spm_id_from=pageDriver&vd_source=7e03b4dee8e084d03879d74acd3a0772
在第15分5秒左右，
跟小甲鱼写了半天，结果输出这玩意儿：

{"errorCode":50}

我在有道翻译看到的Form Data是这样的：

i: I love FishC.com!
from: AUTO
to: AUTO
smartresult: dict
client: fanyideskweb
salt: 16640232015992
sign: 4bb24289fa0e792d3c2fd56096b5ad78
lts: 1664023201599
bv: 79d103f0086f0e2586344a7cd1b549d7
doctype: json
version: 2.1
keyfrom: fanyi.web
action: FY_BY_CLICKBUTTION

这是我的代码：

import urllib.request
import urllib.parse

url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'
data = {}

data['i'] = 'I love FishC.com!'
data['from'] = 'AUTO'
data['to'] = 'AUTO'
data['smartresult'] = 'dict'
data['client'] = 'fanyideskweb'
data['salt'] = '16640215814102'
data['sign'] = '7d8e358cc52232beadfa7f868aef0a1b'
data['lts'] = '1664021581410'
data['bv'] = '79d103f0086f0e2586344a7cd1b549d7'
data['doctype'] = 'json'
data['version'] = '2.1'
data['keyfrom'] = 'fanyi.web'
data['action'] = 'FY_BY_CLICKBUTTION'
data = urllib.parse.urlencode(data).encode('utf-8')

r = urllib.request.urlopen(url, data)

html = r.read().decode('utf-8')

print(html)

很烦，求大神解释一下

最佳答案

月排行榜 / 总排行榜

临时号

2022-9-24 20:58:13

模拟一下有道的反爬算法就可以了
代码:

import urllib.request
import urllib.parse
import random
import time
import hashlib

url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'
headers ={}
data = {}
text = "I love FishC.com!"

lts = str(int(time.time()*1000))
salt = lts + str(random.randint(0,10))
sign_str = 'fanyideskweb' + text + salt + 'Ygy_4c=r#e#4EX^NUGUc5'
m = hashlib.md5(sign_str.encode())
sign = m.hexdigest()

headers["User-Agent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
headers["Referer"] = "https://fanyi.youdao.com/"
headers["Cookie"] = "OUTFOX_SEARCH_USER_ID=1907954952@182.203.147.49; OUTFOX_SEARCH_USER_ID_NCOO=2032702153.1070416; fanyi-ad-id=306808; fanyi-ad-closed=1; ___rl__test__cookies=1654931146702"

data['i'] = text
data['from'] = 'AUTO'
data['to'] = 'AUTO'
data['smartresult'] = 'dict'
data['client'] = 'fanyideskweb'
data['salt'] = salt
data['sign'] = sign
data['lts'] = lts
data['bv'] = '79d103f0086f0e2586344a7cd1b549d7'
data['doctype'] = 'json'
data['version'] = '2.1'
data['keyfrom'] = 'fanyi.web'
data['action'] = 'FY_BY_CLICKBUTTION'
data = urllib.parse.urlencode(data).encode('utf-8')

request = urllib.request.Request(url,data=data,headers=headers)
r = urllib.request.urlopen(request)

html = r.read().decode('utf-8')

print(html)

运行结果:

{"errorCode":0,"translateResult":[[{"tgt":"我爱FishC.com !","src":"I love FishC.com!"}]],"type":"en2zh-CHS"}
[Finished in 800ms]

跳转到最佳答案楼层

临时号 · 发表于 2022-9-24 20:58:13

模拟一下有道的反爬算法就可以了
代码:

import urllib.request
import urllib.parse
import random
import time
import hashlib

url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'
headers ={}
data = {}
text = "I love FishC.com!"

lts = str(int(time.time()*1000))
salt = lts + str(random.randint(0,10))
sign_str = 'fanyideskweb' + text + salt + 'Ygy_4c=r#e#4EX^NUGUc5'
m = hashlib.md5(sign_str.encode())
sign = m.hexdigest()

headers["User-Agent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
headers["Referer"] = "https://fanyi.youdao.com/"
headers["Cookie"] = "OUTFOX_SEARCH_USER_ID=1907954952@182.203.147.49; OUTFOX_SEARCH_USER_ID_NCOO=2032702153.1070416; fanyi-ad-id=306808; fanyi-ad-closed=1; ___rl__test__cookies=1654931146702"

data['i'] = text
data['from'] = 'AUTO'
data['to'] = 'AUTO'
data['smartresult'] = 'dict'
data['client'] = 'fanyideskweb'
data['salt'] = salt
data['sign'] = sign
data['lts'] = lts
data['bv'] = '79d103f0086f0e2586344a7cd1b549d7'
data['doctype'] = 'json'
data['version'] = '2.1'
data['keyfrom'] = 'fanyi.web'
data['action'] = 'FY_BY_CLICKBUTTION'
data = urllib.parse.urlencode(data).encode('utf-8')

request = urllib.request.Request(url,data=data,headers=headers)
r = urllib.request.urlopen(request)

html = r.read().decode('utf-8')

print(html)

运行结果:

{"errorCode":0,"translateResult":[[{"tgt":"我爱FishC.com !","src":"I love FishC.com!"}]],"type":"en2zh-CHS"}
[Finished in 800ms]

Twilight6 · 发表于 2022-9-24 21:01:34

url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'

将 URL 中的 _o 给去掉即可，有道的假链反爬，参考代码：

import urllib.request import urllib.parse url = 'https://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule' data = {} data['i'] = 'I love FishC.com!' data['from'] = 'AUTO' data['to'] = 'AUTO' data['smartresult'] = 'dict' data['client'] = 'fanyideskweb' data['salt'] = '16640215814102' data['sign'] = '7d8e358cc52232beadfa7f868aef0a1b' data['lts'] = '1664021581410' data['bv'] = '79d103f0086f0e2586344a7cd1b549d7' data['doctype'] = 'json' data['version'] = '2.1' data['keyfrom'] = 'fanyi.web' data['action'] = 'FY_BY_CLICKBUTTION' data = urllib.parse.urlencode(data).encode('utf-8') r = urllib.request.urlopen(url, data) html = r.read().decode('utf-8') print(html)

元豪 · 发表于 2022-9-24 21:07:48

Twilight6 发表于 2022-9-24 21:01
url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'

将 URL 中的 _o ...

红色的一坨

Twilight6 · 发表于 2022-9-24 21:10:07

元豪发表于 2022-9-24 21:07
红色的一坨

你没改参数吧？不应该会报错的，可能是你访问太频繁了。我这里正常执行：

重新拷去运行看看

xiaoyouzi11 · 发表于 2022-9-24 21:28:39

我输出是{"type":"EN2ZH_CN","errorCode":0,"elapsedTime":1,"translateResult":[[{"src":"I love FishC.com!","tgt":"我爱FishC.com !"}]]}

元豪 · 发表于 2022-9-25 07:42:09

在IDLE和VS code上都不行啊?

元豪 · 发表于 2022-9-25 08:02:37

临时号发表于 2022-9-24 21:09
模拟一下有道的反爬算法就可以了
代码:

成功了，但能解释一下吗？

wp231957 · 发表于 2022-9-25 12:52:14

元豪发表于 2022-9-25 08:02
成功了，但能解释一下吗？

有道的令牌是用md5加密，这需要你去逆向js
这没啥可解释的，真要解释起来，需要好长的篇幅

临时号 · 发表于 2022-9-26 23:05:04

本帖最后由临时号于 2022-9-26 23:06 编辑

元豪发表于 2022-9-25 08:02
成功了，但能解释一下吗？

在fanyi.min.js这个文件中,定义了对salt,sign,lts的加密算法
在fanyi.min.js的第8380行左右,有这么一段代码

var n = e("./jquery-1.7");
e("./utils");
e("./md5");
var r = function(e) {
    var t = n.md5(navigator.appVersion)
      , r = "" + (new Date).getTime()
      , i = r + parseInt(10 * Math.random(), 10);
    return {
        ts: r,
        bv: t,
        salt: i,
        sign: n.md5("fanyideskweb" + e + i + "Ygy_4c=r#e#4EX^NUGUc5")
    }
};
t.recordUpdate = function(e) {
    var t = e.i
      , i = r(t);
    n.ajax({
        type: "POST",
        contentType: "application/x-www-form-urlencoded; charset=UTF-8",
        url: "/bettertranslation",
        data: {
            i: e.i,
            client: "fanyideskweb",
            salt: i.salt,
            sign: i.sign,
            lts: i.ts,
            bv: i.bv,
            tgt: e.tgt,
            modifiedTgt: e.modifiedTgt,
            from: e.from,
            to: e.to
        },
        success: function(e) {},
        error: function(e) {}
    })
}

这是一段javascript代码,可以看到
对lts的加密代码是 r = "" + (new Date).getTime()
对salt的加密代码是i = r + parseInt(10 * Math.random(), 10);
对sign的加密代码是n.md5("fanyideskweb" + e + i + "Ygy_4c=r#e#4EX^NUGUc5")
将它们转化为对应的python代码就是
lts:lts = str(int(time.time()*1000))
salt:salt = lts + str(random.randint(0,10))
sign:

sign_str = 'fanyideskweb' + text + salt + 'Ygy_4c=r#e#4EX^NUGUc5'
m = hashlib.md5(sign_str.encode())
sign = m.hexdigest()

以上内容听不懂的没有关系,很正常

元豪 · 发表于 2022-9-27 19:10:57

临时号发表于 2022-9-26 23:05
在fanyi.min.js这个文件中,定义了对salt,sign,lts的加密算法
在fanyi.min.js的第8380行左右,有这么一 ...

(似懂非懂的点点头)emm......

临时号 · 发表于 2022-9-27 23:00:01

元豪发表于 2022-9-27 19:10
(似懂非懂的点点头)emm......

听不懂没关系,我在你这个阶段的时候肯定也听不懂,很正常

账号		自动登录	找回密码
密码			立即注册