寄安 发表于 2021-10-1 11:34:10

设置反爬

爬取猫眼电影的榜单信息，大神们不用给我在做一个爬虫了，帮忙看看我这个哪里出错了

import requests
from requests.exceptions import RequestException
import re

url = f"https://maoyan.com/board/4"
headers = {
    'Content-Type': 'text/html; charset=utf-8',
    'Origin': r'https://maoyan.com',
    'Referer': r'https://maoyan.com/board/4',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
}
cookies = {
    '__mta=145013643.1632827792736.1632838202638.1633053347625.31; uuid_n_v=v1;',
    'uuid=85DA7560204D11EC9B102B1A959882D0A97A7D0F881E4ACF90ED4B138FB89D1E;',
    '_lxsdk_cuid=17c2c1edc9ec8-0f7c6d795c2f3d-a7d193d-144000-17c2c1edc9ec8;',
    '_lxsdk=85DA7560204D11EC9B102B1A959882D0A97A7D0F881E4ACF90ED4B138FB89D1E;',
    '_csrf=c996b5dd3a909586fa13721ce47e20c12cb7b63b30767bb486a654d025637589;',
    'Hm_lvt_703e94591e87be68cc8da0da7cbd0be2=1632995467,1632999503,1633005709,1633053333;',
    '_lx_utm=utm_source%3DBaidu%26utm_medium%3Dorganic; __mta=145013643.1632827792736.1632838202638.1633053344019.31;',
    'Hm_lpvt_703e94591e87be68cc8da0da7cbd0be2=1633053348; _lxsdk_s=17c39cfd427-2ae-2d5-358%7C%7C1'
}

def get_one_page(url):
    try:
      response = requests.get(url=url,headers=headers,cookies=cookies)
      if response.status_code == 200:
            return response.text
      return None
    except RequestException:
      return None

def main():
    html = get_one_page(url)
    print(html)

if __name__ == '__main__':
    main()

打印出来的错误信息：
C:\Users\lijin\AppData\Local\Programs\Python\Python39-32\python.exe E:/QMDownload/pythonProject/main.py
Traceback (most recent call last):
File "E:\QMDownload\pythonProject\main.py", line 37, in <module>
    main()
File "E:\QMDownload\pythonProject\main.py", line 33, in main
    html = get_one_page(url)
File "E:\QMDownload\pythonProject\main.py", line 25, in get_one_page
    response = requests.get(url=url,headers=headers,cookies=cookies)
File "C:\Users\lijin\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
File "C:\Users\lijin\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
File "C:\Users\lijin\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\sessions.py", line 528, in request
    prep = self.prepare_request(req)
File "C:\Users\lijin\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\sessions.py", line 444, in prepare_request
    cookies = cookiejar_from_dict(cookies)
File "C:\Users\lijin\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\cookies.py", line 524, in cookiejar_from_dict
    cookiejar.set_cookie(create_cookie(name, cookie_dict))
TypeError: 'set' object is not subscriptable

Process finished with exit code 1

青出于蓝 发表于 2021-10-1 11:49:07

import requests
from requests.exceptions import RequestException
import re

url = f"https://maoyan.com/board/4"
headers = {
    'Content-Type': 'text/html; charset=utf-8',
    'Origin': r'https://maoyan.com',
    'Referer': r'https://maoyan.com/board/4',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
}
cookies={"Cookie":"""BAIDUID=40CCF4A88D7A40027303B674BBBDC765:FG=1; BIDUPSID=40CCF4A88D7A40027303B674BBBDC765; PSTM=1630132482; HMACCOUNT=D52E9D01CA6F704F; __yjs_duid=1_bc89b7a9955e4efdc44cb50fde742e8c1630141832244; BDORZ=FFFB88E999055A3F8A630C64834BD6D0; ab_sr=1.0.1_ZDY3YTFlYTEyOTQyN2FiMTczZDdmNDhkNDQ4ZmVjNDYyOThmMDI0MzdiOTJjYzBmZGVjZDE5YTc3MjJiYzE3OGY4ZDU0YTI4ZTBjZDVhZjVlMTY2NGViMGM1NGE2YjhiNmY4NDQ2YmFmMjBmNGExNjM4NDMwMWRkMTBkYzY1ZDgyODVhZmQ2ZWQzZjYyMTJmMmIzYTQ4NjkxMDZhYzlkNA==; H_PS_PSSID=34648_34068_31660_34655_34711_34524_34584_34707_34517_26350_34423_22160_34691_34678"""}

def get_one_page(url):
    try:
      response = requests.get(url,headers=headers,cookies=cookies)
      if response.status_code == 200:
            return response.text
      return None
    except RequestException:
      return None

def main():
    html = get_one_page(url)
    print(html)

if __name__ == '__main__':
    main()

青出于蓝 发表于 2021-10-1 11:50:42

两个错误哈。
第一个 requests的get函数内第一个参数直接就是url，不是url=url
第二个 cookie的格式错了哈

寄安 发表于 2021-10-1 11:53:09

青出于蓝 发表于 2021-10-1 11:50
两个错误哈。
第一个 requests的get函数内第一个参数直接就是url，不是url=url
第二个 cookie的格式错了 ...

嗯，好的谢谢回复

寄安 发表于 2021-10-1 11:57:52

寄安 发表于 2021-10-1 11:53
嗯，好的谢谢回复

简直了，谢谢你，困扰了我三天的问题，我一直在改，刚开始我还以为是网站网页压缩了，哎，这次总算运行对了，万分感谢

寄安 发表于 2021-10-1 12:09:27

青出于蓝 发表于 2021-10-1 11:50
两个错误哈。
第一个 requests的get函数内第一个参数直接就是url，不是url=url
第二个 cookie的格式错了 ...

可以请问一下你，那个Cookie在那里找到的，就像这个BAIDUID=40CCF4A88D7A40027303B674BBBDC765:FG=1;
有点啰嗦，还请你指点一下

寄安 发表于 2021-10-1 12:11:38

青出于蓝 发表于 2021-10-1 11:50
两个错误哈。
第一个 requests的get函数内第一个参数直接就是url，不是url=url
第二个 cookie的格式错了 ...

我就只在这两个位置找的Cookies

寄安 发表于 2021-10-1 12:18:18

青出于蓝 发表于 2021-10-1 11:50
两个错误哈。
第一个 requests的get函数内第一个参数直接就是url，不是url=url
第二个 cookie的格式错了 ...

额，那个原来可以，就是没有你写的那么漂亮，打扰了，
页: [1]
查看完整版本: 设置反爬