[已解决]遇到没有尾页的网站，除了for循环有没有其它方法？

jjxx2005

遇到没有尾页的网站，除了for循环有没有其它方法？有尾页时我可以定位总页数，但是没有尾页时，我只能肉眼观察总页数，请问有没有更好的方法？

import requests
from lxml import etree
import os
if not os.path.exists('c:/视频'):
    os.mkdir('c:/视频')
url =  'https://www.qiushibaike.com/video/'
UA = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'}
r1 = requests.get(url=url,headers=UA).text
e1 = etree.HTML(r1)
j = 0
for i in range(1,14):
    if i == 1:
        url = 'https://www.qiushibaike.com/video/'
    else:
        url = 'https://www.qiushibaike.com/video/'+ 'page/'+ str(i) +'/'
    r1 = requests.get(url=url, headers=UA).text
    e1 = etree.HTML(r1)
    mp4 = e1.xpath('//div[@class = "col1 old-style-col1"]//source/@src')

    for mp4 in mp4:
        mp4 = 'https:'+mp4
        j += 1
        后缀 = mp4.split('.')[-1]
        文件路径 = 'c:/视频/'+ str(j) +'.'+后缀
        r1 = requests.get(url=mp4, headers=UA).content
        with open(文件路径,'wb') as fp:
            fp.write(r1)
            print(f'第{j}个视频','下载成功！！！')

最佳答案

月排行榜 / 总排行榜

Twilight6

2020-7-17 12:12:03

爬出页码呗：

import requests
from lxml import etree
import os
if not os.path.exists('c:/视频'):
    os.mkdir('c:/视频')
url =  'https://www.qiushibaike.com/video/'
UA = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'}
r1 = requests.get(url=url,headers=UA).text
e1 = etree.HTML(r1)
j = 0
all_age = int(e1.xpath(r'//a/span/text()')[-2][1:-1])
for i in range(1,all_age+1):
    if i == 1:
        url = 'https://www.qiushibaike.com/video/'
    else:
        url = 'https://www.qiushibaike.com/video/'+ 'page/'+ str(i) +'/'
    r1 = requests.get(url=url, headers=UA).text
    e1 = etree.HTML(r1)
    mp4 = e1.xpath('//div[@class = "col1 old-style-col1"]//source/@src')

    for mp4 in mp4:
        mp4 = 'https:'+mp4
        j += 1
        后缀 = mp4.split('.')[-1]
        文件路径 = 'c:/视频/'+ str(j) +'.'+后缀
        r1 = requests.get(url=mp4, headers=UA).content
        with open(文件路径,'wb') as fp:
            fp.write(r1)
            print(f'第{j}个视频','下载成功！！！')

跳转到最佳答案楼层

Twilight6

爬出页码呗：

import requests
from lxml import etree
import os
if not os.path.exists('c:/视频'):
    os.mkdir('c:/视频')
url =  'https://www.qiushibaike.com/video/'
UA = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'}
r1 = requests.get(url=url,headers=UA).text
e1 = etree.HTML(r1)
j = 0
all_age = int(e1.xpath(r'//a/span/text()')[-2][1:-1])
for i in range(1,all_age+1):
    if i == 1:
        url = 'https://www.qiushibaike.com/video/'
    else:
        url = 'https://www.qiushibaike.com/video/'+ 'page/'+ str(i) +'/'
    r1 = requests.get(url=url, headers=UA).text
    e1 = etree.HTML(r1)
    mp4 = e1.xpath('//div[@class = "col1 old-style-col1"]//source/@src')

    for mp4 in mp4:
        mp4 = 'https:'+mp4
        j += 1
        后缀 = mp4.split('.')[-1]
        文件路径 = 'c:/视频/'+ str(j) +'.'+后缀
        r1 = requests.get(url=mp4, headers=UA).content
        with open(文件路径,'wb') as fp:
            fp.write(r1)
            print(f'第{j}个视频','下载成功！！！')

jjxx2005

Twilight6 发表于 2020-7-17 12:12
爬出页码呗：

感谢版主，另外有个问题像您咨询
all_age = int(e1.xpath(r'//a/span/text()')[-2][1:-1])

[-2]是取下标的倒数第2个，但是后面的切片我不太懂 [1:-1]
我发现
e1.xpath(r'//a/span/text()')[-2]   在页号上面有一个空行
但是按您的方法写成e1.xpath(r'//a/span/text()')[-2][1:-1] 空行就消失了  这个切片是什么原理？ 谢谢

Twilight6

jjxx2005 发表于 2020-7-17 12:58
感谢版主，另外有个问题像您咨询
all_age = int(e1.xpath(r'//a/span/text()')[-2][1:-1])

因为这里有两个不可见字符  比如  '\n13\n' 我切片 [1:-1] 就是把不可见字符给去掉了

jjxx2005

Twilight6 发表于 2020-7-17 15:37
因为这里有两个不可见字符  比如  '\n13\n' 我切片 [1:-1] 就是把不可见字符给去掉了

谢谢回复明白了