[已解决]某站批量爬取的问题

tuxiaoqing

目前实现了单个页面的爬取，代码如下：

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. headers = {
   
5. "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
   
6. }
   
7. r = requests.get('https://site.ip138.com/192.168.5.6/',headers=headers)
   
8. html = r.text
   
9. soup = BeautifulSoup(html,'lxml')
   
10. for ul in soup.find_all(attrs={'id':'list'}):
    
11.     for a in ul.find_all(name='a'):
    
12.         print(a.string)

复制代码

现在有一个ip.txt，里面10个ip地址，需要返回这10个ip地址的查询内容并将结果输出为txt（每个ip替换192.168.5.6）

最佳答案

月排行榜 / 总排行榜

疾风怪盗

2020-9-27 09:38:37

是要这样子？

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. headers = {
   
5.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
   
6. }
   
7. iplist = ['192.168.5.6', '192.168.5.5', '119.75.217.109']
   
8. 
   
9. 
   
10. for i in iplist:
    
11.     url = f'https://site.ip138.com/{i}/'
    
12.     print(url)
    
13.     r = requests.get(url=url, headers=headers)
    
14.     html = r.text
    
15.     soup = BeautifulSoup(html, 'lxml')
    
16.     with open('ip.txt',mode='w',encoding='utf-8') as f:
    
17.         for ul in soup.find_all(attrs={'id': 'list'}):
    
18.             for a in ul.find_all(name='a'):
    
19.                 print(a.string)
    
20.                 f.write(a.string)
    
21.                 f.write('\n')
    

复制代码

跳转到最佳答案楼层

疾风怪盗

是要这样子？

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. headers = {
   
5.     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
   
6. }
   
7. iplist = ['192.168.5.6', '192.168.5.5', '119.75.217.109']
   
8. 
   
9. 
   
10. for i in iplist:
    
11.     url = f'https://site.ip138.com/{i}/'
    
12.     print(url)
    
13.     r = requests.get(url=url, headers=headers)
    
14.     html = r.text
    
15.     soup = BeautifulSoup(html, 'lxml')
    
16.     with open('ip.txt',mode='w',encoding='utf-8') as f:
    
17.         for ul in soup.find_all(attrs={'id': 'list'}):
    
18.             for a in ul.find_all(name='a'):
    
19.                 print(a.string)
    
20.                 f.write(a.string)
    
21.                 f.write('\n')
    

复制代码

suchocolate

1. import requests
   
2. from bs4 import BeautifulSoup
   
3. 
   
4. 
   
5. if __name__ == '__main__':
   
6.     headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
   
7.     base_url = 'https://site.ip138.com/'
   
8.     with open('ip.txt') as f:  # 因为不知道你的ip.txt的内容，假定每行1个ip，其他情况你自己改改代码。
   
9.         for ip in f.readlines():
   
10.             url = base_url + ip
    
11.             r = requests.get(url,headers=headers)
    
12.             soup = BeautifulSoup(r.text, 'lxml')
    
13.             with open('result.txt', 'a') as o:
    
14.                 for ul in soup.find_all(attrs={'id':'list'}):
    
15.                     for a in ul.find_all(name='a'):
    
16.                         o.write(a.string)
    

复制代码

tuxiaoqing

您好，一个ip如果对应多个域名，单个ip爬取输出多个域名
然后进行批量，这个有多个域名的ip未输出