函数内写入全局的list写不进去（不完全）

zh_

使用requests库写了一个小爬虫程序，大概结构是这样

1. import requests
   
2. list1 = [] #这里定义了一个list
   
3. ...
   
4. def test():
   
5.         if 满足条件:
   
6.                 list1.append(a) #此处a是我需要的信息
   
7.                 print(a)
   
8.         else:
   
9.                 pass
   
10. def main():
    
11.         test()
    
12.         print(len(list1))

复制代码

现在问题就出来了，print(len(list1))的数量和print(a)的总数量不对，也就是说如果print(a)打印出来60个数据，那么print(len(list1))可能是0，也可能13，这是为什么呢？请大牛指教

塔利班

正常应该一致的，你是不是内容有空的

zh_

塔利班 发表于 2018-10-25 15:48
一般长度不会比列表长度小吧，你是不是内容有空的

a是有内容的，绝对不会是空的，我马上在上传图片了

zh_

写入的内容是代理IP
如图，显示长度只有9，但是打印出来的内容不止9个了吧

塔利班

你还是发代码吧，让我们看啥呢

zh_

代码附上，本菜正在学习一些内容，代码不尽如人意，欢迎大佬指出，本菜会虚心学习

zh_

塔利班 发表于 2018-10-25 15:50
你还是发代码吧，让我们看啥呢

发了···刚顺序搞错了

塔利班

如果有识图出代码的我就用了，你复制出来很麻烦吗，，

塔利班

顺带把你的获得代理的函数补全

zh_

塔利班 发表于 2018-10-25 16:02
顺带把你的获得代理的函数补全

把后缀名改成.py就可以了（文件上传限制后缀名）

塔利班

整的好神秘

zh_

塔利班 发表于 2018-10-25 16:10
整的好神秘

省了复制粘贴的时间了，直接文件走起

塔利班

多线程我也没用过，不过现象可以看得出来，当你运行到print('valid_ip length is : '+ str(len(valid_ip)))
时候，有些线程明显没运行到添加打印，而是当然打印的是当时添加进去的长度
后面还有陆续打印的内容

zh_

塔利班 发表于 2018-10-25 16:26
多线程我也没用过，不过现象可以看得出来，当你运行到print('valid_ip length is : '+ str(len(valid_ip))) ...

是的，我也注意到这个问题，实际打印的length是在print(len(valid_ip))之前就显示出来的个数，应该是实际上把数据append进去了，不过是print(len(valid_ip))的时间早了

塔利班

1. import time,random,requests,datetime,threading
   
2. from bs4 import BeautifulSoup as bs
   
3. 
   
4. ip_pool = []        #存放爬取的ip
   
5. valid_ip = []        #存放经过验证可用的ip
   
6. 
   
7. agents = [
   
8.         "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0; Baiduspider-ads) Gecko/17.0 Firefox/17.0",
   
9.     "Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
   
10.     "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)",
    
11.     "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.0 Safari/532.5",
    
12.     "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.310.0 Safari/532.9",
    
13.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7",
    
14.     "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14",
    
15.     "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14",
    
16.     "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
    
17.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.27 (KHTML, like Gecko) Chrome/12.0.712.0 Safari/534.27",
    
18.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1",
    
19.     "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2",
    
20.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
    
21.     "Mozilla/5.0 (Windows; U; Windows NT 6.0 x64; en-US; rv:1.9pre) Gecko/2008072421 Minefield/3.0.2pre",
    
22.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4",
    
23.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10",
    
24.     "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)",
    
25.     "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5",
    
26.     "Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E)",
    
27.     "Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; BIDUBrowser 7.6)",
    
28.     "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko",
    
29.     "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0",
    
30.     "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.99 Safari/537.36",
    
31.     "Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; Touch; LCJB; rv:11.0) like Gecko",
    
32.     "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
    
33.     "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
    
34.     "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0",
    
35.     "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110622 Firefox/6.0a2",
    
36.     "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
    
37.     "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b4pre) Gecko/20100815 Minefield/4.0b4pre",
    
38.     "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )",
    
39.     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)",
    
40.     "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a",
    
41.     "Mozilla/2.02E (Win95; U)",
    
42.     "Mozilla/3.01Gold (Win95; I)",
    
43.     "Mozilla/4.8 [en] (Windows NT 5.1; U)",
    
44.     "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)",
    
45.     "HTC_Dream Mozilla/5.0 (Linux; U; Android 1.5; en-ca; Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
46.     "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.2; U; de-DE) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/234.40.1 Safari/534.6 TouchPad/1.0",
    
47.     "Mozilla/5.0 (Linux; U; Android 1.5; en-us; sdk Build/CUPCAKE) AppleWebkit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
48.     "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
49.     "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
50.     "Mozilla/5.0 (Linux; U; Android 1.5; en-us; htc_bahamas Build/CRB17) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
51.     "Mozilla/5.0 (Linux; U; Android 2.1-update1; de-de; HTC Desire 1.19.161.5 Build/ERE27) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
52.     "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
53.     "Mozilla/5.0 (Linux; U; Android 1.5; de-ch; HTC Hero Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
54.     "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
55.     "Mozilla/5.0 (Linux; U; Android 2.1; en-us; HTC Legend Build/cupcake) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
56.     "Mozilla/5.0 (Linux; U; Android 1.5; de-de; HTC Magic Build/PLAT-RC33) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 FirePHP/0.3",
    
57.     "Mozilla/5.0 (Linux; U; Android 1.6; en-us; HTC_TATTOO_A3288 Build/DRC79) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
58.     "Mozilla/5.0 (Linux; U; Android 1.0; en-us; dream) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
    
59.     "Mozilla/5.0 (Linux; U; Android 1.5; en-us; T-Mobile G1 Build/CRB43) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari 525.20.1",
    
60.     "Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile_G2_Touch Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
61.     "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
62.     "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
63.     "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Milestone Build/ SHOLS_U2_01.03.1) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
64.     "Mozilla/5.0 (Linux; U; Android 2.0.1; de-de; Milestone Build/SHOLS_U2_01.14.0) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
65.     "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
    
66.     "Mozilla/5.0 (Linux; U; Android 0.5; en-us) AppleWebKit/522  (KHTML, like Gecko) Safari/419.3",
    
67.     "Mozilla/5.0 (Linux; U; Android 1.1; en-gb; dream) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
    
68.     "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
69.     "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
    
70.     "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
71.     "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
72.     "Mozilla/5.0 (Linux; U; Android 2.2; en-ca; GT-P1000M Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    
73.     "Mozilla/5.0 (Linux; U; Android 3.0.1; fr-fr; A500 Build/HRI66) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13",
    
74.     "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
    
75.     "Mozilla/5.0 (Linux; U; Android 1.6; es-es; SonyEricssonX10i Build/R1FA016) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
    
76.     "Mozilla/5.0 (Linux; U; Android 1.6; en-us; SonyEricssonX10i Build/R1AA056) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1"]
    
77. 
    
78. 
    
79. def get_proxies_ip():        #获取代理IP
    
80. 
    
81.         page = input('请输入想要爬取代理IP的页数(不超过3459):')
    
82.         page = int(page)
    
83.         sleep_time = input('请输入每次爬取后休眠的时间(单位：秒):')
    
84.         sleep_time = int(sleep_time)
    
85.        
    
86.         for num in range(1,page+1):
    
87.                 url = 'http://www.xicidaili.com/nn/' + str(num)
    
88.                 headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
    
89.        
    
90.                 res = requests.get(url, headers=headers)
    
91.        
    
92.                 soup = bs(res.text, 'lxml')
    
93.                 a = (soup.find(name='table', id='ip_list')).find_all('tr')
    
94.                 del[a[0]]
    
95.        
    
96.                 for each in a:
    
97.                         b = each.find_all('td')
    
98.                         ip_pool.append((b[1].string + ':' + b[2].string + '@' + b[5].string))
    
99.                 print('爬取完成第' + str(num) + '页')
    
100.                 time.sleep(sleep_time)
     
101. 
     
102. 
     
103. def test_proxies_ip(each):
     
104.         global a
     
105.         url = "http://www.baidu.com"
     
106.         (ip_port, connect_type) = each.split('@')
     
107.         headers = {'User-Agent':random.choice(agents)}
     
108.         if connect_type=='http':
     
109.                 ip_port = 'http://'+ip_port
     
110.                 proxies = {'http':ip_port}
     
111.         else:
     
112.                 ip_port = 'https://'+ip_port
     
113.                 proxies = {'https':ip_port}
     
114.         try:
     
115.                 req = requests.get(url, headers=headers, proxies=proxies, timeout=10)
     
116.        
     
117.                 if req.status_code == 200 :
     
118.                         valid_ip.append(proxies)
     
119.                         print(proxies)###################################################此处可以打印，但是没写入valid_ip
     
120.                 else:
     
121.                         pass
     
122.         except:
     
123.                 pass
     
124.         finally:
     
125.                 a-=1
     
126.         #print(valid_ip)
     
127. 
     
128. 
     
129. get_proxies_ip()
     
130. a=len(ip_pool)
     
131. print('ip_pool length is : ' + str(len(ip_pool)))   
     
132. for each in ip_pool:
     
133.         t = threading.Thread(target=test_proxies_ip, args=(each,))
     
134.         t.start()
     
135. while a:
     
136.         pass
     
137. print('valid_ip length is : '+ str(len(valid_ip)))
     
138. 
     
139. 
     

复制代码

塔利班

奇怪，不让我发出来，你这么就能看到了

zh_

塔利班 发表于 2018-10-25 17:00
奇怪，不让我发出来，你这么就能看到了

可以兄弟，这样很方便