[已解决]关于爬取小说的章节排序和格式问题

纳兰小寒

写入每一行数据的时候，末尾加个换行符

1. def writeFile(title,content,name):
   
2.     with open(name +'.txt','a',encoding='utf-8')as ganrong:
   
3.         #设置文件编码，避免写入时乱码
   
4.         ganrong.write('\n'+title+'\n')#加个换行符
   
5.         for line in content:
   
6.             ganrong.write(line+'\n')
   
7.     print('%s was writed ..'%title,name)

复制代码

抓到了你说的第二个问题，爬虫返回的是“<title>500 Internal Server Error</title>”
换IP或者换个浏览器头就好了，再或者在采集的时候加个间隔时间

1. def url_open(url):  # 网页打开函数，以防被禁
   
2.     headers = {
   
3.         'User-Agent': rand_ua()}
   
4.     req = requests.get(url, headers=headers)
   
5.     req.encoding = 'utf-8'
   
6.     return req.text
   
7. 
   
8. def rand_ua():
   
9.     headers = ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36"]
   
10.     headers.append('Mozilla/5.0 (Windows NT 6.1; rv:53.0) Gecko/20100101 Firefox/53.0')
    
11.     headers.append('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36')
    
12.     headers.append('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36')
    
13.     headers.append('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/57.0.2987.98 Chrome/57.0.2987.98 Safari/537.36')
    
14.     return headers[rd.randint(0,4)]

复制代码