|
马上注册,结交更多好友,享用更多功能^_^
您需要 登录 才可以下载或查看,没有账号?立即注册
x
- import requests
- import chardet
- from urllib.parse import urlparse
- from bs4 import BeautifulSoup
- import sys
- sys.setdefaultencoding="utf-8"
- def urlsearch(keyword,page):
- url="http://search.51job.com/jobsearch/search_result.php?fromJs=1&jobarea=070300%2C00&keyword="+keyword+"&curr_page="+str(page)
- return url
- page=1
- url=urlsearch("python",page)
- headers={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.3831.407 Safari/537.36"}
- response=requests.get(url,headers=headers)
- response.encoding="gbk"
- text=response.text
- print(text)
- result=BeautifulSoup(text,"lxml")
- result.encode("gbk")
- print(result.text)
- # result.prettify(encoding="utf-8")
- # print(result.text)
复制代码 <div class="e">
<strong>帮助</strong><br>
<a target="_blank">帮助中心</a><br>
<a target="_blank">常见问题</a><br>
<a target="_blank">新手引导</a>
</div>
<div class="e">
<strong>导航</strong><br>
<a target="_blank">网站地图</a><br>
<a target="_blank">职位搜索</a><br>
<a target="_blank">简历中心</a><br>
<a target="_blank">企业名录</a>
</div>
<div class="code c_first">
<img width="80" height="80" src="http://img01.51jobcdn.com/im/2016/code/web_bottom.png" alt="APP下载">
<span><a >APP下载</a></span>
</div>
<div class="code">
<img width="80" height="80" src="http://img01.51jobcdn.com/im/2016/code/weixin.png" alt="官方微信">
<span>官方微信</span>
</div>
<div class="clear"></div>
</div>
<p class="note nag">
<span>未经51Job同意,不得转载本网站之所有招聘信息及作品 | 无忧工作网版权所有©1999-2017</span>
</p>
</div>
</div>
</body>
</html>
Traceback (most recent call last):
File "D:/OFFICE/program/pyqt/ui/spider/test.py", line 21, in <module>
print(result.text)
UnicodeEncodeError: 'gbk' codec can't encode character '\xa0' in position 5287: illegal multibyte sequence
Process finished with exit code 1
这是我爬去的51job的网页,但是通过requests转换成jbk以后,bs4就不认识了。强行报错。
我尝试过更改编码为UTF-8,但是转换后直接成空的了。
求助求助~~
给你一段我用了很久的爬虫代码,能自动分析网页编码的
- # -*- encoding: utf-8 -*-
- from __future__ import unicode_literals,division
- import re,time,requests
- def getHtml(url,delay_time=0.8,code_type=None):
- hea = {'User-Agent':'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36'}
- html = requests.get(url,headers = hea)
-
- if code_type is None:
- pattern=r'<meta +.*charset="?([a-zA-Z0-9\-]+).*/>'
- m=re.search(pattern,html.text)
- code_type=m.group(1) if m else 'utf-8'
-
- code_type=code_type.lower()
- if 'utf8'==code_type:
- code_type='utf-8'
-
- assert code_type in set(['utf-8','gbk','gb18030','gb2312'])
-
- time.sleep(delay_time)
- html.encoding = code_type
- return html.text
复制代码
urllib库我没怎么用过不太熟,你用type看看返回的是不是str,然后html.decode('gbk').encode('utf-8')也行
|
|