[已解决]第55讲动动手题1

sharank

import re
import urllib.request
import urllib.parse
from bs4 import BeautifulSoup

def main():
    word = input('请输入搜索关键字: ')
    key = urllib.parse.urlencode({'word':word}).encode('utf-8')
    response = urllib.request.urlopen('http://baike.baidu.com/search/word?%s' % key)
    html = response.read().decode('utf-8')
    soup = BeautifulSoup(html, 'html.parser')

    for each in soup.find_all(href=re.compile('view')):
        content = ''.join([each.text])
        url2 = ''.join(['http://baike.baidu.com/', each['href']])
        response2 = urllib.request.urlopen(url2)
        html2 = response2.read()
        soup2 = BeautifulSoup(html2)

        if soup2.h2:
            content = ''.join([content, soup2.h2.text])
        content = ' '.join([content, '->', url2])
        print(content)

if __name__ == '__main__':
    main()

UnicodeEncodeError: 'ascii' codec can't encode characters in position 37-40: ordinal not in range(128)
为什么运行结果会出错？怎样避免这类错误？

最佳答案

月排行榜 / 总排行榜

zltzlt

2020-8-7 18:39:18

import re
import urllib.request
import urllib.parse
from bs4 import BeautifulSoup

def main():
    word = input('请输入搜索关键字: ')
    key = urllib.parse.urlencode({'word':word}).encode('utf-8')
    response = urllib.request.urlopen('http://baike.baidu.com/search/word?%s' % key)
    html = response.read().decode('utf-8')
    soup = BeautifulSoup(html, 'html.parser')

    for each in soup.find_all(href=re.compile('view')):
        content = ''.join([each.text])
        url2 = ''.join(['https://baike.baidu.com/', urllib.parse.quote(each['href'])])
        response2 = urllib.request.urlopen(url2)
        html2 = response2.read()
        soup2 = BeautifulSoup(html2, 'html.parser')

        if soup2.h2:
            content = ''.join([content, soup2.h2.text])
        content = ' '.join([content, '->', url2])
        print(content)

if __name__ == '__main__':
    main()

跳转到最佳答案楼层

Twilight6

按照这里来改就行，

soup.find_all(href=re.compile("view")) 改成 soup.find_all(href=re.compile("item"))[7:-1]

055课 爬百度百科“网络爬虫”的词条 问题
https://fishc.com.cn/thread-169631-1-1.html
(出处: 鱼C论坛)

参考代码：

import re
import urllib.request
import urllib.parse
from bs4 import BeautifulSoup

def main():
    word = input('请输入搜索关键字: ')
    key = urllib.parse.urlencode({'word':word}).encode('utf-8')
    response = urllib.request.urlopen('http://baike.baidu.com/search/word?%s' % key)
    html = response.read().decode('utf-8')
    soup = BeautifulSoup(html, 'html.parser')

    for each in soup.find_all(href=re.compile("item"))[7:-1]:
        content = ''.join([each.text])
        url2 = ''.join(['http://baike.baidu.com/', each['href']])
        response2 = urllib.request.urlopen(url2)
        html2 = response2.read()
        soup2 = BeautifulSoup(html2)

        if soup2.h2:
            content = ''.join([content, soup2.h2.text])
        content = ' '.join([content, '->', url2])
        print(content)

if __name__ == '__main__':
    main()

zltzlt

import re
import urllib.request
import urllib.parse
from bs4 import BeautifulSoup

def main():
    word = input('请输入搜索关键字: ')
    key = urllib.parse.urlencode({'word':word}).encode('utf-8')
    response = urllib.request.urlopen('http://baike.baidu.com/search/word?%s' % key)
    html = response.read().decode('utf-8')
    soup = BeautifulSoup(html, 'html.parser')

    for each in soup.find_all(href=re.compile('view')):
        content = ''.join([each.text])
        url2 = ''.join(['https://baike.baidu.com/', urllib.parse.quote(each['href'])])
        response2 = urllib.request.urlopen(url2)
        html2 = response2.read()
        soup2 = BeautifulSoup(html2, 'html.parser')

        if soup2.h2:
            content = ''.join([content, soup2.h2.text])
        content = ' '.join([content, '->', url2])
        print(content)

if __name__ == '__main__':
    main()

sharank

zltzlt 发表于 2020-8-7 18:39

确实可以运行了！感谢！
但是还想问一下，为啥这里要用urllib.parse.quote, 而不是each['href']？

sharank

Twilight6 发表于 2020-8-7 18:38
按照这里来改就行，

055课 爬百度百科“网络爬虫”的词条 问题

可以运行，但为啥还有错误？
UnicodeEncodeError: 'ascii' codec can't encode characters in position 34-42: ordinal not in range(128)

zltzlt

sharank 发表于 2020-8-7 20:43
确实可以运行了！感谢！
但是还想问一下，为啥这里要用urllib.parse.quote, 而不是each['href']？

因为 url 中带有中文字符

sharank

zltzlt 发表于 2020-8-7 20:45
因为 url 中带有中文字符

好的，感谢

sharank

Twilight6 发表于 2020-8-7 18:38
按照这里来改就行，

055课 爬百度百科“网络爬虫”的词条 问题

为啥这里是item呀？而且还是[7:-1]

sharank

解决了！！！
其实把re.compile('view')改成re.compile('\#view')就可以了
不过还是得看需求