新人求助有关爬虫的问题

kaishao

代码如下

1. import urllib.request
   
2. url = r'https://www.3dmgame.com/bagua/1643.html'
   
3. headers =r'User-Agent', 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36'
   
4. req = urllib.request.Request(url,headers)
   
5. response = urllib.request.urlopen(req)
   
6. html = response.read().decode('utf-8')
   
7. print(html)

复制代码

甚是奇怪，为什么会输出：

1. TypeError: can't concat str to bytes

复制代码

falseh

没有爬取到你想爬取的网站

chxchxkkk

import urllib.request
url = r'https://www.3dmgame.com/bagua/1643.html'
headers =r'User-Agent', 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36'
req = urllib.request.Request(url,headers)
response = urllib.request.urlopen(req)

html = reqs.read().decode('utf-8')

print(html)

这个reqs不存在哦，应该是response吧

kaishao

chxchxkkk 发表于 2019-8-10 10:57
import urllib.request
url = r'https://www.3dmgame.com/bagua/1643.html'
headers =r'User-Agent', 'Mo ...

谢谢提醒，貌似改了之后还是会显示

1. TypeError: can't concat str to bytes

复制代码

这网站是不是有问题。。。还是我学错了什么。。。

Stubborn

需要编译成字节,试试在前面加b而不是r，信新人爬虫其实推荐用requests,很方便

1. import requests
   
2. url = 'https://www.3dmgame.com/bagua/1643.html'
   
3. headers ={'User-Agent',:'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36'}
   
4. req = request.get(url,headers=headers).text
   
5. print(req)

复制代码

kaohsing

这样试试：

kaishao

kaohsing 发表于 2019-8-11 06:49
这样试试：

谢谢大佬

waitforlove

1. import urllib.request
   
2. url = r'https://www.3dmgame.com/bagua/1643.html'
   
3. headers ={'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36'}
   
4. req = urllib.request.Request(url,headers=headers)
   
5. response = urllib.request.urlopen(req)
   
6. html = response.read().decode('utf-8')
   
7. print(html)

复制代码