关于都豆瓣动态爬虫，应该怎么样去解决

Stubborn · 发表于 2019-4-9 17:13:47

马上注册，结交更多好友，享用更多功能^_^

您需要登录才可以下载或查看，没有账号？立即注册

x

这个是URL:https://movie.douban.com/annual/2018?source=navigation#1
向下滑动是，会动态的加载，这种应该怎么爬虫js加载后的页面数据。
用selenium + 无界面浏览器吗？

<!doctype html>
<html lang="zh-cmn-Hans">
<head>
<meta charset="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no, viewport-fit=cover">
<meta name="apple-mobile-web-app-capable" content="yes">
<link rel="shortcut icon" href="https://img3.doubanio.com/favicon.ico">
<meta name="format-detection" content="telephone=no">
<meta name="url_name" content="movie_annual2018">
<meta name="user_id" content="">
<meta property="og:site_name" content="豆瓣" />
<meta property="og:title" content="【豆瓣2018年度电影榜单】" />
<meta property="og:description" content="汇聚今年最好的电影，这些高分佳作你千万别错过！" />
<meta property="og:url" content="https://movie.douban.com/annual/2018?source=broadcast" />
<meta property="og:image" content="https://img3.doubanio.com/img/files/file-1545322594.jpg" />
<title>豆瓣2018年度电影榜单・携程冠名</title>
<script>
window.ITHIL = {};
ITHIL.isFrodo = 'False' === 'True';
ITHIL.isWechat = 'False' === 'True';
</script>
<script>
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
var hash = '2018' === '2018' ? '6e5dcf7c287704f738c7febc2283cf0c' : '16a14f3002af32bf3a75dfe352478639'
hm.src = "https://hm.baidu.com/hm.js?" + hash;
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
</script>
</head>
<body>
<div id="app"></div>
<script src="https://img3.doubanio.com/f/ithil/31683c94fc5c3d40cb6e3d541825be4956a1220d/js/lib/es5-shim.min.js"></script>
<script src="https://img3.doubanio.com/f/ithil/a7de8db438da176dd0eeb59efe46306b39f1261f/js/lib/es6-shim.min.js"></script>
<script src="https://img3.doubanio.com/dae/cdnlib/libs/jweixin/1.0.0/jweixin.js"></script>
<script src="https://img3.doubanio.com/f/ithil/b92012acc8222b31e7f1307c154fdb90b56d64d1/gen/ithil2018.bundle.js"></script>
<div alt="main-pic" style="display: none">
<img type="hidden" alt="cover" src="https://img3.doubanio.com/img/files/file-1545322594.jpg">
</div>
</body>
</html>

复制代码

wongyusing · 发表于 2019-4-9 18:41:35

https://movie.douban.com/ithil_j/activity/movie_annual2018/widget/9/comments?random=0&start=0&limit=6

复制代码

请求上面这个地址，
修改最后的limit参数就行了

Stubborn · 发表于 2019-4-9 18:43:10

wongyusing 发表于 2019-4-9 18:41
请求上面这个地址，
修改最后的limit参数就行了

这里面好像都是用户评论的数据，我稍微看了一点

账号		自动登录	找回密码
密码			立即注册

关于都豆瓣动态爬虫，应该怎么样去解决

马上注册，结交更多好友，享用更多功能^_^

浏览过的版块