环境
依赖安装
pip install flask-cors flask selenium
安装chromedriver
mac下安装selenium+phantomjs+chromedriver
实现代码
1、hook.js
监听 XMLHttpRequest 请求
// 打开链接，复制代码到这里
// https://unpkg.com/ajax-hook@;2.0.3/dist/ajaxhook.min.js// https://unpkg.com/axios/dist/axios.min.js
ah.proxy({
//请求成功后进入
onResponse: (response, handler) => {
if (response.config.url.startsWith('/api/movie')) {
axios.post('http://localhost:5000/receiver/movie', {
url: window.location.href,
data: response.response
})
console.log(response.response)
handler.next(response)
}
}
})
2、main.py
驱动chrome
# -- coding: utf-8 --
from selenium import webdriver
import time
browser = webdriver.Chrome()
browser.get('https://dynamic2.scrape.center/';)
browser.execute_script(open('hook.js').read())
time.sleep(2)
for index in range(3):
print('current page', index)
btn_next = browser.find_element_by_css_selector('.btn-next')
btn_next.click()
time.sleep(2)
browser.close()
browser.quit()
3、server.py
接收数据的服务，可以进一步将数据存入数据库
# -- coding: utf-8 --
import json
from flask import Flask, request, jsonify
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
@app.route('/receiver/movie', methods=['POST'])
def receive():
content = json.loads(request.data)
print(content)
# to something
return jsonify({'status': True})
if name == '__main__':
app.run(host='0.0.0.0', port=5000, debug=True)
参考
</div>
