前言
Selenium很多难解决的问题,我们要首先想到从JS脚本出发,毕竟Selenium还是支持驱动浏览器运行JS脚本的。
这篇文章的内容主要是Selenium日常开发中会遇到的坑,以Java代码为主,当然Python的小伙伴不用担心,这里所有的解决方案都是可以在Python中通用的。
Selenium
主要参考
Selenium使用总结(Java版本):
Selenium准备
chromedriver各版本镜像:
chromedriver版本与chrome客户端对应支持关系:
最新版本截图:
----------ChromeDriver v2.46 (2019-02-01)---------- Supports Chrome v71-73 Resolved issue 2728: Is Element Displayed command does not work correctly with v0 shadow DOM inserts [[Pri-1]] Resolved issue 755: /session/:sessionId/doubleclick only generates one set of mousedown/mouseup/click events [[Pri-2]] Resolved issue 2744: Execute Script returns wrong error code when JavaScript returns a cyclic data structure [[Pri-2]] Resolved issue 1529: OnResponse behavior can lead to port exhaustion [[Pri-2]] Resolved issue 2736: Close Window command should handle user prompts based on session capabilities [[Pri-2]] Resolved issue 1963: Sending keys to disabled element should throw Element Not interactable error [[Pri-2]] Resolved issue 2679: Timeout value handling is not spec compliant [[Pri-2]] Resolved issue 2002: Add Cookie is not spec compliant [[Pri-2]] Resolved issue 2749: Update Switch To Frame error checks to match latest W3C spec [[Pri-3]] Resolved issue 2716: Clearing Text Boxes [[Pri-3]] Resolved issue 2714: ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:15756. Could not start driver. [[Pri-3]] Resolved issue 2722: Execute Script does not correctly convert document.all into JSON format [[Pri-3]] Resolved issue 2681: ChromeDriver doesn't differentiate "no such element" and "stale element reference" [[Pri-3]] ----------ChromeDriver v2.45 (2018-12-10)---------- Supports Chrome v70-72 Resolved issue 1997: New Session is not spec compliant [[Pri-1]] Resolved issue 2685: Should Assert that the chrome version is compatible [[Pri-2]] Resolved issue 2677: Find Element command returns wrong error code when an invalid locator is used [[Pri-2]] Resolved issue 2676: Some ChromeDriver status codes are wrong [[Pri-2]] Resolved issue 2665: compile error in JS inside of WebViewImpl::DispatchTouchEventsForMouseEvents [[Pri-2]] Resolved issue 2658: Window size commands should handle user prompts [[Pri-2]] Resolved issue 2684: ChromeDriver doesn't start Chrome correctly with options.addArguments("user-data-dir=") [[Pri-3]] Resolved issue 2688: Status command is not spec compliant [[Pri-3]] Resolved issue 2654: Add support for strictFileInteractability [[Pri-]] 复制代码
Selenium 滚动至元素
滚动至元素参考:
实现代码片段:
// 获取元素 WebElement element = webDriver.findElement(By.cssSelector(elementsCss)); // 获取元素左上坐标值 Point elementPoint = element.getLocation(); int documentScrollTop = elementPoint.getY(); // 将页面根据元素滚动至合适位置 jsExecutor.executeScript("window.scrollTo(0," + documentScrollTop + ")"); 复制代码
Selenium等待:显示,隐式
参考:
huilansame.github.io/huilansame.…
强制等待
sleep(3) # 强制等待3秒再执行下一步 复制代码
隐性等待
隐形等待是设置了一个最长等待时间,如果在规定时间内网页加载完成,则执行下一步,否则一直等到时间截止,然后执行下一步。注意这里有一个弊端,那就是程序会一直等待整个页面加载完成,也就是一般情况下你看到浏览器标签栏那个小圈不再转,才会执行下一步。
# -*- coding: utf-8 -*- from selenium import webdriver driver = webdriver.Firefox() driver.implicitly_wait(30) # 隐性等待,最长等30秒 driver.get('https://huilansame.github.io') print driver.current_url driver.quit() 复制代码
需要特别说明的是:隐性等待对整个driver的周期都起作用,所以只要设置一次即可,我曾看到有人把隐性等待当成了sleep在用,走哪儿都来一下…
显性等待
显性等待,WebDriverWait,配合该类的until()和until_not()方法,就能够根据判断条件而进行灵活地等待了。它主要的意思就是:程序每隔xx秒看一眼,如果条件成立了,则执行下一步,否则继续等待,直到超过设置的最长时间,然后抛出TimeoutException。
# -*- coding: utf-8 -*- from selenium import webdriver from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By driver = webdriver.Firefox() driver.implicitly_wait(10) # 隐性等待和显性等待可以同时用,但要注意:等待的最长时间取两者之中的大者 driver.get('https://huilansame.github.io') locator = (By.LINK_TEXT, 'CSDN') try: WebDriverWait(driver, 20, 0.5).until(EC.presence_of_element_located(locator)) print driver.find_element_by_link_text('CSDN').get_attribute('href') finally: driver.close() 复制代码
Selenium定位元素后偏差
这是一个奇怪的问题,之所以会出现这个坐标偏差是因为windows系统下电脑设置的显示缩放比例造成的,location获取的坐标是按显示100%时得到的坐标,而截图所使用的坐标却是需要根据显示缩放比例缩放后对应的图片所确定的,因此就出现了偏差。
解决这个问题有三种方法:
1.修改电脑显示设置为100%。这是最简单的方法;
2.缩放截取到的页面图片,即将截图的size缩放为宽和高都除以缩放比例后的大小;
3.修改Image.crop的参数,将参数元组的四个值都乘以缩放比例。
Selenium加载Flash
看服务报告pc端截图重构内ChromeUtil.java如何使用
网上方案:
prefs.put("profile.default_content_setting_values.plugins", 1); prefs.put("profile.content_settings.plugin_whitelist.adobe-flash-player", 1); prefs.put("profile.content_settings.exceptions.plugins.*,*.per_resource.adobe-flash-player", 1); 复制代码
经测试Chrome65+无法使用,无效。
方法一
基本思路:通过Selenium自动访问chrome单个网页的设置页,操作元素,始终允许加载flash。
让Selenium自动选择下面的按钮
这个操作的Demo代码:
package util; import org.openqa.selenium.*; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa.selenium.chrome.ChromeOptions; import org.openqa.selenium.support.ui.Select; import java.io.File; import java.net.URL; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; public class ChromeUtil { /** * 格式化url进入该url设置页 * @param url * @return */ private static String _base_url(String url){ if (url.isEmpty()){ return url; } try { URL urls = new URL(url); return String.format("%s://%s",urls.getProtocol(),urls.getHost()); }catch (Exception e){ return url; } } /** * 元素选择 * @param driver * @param element * @return */ private static WebElement _shadow_root(WebDriver driver, WebElement element){ return (WebElement)((JavascriptExecutor) driver).executeScript("return arguments[0].shadowRoot", element); } /** * 允许网页的flash运行,chrome67版本可行,75版本提示升级flash * @param driver * @param url */ public static void allow_flash(WebDriver driver, String url) { url = _base_url(url); driver.get(String.format("chrome://settings/content/siteDetails?site=%s",url)); WebElement webele_settings = _shadow_root(driver,(((ChromeDriver)driver).findElementByTagName("settings-ui"))); WebElement webele_container = webele_settings.findElement(By.id("container")); WebElement webele_main = _shadow_root(driver,webele_container.findElement(By.id("main"))); WebElement showing_subpage = _shadow_root(driver,webele_main.findElement(By.className("showing-subpage"))); WebElement advancedPage = showing_subpage.findElement(By.id("advancedPage")); WebElement settings_privacy_page = _shadow_root(driver,advancedPage.findElement(By.tagName("settings-privacy-page"))); WebElement pages = settings_privacy_page.findElement(By.id("pages")); WebElement settings_subpage = pages.findElement(By.tagName("settings-subpage")); WebElement site_details = _shadow_root(driver,settings_subpage.findElement(By.tagName("site-details"))); WebElement plugins = _shadow_root(driver,site_details.findElement(By.id("plugins"))); WebElement permission = plugins.findElement(By.id("permission")); Select sel = new Select(permission); sel.selectByValue("allow"); } /** * @param args */ public static void main(String[] args) { System.setProperty("webdriver.chrome.driver", Constants.PATH_Dict.DRIVER_PATH.getValue()); WebDriver webDriver = null; try { // 初始化webDriver ChromeOptions options = new ChromeOptions(); // options.addArguments("--headless"); // 无头模式 // options.addArguments("--no-sandbox"); // Linux关闭沙盒模式 // options.addArguments("--disable-gpu"); // 禁用显卡 webDriver = new ChromeDriver(options); webDriver.manage().window().setSize(new Dimension(1300, 800)); String url = "https://shanghai.fang.anjuke.com/"; // 获取重定向后网址再打开Flash权限 webDriver.get(url); allow_flash(webDriver,webDriver.getCurrentUrl()); webDriver.get(url); Thread.sleep(1 * 60 * 1000); } catch(Exception e) { e.printStackTrace(); } finally { if(webDriver != null) { webDriver.quit(); } } } } 复制代码
方法二
在chrome设置里将所有网站加入flash白名单,但实测selenium会打开新的chrome,不读取通用设置,类似无痕窗口,有空再试试。
总结
- 全局flash加载的设置按钮在selenium不起作用
- 使用pref加载也没有用
禁止javascript
禁止运行javascript还是可以通过pref的:
HashMap<String, Object> chromePrefs = new HashMap<>(2); chromePrefs.put("profile.managed_default_content_settings.javascript", 2); options.setExperimentalOption("prefs", chromePrefs); 复制代码
Selenium调整网页缩放大小
运行js
document.body.style.zoom='0.5'