Automating browsers has been a common need for many scenarios, from testing web application products to advanced web scraping. The selenium WebDriver has made it easy to automate all popular browsers with just a few lines of code, but tasks that seem simple when a human uses a browser can be quite difficult during automation.
What's the big deal with page load?
When using a browser, we are never really concerned with the exact moment a page is fully loaded. We can visually see it and make a best-effort guess at what point we can interact with it, and often don't need to wait for a page to be completely idle before using it.
But what if you wanted to make automatic screenshot of a website using selenium? Now you are confronted with the problem of needing to wait until the page is fully rendered, or risk receiving a partially loaded page on the screenshot. To make matters worse, websites consist of many different parts and technologies: css stylesheets, javascript code that runs in the browser, media files like images and videos, ... the list goes on. How do you know when to take that screenshot?
Using a static timeout
The first solution that comes to mind is simply wait a specific duration between navigating to the page and interacting with it:
from selenium import webdriver
from time import sleep
driver = webdriver.Firefox()
driver.get("https://example.com")
sleep(5) # Wait 5 seconds
Using timeouts like a simple sleep() call can be enough to allow most websites to load, but it comes with two problems: Slow websites may need more time than the timeout, and faster websites will complete much earlier than the timeout, wasting resources on nothing. For simple use cases where only a few target websites need to be loaded and interacted with, using high timeouts of 10s to 30s can be enough to get rid of the problem, without causing complexity - but anything that need to interact with many websites or be more reliable needs a different solution.
Waiting for the document to become ready
A different approach to waiting for load completion is to observe the DOM loading state, by waiting for document.ReadyState == "complete" in javascript evaluated in the browser sandbox:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def wait_for_page_load(driver, timeout=30):
def page_loaded(driver):
return driver.execute_script("return document.readyState") == "complete"
wait = WebDriverWait(driver, timeout)
wait.until(page_loaded)
driver = webdriver.Firefox()
driver.get("https://example.com")
wait_for_page_load(driver)
If you are familiar with javascript, this will look familiar to you, and feel like the correct solution. It guarantees that the DOM has finished loading, more specifically:
- the HTML document, linked CSS and Javascript files have been downloaded, parsed and evaluated.
- media files like images and videos are fully loaded and rendered (videos to a "playable" state as defined by their html attributes)
So how is this not good enough? Well, there are two exceptions to the list: images using lazy load and async javascript code. Consider this example:
const contentDiv = document.getElementById('content');
fetch('https://example.com/api/post')
.then((response) => response.json())
.then((data) => {
contentDiv.innerHTML = `<h2>${data.title}</h2><p>${data.body}</p>`;
});
The DOM is considered completely loaded when it executed the fetch() statements, but it doesn't wait for the .then callbacks to receive data and execute as well. They may still be waiting for network responses and change large parts of the website when the DOM loading is complete.
Dynamically loading page contents is a fairly common choice for many websites that deal with frequently changing data or complex UIs, and most of these won't be visually complete by the time the DOM is loaded, making this approach unsuitable for those cases.
Waiting for network to be idle
Since DOM load approach fails on in-flight network requests, the next logical step is to try and observe the network, wait for requests to stop, then consider the page loaded:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from time import time
def wait_for_network_idle(driver, timeout=30, max_requests=0, wait_time=0.5):
def check_network_idle(driver):
# Get active network requests count using Performance API
script = """
let performance = window.performance || {};
let timing = performance.timing || {};
return {
'pending': window.performance.getEntriesByType('resource')
.filter(r => !r.responseEnd).length,
'completed': window.performance.getEntriesByType('resource').length
};
"""
start_time = time()
last_requests = None
while time() - start_time < wait_time:
current = driver.execute_script(script)
if current['pending'] <= max_requests:
if last_requests == current['completed']:
return True
last_requests = current['completed']
return False
wait = WebDriverWait(driver, timeout)
wait.until(check_network_idle)
driver = webdriver.Firefox()
driver.get("https://example.com")
wait_for_network_idle(driver)
Admittedly, waiting for network idle state works for a majority of websites, but not even this complex solution solves the problem entirely. It suffers from a few subtle but important drawbacks: Network idle only means data has been transferred, but it may still be processed by the browser. The network may become idle even before the DOM is completed loaded, or it could become idle when the browser is still busy executing some heavy javascript code that changes the page layout/contents.
Even worse, pages that exchange information with the server in realtime (live news, chats, streams, ...) may never reach an idle network state, potentially getting stuck waiting for it forever.
So how do you wait for a page to be loaded?
The answer heavily depends on your use case. Only need to compare the new website version looks like the old one after every change? Using a 30s timeout is probably fine. Need more advanced performance testing or end-to-end tests? Use purpose-built tools like lighthouse and playwright that solve the problem for you.
But if you are building a new piece of software that needs to solve the problem itself, for example an API service that lets users make screenshots of arbitrary websites, you are stuck needing to come up with a solution to the page load problem. In this case, opt for an implementation that incorporates all above approaches, specifically ensuring a page has reached DOM ready state "complete" combined with waiting for network to be idle, including a timeout to prevent it waiting forever on realtime websites. It could look like this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
def wait_for_page_load_combined(driver, timeout=30):
def page_loaded(driver):
return driver.execute_script("return document.readyState") == "complete"
def check_network_idle(driver):
script = """
return {
'pending': window.performance.getEntriesByType('resource')
.filter(r => !r.responseEnd).length,
'completed': window.performance.getEntriesByType('resource').length
};
"""
return driver.execute_script(script)['pending'] == 0
try:
# Wait for document ready state
WebDriverWait(driver, timeout/2).until(page_loaded)
# And network idle
WebDriverWait(driver, timeout/2).until(check_network_idle)
except TimeoutException:
# Static fallback if other methods fail
driver.implicitly_wait(timeout)
driver = webdriver.Firefox()
driver.get("https://example.com")
wait_for_page_load_combined(driver)
The above code first waits for the DOM to be completely loaded, then waits for the network to be idle or the 30 second fallback timeout to expire, which ever happens first. It is fairly stable as far as detecting complete page load goes, but even more advanced solutions observing the javascript runtime exist.