How to Find All URLs Using Selenium: A Practical Handbook

What you will learn here

Finding every URL on a page is one of the first tasks anyone meets when they start automating a browser, and Selenium makes it deceptively easy to get a first result. The harder part is getting a complete, clean and de-duplicated list from real pages that load links lazily, hide them behind clicks or render them entirely in JavaScript. This handbook walks through the core technique, then layers on the practical refinements that separate a toy script from a dependable link collector. We keep the snippets short and focused so you can adapt them to your own targets without wading through boilerplate.

The examples use Python because it is the most common pairing with Selenium, but every idea maps directly to the Java, C#, JavaScript and Ruby bindings, since they all share the same WebDriver model underneath.

Why Selenium for link discovery at all

If a page ships its links in the raw HTML, you rarely need a full browser; a simple HTTP request and an HTML parser will be faster and far lighter on resources. Selenium earns its place when the links you want only exist after JavaScript executes, after the user scrolls, or after a button is pressed. Because Selenium drives a real browser, it sees the page the way a human does, including everything the scripts add to the DOM after the initial load. That fidelity is the whole reason to accept its extra weight.

The core technique in one query

At its simplest, finding all URLs means selecting every anchor element and reading its destination. In Selenium that is two steps: locate the elements, then pull the href attribute from each.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com")

anchors = driver.find_elements(By.TAG_NAME, "a")
urls = [a.get_attribute("href") for a in anchors]
print(len(urls), "links found")

That is the heart of it. find_elements returns a list of every matching element, and get_attribute("href") asks the browser for the fully resolved, absolute URL of each link. Everything else in this guide is about making that list complete and trustworthy.

Why read the attribute, not the text

A beginner mistake is to read the visible text of a link and treat it as the URL. The text is only the label a user clicks; the real destination lives in the href attribute. Crucially, when you call get_attribute("href"), Selenium returns the address already resolved against the page's base URL, so a relative path like /about comes back as a full https://... address you can follow immediately. Reading the attribute saves you from rebuilding absolute URLs by hand.

Filtering out the noise

A raw anchor sweep picks up plenty of links you probably do not want: empty hrefs, in-page anchors that start with a hash, javascript: handlers, and mailto: or tel: links. A short filter keeps only real navigable web addresses.

clean = [
    u for u in urls
    if u and u.startswith(("http://", "https://"))
]

Tailor the rule to your goal. A site crawler usually wants only on-domain links, while a link auditor may want every external destination too. Decide early what counts as a URL for your task and bake it into the filter.

Removing duplicates cleanly

Pages repeat links constantly, in navigation, footers and inline references, so a flat list is full of repeats. The simplest fix is to collect into a set rather than a list.

unique = set()
for a in driver.find_elements(By.TAG_NAME, "a"):
    href = a.get_attribute("href")
    if href and href.startswith("http"):
        unique.add(href)

For sharper deduplication, normalise each URL first by stripping the fragment after a hash and any trailing slash, so links that point to the same resource collapse into one entry. Over-normalising can merge genuinely different pages, so test your rule on real output before trusting it.

Capturing links that load with JavaScript

Many modern pages add links after the initial render, so querying too early returns an incomplete list. The fix is to wait for the content rather than guess at a fixed delay. An explicit wait pauses only as long as needed for the elements to appear.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.TAG_NAME, "a"))
)

Explicit waits are far more reliable than a blanket sleep, because they react to the real state of the page instead of hoping a guessed pause is long enough. Wait for a specific element you know appears late, then run your anchor sweep.

Handling infinite scroll and lazy loading

Feeds and listing pages often reveal more links only as you scroll. To collect them all, scroll to the bottom, wait for new content, and repeat until the page stops growing.

last = 0
while True:
    driver.execute_script(
        "window.scrollTo(0, document.body.scrollHeight)")
    WebDriverWait(driver, 5).until(lambda d: True)
    height = driver.execute_script(
        "return document.body.scrollHeight")
    if height == last:
        break
    last = height

After the loop finishes, run your usual anchor query once on the fully expanded page. The same idea applies to "load more" buttons: click them in a loop until the control disappears, then harvest the links.

Finding links inside a specific region

Sometimes you only want links from one part of the page, such as a results list or a sidebar, not the whole document. Scope the search by finding the container first, then querying anchors within it.

container = driver.find_element(By.CSS_SELECTOR, "main.results")
local = container.find_elements(By.TAG_NAME, "a")

Scoping the query keeps boilerplate navigation and footer links out of your dataset and makes the result far more relevant for tasks like collecting only the search results on a page.

Going further: a small crawl

Finding links on one page is the building block of crawling a site. The pattern is a queue: collect on-domain links from the current page, add any you have not visited to the queue, and repeat. Keep a visited set so you never process the same page twice, and respect a sensible depth or page limit so the crawl terminates. This is where careful filtering pays off, because a leaky on-domain rule can send your crawler wandering across the whole web.

Who this guide suits

The techniques here serve SEO teams auditing internal links and finding broken destinations, researchers mapping the structure of a site, data engineers building scrapers for listing pages, and automation builders who need to enumerate everything reachable from a starting point. Anyone whose target relies on JavaScript-rendered navigation will find Selenium the natural tool for the job.

Common use cases

• Auditing a site's internal links and surfacing broken ones.
• Collecting every product or article link from a listing or feed.
• Building a sitemap or link graph of an unfamiliar site.
• Gathering outbound links for competitive or backlink research.
• Seeding a wider crawler with a clean set of starting URLs.

A checklist before you scale

• Am I waiting for dynamic links before I query, not racing the page?
• Have I filtered out empty, hash, mailto and javascript links?
• Am I deduplicating with a set or a seen-list as needed?
• Do I scope the search when I only want one region's links?
• Does my on-domain rule actually keep the crawl on the target site?
• Is a proxy attached before I follow many links from one IP?

Common mistakes to avoid

The frequent pitfalls are querying anchors before JavaScript has added them, reading link text instead of the href attribute, forgetting to deduplicate and drowning in repeats, treating relative paths as if they were complete URLs, and scaling a crawl from a single IP until the target throttles you. Each is simple to correct once you know to watch for it, and most reveal themselves the first time you run the script against a real, dynamic page.

Where proxies fit in

Selenium discovers links by driving a browser, and every page it loads is a real request to the target server. Find a handful of URLs and nobody notices. Start following the links you discover, crawling page after page from a single address, and most sites will throttle or block you. Routing the browser through rotating proxies spreads those requests across many IPs so the site sees ordinary, distributed traffic rather than a hammering crawler. For link discovery and crawling specifically, residential and rotating pools tend to fit best, since they look like genuine visitors.

Recommended proxy providers

Discovering URLs is the cheap half of the job; fetching the pages behind them reliably is where a good proxy pays off. We judge providers on value and fit, not on marketing claims.

Alongside our featured value pick, a few established names are worth a fair look:

• Bright Data brings a very large network and fine-grained targeting, suited to big crawls where breadth justifies a premium.
• Smartproxy pairs straightforward setup with clear documentation, a comfortable middle ground as a crawler grows.
• Oxylabs is built for heavy, sustained jobs with wide coverage and dependable support when uptime matters most.

Whatever you shortlist, run each against your real targets and weigh success rate against cost before you commit.

How to get started

Install Selenium and a matching browser driver, point it at a page you know, and run the two-line anchor sweep to see your first list of URLs. Add a filter and a set to clean it up, then introduce an explicit wait so dynamic links are included. Once the script is solid on one page, attach a proxy and a polite delay, and only then scale up to a crawl while watching your success rate.

Key takeaways

Finding all URLs with Selenium starts with one idea: select every anchor and read its href attribute, which the browser returns as a clean absolute URL. The craft lies in waiting for JavaScript links, handling scroll and "load more", filtering noise, and deduplicating. Remember that every page Selenium loads is a real request, so pair it with a value-focused, rotating proxy provider and disciplined crawl limits to collect links at scale without getting blocked.