How to Find All Href Attributes Using BeautifulSoup

What "finding all hrefs" actually means

On almost every web page the links live inside anchor tags, and the destination of each link sits in the tag's href attribute. Finding all hrefs simply means walking through every anchor on the page and reading that attribute, so you end up with a list of destinations. It sounds trivial, and the happy-path version is, but real pages are messy: some anchors have no href at all, many use relative paths, and others point at fragments or JavaScript handlers. A robust extraction handles all of that without crashing. This handbook walks from the simplest one-liner to a version you can trust on pages you did not write.

Why this is one of the most common scraping tasks

Link extraction is the seed of almost every crawl. Before you can scrape a catalogue, an archive or a set of articles, you usually have to discover the URLs, and those URLs live in hrefs. Whether you are building a sitemap, auditing internal linking for SEO, gathering product pages or following pagination, it begins with finding all the hrefs on a starting page and deciding which ones to follow. Getting this step clean and reliable pays off across the whole project.

What you need before you start

You need Python with the beautifulsoup4 and requests packages installed, and ideally lxml as a fast parser. If BeautifulSoup is not set up yet, the companion handbook on installing it covers that. You also need the HTML to parse, which you either fetch over the network or load from a local file while you develop. Working against a saved copy first is a good habit: it keeps your test runs fast and avoids hammering a live site while you iterate on the code.

The simplest way to get every href

The core idea is to parse the HTML, find every anchor that has an href, and read it. Selecting with href=True is the detail that keeps the loop safe, because it skips anchors without the attribute entirely.

from bs4 import BeautifulSoup

html = """
<ul>
  <li><a href="/about">About</a></li>
  <li><a href="https://example.com/blog">Blog</a></li>
  <li><a name="top">No href here</a></li>
</ul>
"""

soup = BeautifulSoup(html, "lxml")
links = [a["href"] for a in soup.find_all("a", href=True)]
print(links)   # ['/about', 'https://example.com/blog']

Notice the anchor with only a name attribute is silently ignored, so the list comprehension never raises. That single guard removes the most common cause of a crash in link extraction.

Reading hrefs defensively without href=True

Sometimes you want every anchor for other reasons, such as reading its text too, and only conditionally need the href. In that case read the attribute with .get(), which returns None rather than raising when the attribute is missing.

for a in soup.find_all("a"):
    href = a.get("href")
    if href is None:
        continue
    text = a.get_text(strip=True)
    print(text, "->", href)

Both approaches solve the same problem. Use find_all("a", href=True) when you only care about links, and .get("href") when you are inspecting anchors more broadly.

Turning relative hrefs into absolute URLs

A href like /about or ../page.html is meaningless without the page it came from. To make the list usable you resolve each href against the page's own URL with urljoin, which correctly handles relative paths, absolute paths and already-absolute URLs.

from urllib.parse import urljoin

base_url = "https://example.com/docs/index.html"
absolute = [urljoin(base_url, a["href"]) for a in soup.find_all("a", href=True)]
print(absolute)

After this step every entry is a full URL you could pass straight to requests.get(), regardless of how the original markup wrote it.

Fetching the page first, then extracting

In a real project the HTML comes from the network. Fetch it with requests, pass the response text to BeautifulSoup, and extract as above. Sending a realistic user agent helps you avoid the most trivial blocks.

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

url = "https://example.com/docs/"
headers = {"User-Agent": "Mozilla/5.0 (compatible; LinkBot/1.0)"}
resp = requests.get(url, headers=headers, timeout=15)
soup = BeautifulSoup(resp.text, "lxml")
links = {urljoin(url, a["href"]) for a in soup.find_all("a", href=True)}
print(len(links), "unique links found")

Using a set deduplicates links automatically, which matters because pages often repeat the same destination in a header, body and footer.

Filtering the links you actually want

A raw href list usually contains noise: mail links, anchors to fragments, external sites, asset files. Collect broadly, then filter with a clear rule rather than trying to be clever in the selector.

• Internal only: keep links whose domain matches the site you are crawling and drop the rest.
• By path: keep links that start with a section prefix such as /products/.
• By type: drop mailto:, tel: and # fragments that are not pages to crawl.
• By extension: exclude or include file types like .pdf depending on your goal.

Targeting links inside a specific section

If you only want the links in a navigation menu or article body, narrow the search first. Use a CSS selector with soup.select() to scope to a container, then read hrefs inside it: soup.select("nav.main a[href]") returns only the anchors within that menu. Scoping early is cleaner than collecting everything and filtering by location afterwards.

Where proxies enter the picture

Extracting hrefs from one page you already have needs no proxy at all. The moment you start fetching many pages to harvest links from, a single IP becomes a liability: sites rate-limit and block addresses that request too often. Routing your fetches through a rotating proxy pool spreads the requests across many IPs, so link harvesting keeps running instead of stalling on a block.

proxies = {
    "http":  "http://user:pass@proxy-host:port",
    "https": "http://user:pass@proxy-host:port",
}
resp = requests.get(url, headers=headers, proxies=proxies, timeout=15)

Swap in IPs from a rotating endpoint and the same extraction code scales from one page to thousands without your own address bearing the load.

Choosing a proxy type for link crawling

The right type depends on how strict the target is. Tolerant sites and large internal crawls often run fine on affordable datacenter or IPv4 proxies. Sites that distrust datacenter ranges usually need residential or ISP proxies that look like ordinary home traffic, and the most defensive platforms may call for mobile IPs. Start with the cheapest type that survives on your target and step up only if blocks appear.

Who this technique suits

Finding all hrefs is foundational for SEO professionals auditing internal links, developers building crawlers, researchers mapping a site's structure, and anyone gathering URLs to feed a later scraping step. If your task starts with "first I need to discover the pages", this is where it begins.

Common use cases

• Discovering every page to crawl from a category or index page.
• Auditing internal linking and finding orphaned or broken links.
• Collecting pagination and "next page" URLs to follow a series.
• Building a seed list of product or article URLs for deeper scraping.
• Mapping outbound links to understand a site's external references.

Common mistakes to avoid

The classic errors are reading tag["href"] on anchors that have none and crashing, forgetting to resolve relative URLs so the list is unusable, fetching every page from one IP until the blocks start, and scraping so aggressively that a site you depend on starts fighting back. Selecting with href=True, resolving with urljoin, rotating proxies and pacing your requests prevent the lot.

Limitations to keep in mind

BeautifulSoup reads the HTML the server returns, so links injected by JavaScript after load will not appear. If a page builds its menu dynamically, you will need a browser tool such as Selenium to render it first, then parse the result. BeautifulSoup is the right tool for static markup; pair it with a renderer when the links only exist after scripts run.

Recommended proxy providers

Because reliable fetching is what lets link extraction scale, choose a proxy provider with care. The options below are listed fairly, with our featured value pick first.

• Cheapest Proxies is our Featured Value Pick. For developers harvesting links across many pages without overspending, it is a sensible first stop for affordable rotating residential, ISP or datacenter IPs.
• A large residential network is worth considering when your targets are strict sites that demand IPs resembling ordinary home traffic.
• A datacenter-focused provider may suit fast, high-volume internal crawls against tolerant sites where cost matters most.

How to get started

Save a copy of a target page, run the find_all("a", href=True) snippet against it, and confirm you get a clean list. Add urljoin to make every link absolute, then a filter for the links you actually want. Only once that works locally should you wire in requests and a proxy pool to crawl at scale. Building up from a saved page keeps early mistakes cheap.

Key takeaways

Use soup.find_all("a", href=True) to collect every link safely, read the values with a comprehension, and resolve them with urljoin so they are usable. Filter for the subset you need, scope with a selector when you want one section, and remember that JavaScript-injected links need a browser to appear. For one page no proxy is needed; for many pages, rotate IPs through a well-matched pool and pace your requests so the harvesting keeps running.