How to Scrape Multiple Pages Using BeautifulSoup

From one page to many: the core idea

BeautifulSoup itself only ever parses a single block of HTML. Scraping multiple pages is therefore not a BeautifulSoup feature but a pattern you build around it: fetch a page, parse it, extract what you want, work out the next page, and repeat. Almost every multi-page scrape reduces to that loop. The interesting decisions are how you discover each page's URL, how you stop, and how you keep the site from blocking you halfway through. This handbook covers each of those in turn, building from a simple page-number loop to a polite, proxy-backed crawl you can trust on a real site.

Why multi-page scraping needs more care than single-page

Pulling data off one page is forgiving; a single request rarely upsets anyone. The moment you loop over dozens or hundreds of pages, two new problems appear. First, you are now making a stream of requests from one address, which sites notice and throttle. Second, small bugs that were harmless on one page, like assuming an element always exists, now crash a long run near the end and waste everything before it. Treating reliability and politeness as first-class concerns, not afterthoughts, is what separates a scrape that completes from one that dies at page forty.

What you need before you start

You need Python with beautifulsoup4, requests and ideally lxml installed. You also need to understand the target's pagination: open it in a browser, click through a few pages, and watch the URL. If a page number appears in the address, you can build URLs directly. If the URL barely changes but a "next" button advances the content, you will follow links instead. Knowing which pattern you face decides the whole approach.

Pattern one: looping over page numbers in the URL

When the URL carries a page number, building each address is a counter loop. Fetch, parse, extract, and stop when a page returns nothing.

import time
import requests
from bs4 import BeautifulSoup

base = "https://example.com/listings?page={}"
headers = {"User-Agent": "Mozilla/5.0 (compatible; PageBot/1.0)"}
rows = []

for page in range(1, 100):                 # a sane upper bound
    url = base.format(page)
    resp = requests.get(url, headers=headers, timeout=15)
    soup = BeautifulSoup(resp.text, "lxml")
    items = soup.select("div.item")
    if not items:                          # empty page = we are done
        break
    for it in items:
        title = it.select_one("h3")
        rows.append(title.get_text(strip=True) if title else "")
    time.sleep(1)                          # be polite between pages

print(len(rows), "items collected")

The empty-page check is the natural stop condition, and the upper bound on the range stops a runaway loop if the site never returns an empty page.

Pattern two: following the "next page" link

When there is no clean page number, let the markup guide you. On each page, find the next-page anchor, resolve its href, and use it as the following request. Stop when the link disappears.

from urllib.parse import urljoin

url = "https://example.com/listings"
seen = set()

while url and url not in seen:
    seen.add(url)
    resp = requests.get(url, headers=headers, timeout=15)
    soup = BeautifulSoup(resp.text, "lxml")
    # ... extract items from this page ...
    nxt = soup.select_one("a.next[href]")
    url = urljoin(url, nxt["href"]) if nxt else None
    time.sleep(1)

The seen set guards against loops where a page links back to one you already visited, and the while condition ends cleanly when there is no next link.

Reusing a session for speed and consistency

Creating a requests.Session() once and using it for every page reuses the underlying connection and carries cookies between requests, which is both faster and more consistent. Set your headers on the session so every request looks the same, and the crawl behaves more like a real browsing session.

session = requests.Session()
session.headers.update(headers)
resp = session.get(url, timeout=15)

Deduplicating data as you collect

Paginated sites frequently repeat items across page boundaries, and following links can revisit content. Deduplicate as you go rather than at the end, using a stable key such as a product ID or URL.

• Keep a set of keys you have already stored and skip any item whose key is in it.
• Prefer a unique field from the data, like an ID or canonical URL, over the visible title, which can repeat.
• Track visited page URLs separately from collected item keys so the two concerns stay clear.

Throttling so the crawl stays welcome

A delay between requests is the cheapest insurance there is. Even a short pause turns a burst that looks like an attack into traffic a site can absorb. Vary the delay a little so it is not perfectly regular, and back off further if you start seeing errors. Patience here protects both the site and your access to it.

Where proxies become essential

For a handful of pages your own IP is fine. For a real crawl it is the bottleneck, because every page comes from the same address and the site soon throttles it. Routing requests through a rotating proxy pool spreads them across many IPs, so the load on any single address stays low and the crawl keeps moving.

proxies = {
    "http":  "http://user:pass@rotating-endpoint:port",
    "https": "http://user:pass@rotating-endpoint:port",
}
resp = session.get(url, proxies=proxies, timeout=15)

With a rotating endpoint, each request can leave from a different IP without changing your loop at all, which is exactly what a long crawl needs.

Choosing the right proxy type for the job

The right type depends on how strict the target is. Affordable datacenter or IPv4 proxies often handle tolerant sites and large crawls well. Stricter consumer platforms tend to need residential or ISP proxies that resemble ordinary home traffic, and the most defensive sites may require mobile IPs. Start with the cheapest type that survives, watch your block rate, and only move to a more trusted type when the data tells you to.

Handling errors so one bad page does not sink the run

Long crawls hit timeouts, transient errors and the occasional malformed page. Wrap each fetch in error handling, retry a failed request on a fresh IP, and skip a page that will not parse rather than letting it crash everything. Logging which pages failed lets you revisit them later without re-running the whole job.

Who this technique suits

Multi-page scraping is the backbone of price monitoring, listing aggregation, SEO competitor research, dataset building and any task where the data spans more than one page. If your target paginates results, this loop is the heart of your scraper.

Common use cases

• Collecting every listing across a paginated catalogue or marketplace.
• Gathering all articles in an archive or blog for analysis.
• Tracking prices or stock across many product pages over time.
• Building a dataset from search results that span dozens of pages.
• Auditing a large site section page by page for SEO purposes.

Common mistakes to avoid

Scrapers fail in predictable ways: no delay between requests so the IP is blocked, no upper bound so an empty page never stops the loop, no visited set so the crawl loops forever, assuming an element exists so one odd page crashes the run, and no proxy so everything routes through a single address. Each of these has a one-line fix shown above, and applying them up front saves hours of debugging later.

Limitations of BeautifulSoup for multi-page work

BeautifulSoup parses the HTML the server returns, so pagination driven entirely by JavaScript, such as infinite scroll that loads content via background requests, will not appear in the raw markup. In those cases you either call the underlying data endpoint directly or render the page with a browser tool like Selenium first, then parse the result. BeautifulSoup remains ideal for server-rendered pagination; pair it with a renderer when the pages only build after scripts run.

Recommended proxy providers

Because a multi-page crawl lives or dies on reliable fetching, pick a proxy provider deliberately. The options below are listed fairly, with our featured value pick first.

• Cheapest Proxies is our Featured Value Pick. For crawling many pages without overspending while you test, it is a sensible first stop for affordable rotating residential, ISP or datacenter IPs.
• A large residential network is worth considering when your targets are strict sites that demand IPs resembling ordinary home traffic across a long crawl.
• A datacenter-focused provider may suit high-volume crawls against tolerant sites where speed and low cost matter most.

How to get started

Identify which pagination pattern the target uses, then adapt the matching loop above against a saved page or two. Add a delay, a visited set and basic error handling, and confirm it completes a small range cleanly. Only then wire in a rotating proxy pool and let it run the full crawl. Building up from a short, local test keeps the inevitable early mistakes cheap and fast to fix.

Key takeaways

Scraping multiple pages is a loop around single-page parsing: build URLs from page numbers or follow next links, stop on an empty page or a missing link, and deduplicate as you go. Politeness and proxies are not optional at scale: pace your requests, keep a visited set, handle errors, and rotate IPs through a well-matched pool so the whole crawl finishes. For JavaScript-driven pagination, reach for a browser tool and then parse with BeautifulSoup.