How to Extract Text with Formatting Using BeautifulSoup

What this handbook covers

The first thing most people learn about BeautifulSoup is get_text(), and the first frustration is that it returns a flat wall of text with all the paragraph breaks, list structure and spacing collapsed. This guide is about the opposite goal: extracting text that keeps enough of the original formatting to stay readable and useful. We cover the separator option that fixes the most common complaint, the block-aware approach that preserves paragraphs, and structured handling for lists and tables, then explain when to reach for a dedicated converter instead. The snippets are short so you can lift them straight into your scraper.

Throughout, the focus is practical: not the cleanest possible text in the abstract, but text formatted the way your downstream task actually needs.

Why plain get_text falls short

By default, get_text() walks the subtree and concatenates every string it finds with nothing between them. For inline content that is fine, but for block content it is a problem: the end of one paragraph runs straight into the start of the next, list items merge, and the document becomes an unbroken stream. HTML carries its structure in tags rather than whitespace, so when you strip the tags without compensating, the structure vanishes. Recovering it is what this guide is about.

The quick fix: a separator

The single most useful option is the separator argument. It tells BeautifulSoup what to place between the text of adjacent elements as it flattens them. A newline separator instantly turns a dense block into line-separated text.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
text = soup.get_text(separator="\n", strip=True)
print(text)

Adding strip=True trims whitespace from each fragment and drops the empties, so you get clean, readable lines rather than a newline followed by ragged indentation. For many jobs, this one change is all the formatting you need.

Choosing the right separator

The separator is a design decision, not a fixed answer. A space keeps everything on one line but stops words from neighbouring tags fusing together, which suits search indexing. A newline gives line-by-line output good for display or logging. A double newline approximates paragraph spacing. Pick the separator that matches how the text will be consumed, and remember you can post-process further once the basic structure is in place.

Preserving paragraphs properly

A flat get_text() with a separator gets you part way, but it cannot tell a paragraph break from a line break inside a paragraph. For true paragraph structure, select the block elements yourself and extract each separately.

blocks = soup.find_all(["p", "h1", "h2", "h3", "li"])
parts = [b.get_text(" ", strip=True) for b in blocks]
formatted = "\n\n".join(p for p in parts if p)

Here you control the structure: each block becomes its own paragraph, and joining with a blank line preserves the separation the author intended. This block-aware approach is the reliable way to keep article structure intact.

Handling lists with their structure

Lists carry meaning in their bullets and order that a flat extraction throws away. To keep it, walk the list items and format each one yourself.

for ul in soup.find_all("ul"):
    for li in ul.find_all("li", recursive=False):
        print("- " + li.get_text(" ", strip=True))

Prefixing items with a dash for unordered lists, or a number for ordered ones, reproduces the structure in plain text. Using recursive=False on the item search keeps nested lists from being double-counted, which you can then handle with indentation if you need it.

Turning tables into readable text

Tables are the hardest structure to flatten, because their meaning lives in the grid of rows and columns. Iterate the rows and join the cells with a consistent separator so the columns stay aligned in spirit.

for row in soup.select("table tr"):
    cells = [c.get_text(" ", strip=True)
             for c in row.find_all(["td", "th"])]
    print("\t".join(cells))

Joining cells with a tab keeps each row on one line with its columns separated, which is easy to read and trivial to load into a spreadsheet later. For complex tables with merged cells you will need more logic, but this pattern covers the common case well.

Keeping inline emphasis when it matters

Sometimes the bold and italic emphasis carries meaning you want to keep. BeautifulSoup will not preserve it for you in plain text, but you can detect those tags during extraction and wrap their text in markers, for example surrounding bold content with asterisks. This nudges you toward a light Markdown-style output, which is often exactly what content pipelines want.

When to use a dedicated converter instead

If your goal is faithful rich formatting, headings, links, bold, lists and all, hand-rolling it on top of get_text() becomes tedious. At that point a purpose-built HTML-to-Markdown or HTML-to-text library, itself usually built on a parser, will produce cleaner results with less code. Use BeautifulSoup directly for light structure and full control; reach for a converter when you want complete formatting fidelity without writing it yourself.

Cleaning up the result

Even with a good separator, real pages leave artefacts: stray runs of blank lines, leading and trailing whitespace, and the text of script or style tags if you did not exclude them. Strip those tags before extraction, collapse excessive blank lines with a small regular expression, and trim the whole string at the end. A short clean-up pass turns serviceable output into something you would be happy to store or display.

Who this guide suits

This handbook helps content teams turning web pages into clean reading copy, data engineers preparing text for search or analysis, researchers archiving readable versions of public pages, and anyone building a pipeline that needs structured rather than flattened text. The same techniques apply whether you process one page or thousands.

Common use cases

• Converting articles into clean, paragraph-separated reading text.
• Extracting list and table content for spreadsheets or analysis.
• Building readable snippets for search indexing.
• Archiving the meaningful text of pages without the markup.
• Preparing source text for summarisation or content workflows.

A checklist for formatted extraction

• Did I set a separator so words and lines do not fuse together?
• Am I extracting block elements for true paragraph structure?
• Do I handle lists and tables explicitly rather than flattening them?
• Have I removed script and style tags before extracting?
• Did I collapse excess blank lines and trim the result?
• Is a proxy attached before I scrape many pages?

Common mistakes to avoid

The usual traps are calling get_text() with no separator and getting a fused block, iterating over every tag and duplicating text that parents already contain, flattening lists and tables so their structure is lost, forgetting to strip script and style content, and assuming BeautifulSoup fetches pages when it only parses them. Each is quick to fix once you recognise it in your output, and most show up the first time you test on a real article.

Where proxies fit in

It bears repeating: BeautifulSoup does not download anything. It parses HTML you have already fetched, usually with requests or httpx. The formatting work is cheap; the fetching is where scale bites. When you extract formatted text from many pages, repeated requests from one IP get throttled or blocked, so routing those requests through rotating proxies spreads the load and keeps the pipeline moving. For broad scraping, residential and rotating pools tend to fit best because they resemble genuine visitors.

Recommended proxy providers

Formatting the text is the easy half; fetching the pages behind it reliably is where a good proxy earns its keep. We weigh providers on value and fit rather than on marketing.

Beyond our featured value pick, a few established names deserve a fair look:

• Bright Data brings a large network and granular targeting, suited to big content pipelines that accept a premium.
• Smartproxy pairs simple onboarding with clear docs, a comfortable middle ground as scrapers grow.
• Oxylabs handles heavy, sustained extraction with wide coverage and strong support when reliability matters most.

Whichever you shortlist, test each against your real targets and weigh success rate against cost before committing.

How to get started

Install BeautifulSoup with pip, fetch a sample article, and parse it. Try get_text(separator="\n", strip=True) first, then switch to the block-aware approach to preserve paragraphs, and add list or table handling where your pages need it. Once the formatting is right on a single page, attach a proxy and a polite delay, then scale up gradually while watching your success rate.

Key takeaways

Extracting formatted text with BeautifulSoup comes down to controlling structure: a separator stops text fusing, block-aware extraction preserves paragraphs, and explicit handling keeps lists and tables meaningful. Reach for a dedicated converter when you need full rich formatting. Remember BeautifulSoup parses but does not fetch, so pair it with a value-focused, rotating proxy provider and disciplined bandwidth habits for a clean, affordable extraction pipeline.