How to Get Text Using lxml: A Practical Handbook

What this handbook covers

lxml is one of the fastest and most capable HTML and XML parsing libraries in the Python ecosystem, and a huge share of real scraping work ends with the same question: how do I get the text out? This guide is a focused answer to that. We will walk through the handful of methods lxml gives you for reading text, explain when each fits, and show short snippets you can drop straight into a script. Along the way we cover the practical edges that trip people up: whitespace, nested tags, encodings and the difference between a couple of methods that look similar but behave differently.

Text extraction is rarely the hard part of a scraper on its own. The hard part is doing it cleanly at scale without your requests getting blocked, so we close with how lxml fits into a wider pipeline and where proxies come in.

A quick primer on how lxml sees a document

When lxml parses HTML it builds a tree of element objects. Each element can carry text in two places: the text that sits directly inside it before any child tag, and the tail text that follows a child tag before the next one. Understanding this split is the key to predicting what every extraction method returns, because some read only the direct text while others walk the whole subtree.

Once you hold a root element you can navigate with XPath, CSS selectors via cssselect, or plain attribute access, and pull text at any node you reach.

Getting started: parse some HTML

Before extracting anything you need a parsed tree. For HTML, reach for the lxml.html module, which is forgiving of imperfect markup:

from lxml import html

source = """<div class="bio">
  Hello <b>world</b> from <span>lxml</span>.
</div>"""

tree = html.fromstring(source)
node = tree  # the <div> element

From here every technique below operates on node or on results of a query against the tree.

The fastest answer: text_content()

If you simply want all the visible text inside an element, ignoring how it is split across nested tags, call text_content():

print(node.text_content())
# "\n  Hello world from lxml.\n"

This method recursively collects text from the element and every descendant, then concatenates it. It is the right default when you want everything inside a container, such as the full body of an article or the complete label of a button, and you do not care about the internal structure.

.text versus text_content(): the crucial distinction

The single most common confusion with lxml is the difference between the .text attribute and text_content(). The attribute returns only the text that sits directly inside an element before its first child tag:

print(repr(node.text))          # "\n  Hello " only
print(repr(node.text_content())) # the whole subtree

So .text stops at the first nested tag, while text_content() keeps going. If your extraction is mysteriously missing words that appear inside bold or span tags, you are almost certainly reading .text when you meant to call text_content().

Streaming pieces with itertext()

When you want the text as a sequence of fragments rather than one merged string, itertext() yields each piece in document order:

parts = [t for t in node.itertext()]
# ['\n  Hello ', 'world', ' from ', 'lxml', '.\n']

This is handy when you need to process or filter the fragments, for example to drop the contents of script and style tags, or to rejoin the pieces with a custom separator. It gives you finer control than the all-in-one text_content().

Targeting text with XPath

XPath is where lxml shines. Appending /text() to an expression returns the text nodes directly, and //text() gathers descendant text:

tree.xpath('//div[@class="bio"]/text()')   # direct text nodes
tree.xpath('//div[@class="bio"]//text()')  # all descendant text

The result is a list of strings you can then strip, filter and join. XPath also lets you grab text from a precise location, such as the second paragraph or a cell in a specific table column, which is far cleaner than slicing strings by hand.

Using CSS selectors instead of XPath

If you prefer CSS, install cssselect and call cssselect() on the tree to get matching elements, then read their text:

for el in tree.cssselect("div.bio span"):
    print(el.text_content())

Under the hood lxml translates the CSS selector into XPath, so the two are interchangeable. Pick whichever syntax your team already knows.

Cleaning up whitespace

Real HTML is full of indentation and line breaks, so raw text is often messy. A reliable pattern is to strip each fragment, drop the empties, and join what remains:

raw = tree.xpath('//div[@class="bio"]//text()')
clean = " ".join(t.strip() for t in raw if t.strip())
# "Hello world from lxml ."

For heavier normalisation, collapse runs of whitespace with a regular expression. Deciding up front how clean you need the text to be saves a lot of rework downstream.

Handling encodings correctly

Text extraction can produce garbled characters if the encoding is wrong. When you fetch a page yourself, prefer to pass bytes to html.fromstring() and let lxml detect the encoding from the document, rather than decoding to a string first with a guessed charset. For pages with a declared meta charset this usually just works, and it avoids the classic mojibake where accented characters turn into nonsense.

Dealing with messy or broken markup

One reason to choose lxml.html over a strict XML parser is tolerance. Like a browser, it repairs unclosed tags and other common defects automatically, so you rarely need to pre-clean a page. For genuinely broken documents you can pair lxml with a more lenient parser, but the built-in HTML parser copes with the vast majority of real-world pages you will scrape.

Extracting text from many elements at once

Most scraping jobs want text from a list of repeating items, such as every product card or every search result. The pattern is to select the repeating elements, then read text from each:

names = [el.text_content().strip()
         for el in tree.cssselect(".product .title")]

Keeping the selection and the text extraction as separate steps makes the code easy to read and easy to adjust when a site changes its layout.

Who this guide is for

Anyone parsing HTML in Python benefits from knowing these methods. Data engineers building scrapers, SEO teams pulling on-page text, researchers gathering public information, and automation builders extracting fields from rendered pages all face the same text-extraction choices. The methods here apply equally whether you scrape one page or millions.

Common use cases

• Pulling article bodies or product descriptions for content analysis.
• Collecting prices, titles and ratings from listing pages.
• Extracting headings and metadata for SEO audits.
• Gathering reviews or comments for sentiment work.
• Building training datasets from public web text.

A buyer-style checklist for clean extraction

• Did I choose text_content() when I need the whole subtree?
• Am I accidentally reading .text and losing nested words?
• Do I strip and filter fragments before joining?
• Did I pass bytes so lxml can detect the encoding?
• Is my selector specific enough to survive minor layout changes?
• Have I attached a proxy before scaling beyond a few pages?

Common mistakes to avoid

The usual traps are reading .text when you wanted everything, forgetting to strip whitespace and ending up with ragged strings, decoding bytes with the wrong charset and corrupting accented text, writing brittle selectors tied to incidental classes, and assuming lxml fetches pages when it only parses them. Each is quick to fix once you know to look for it.

Where proxies fit in

It is worth being clear: lxml does not download anything. It parses HTML you have already fetched, usually with requests or httpx. The moment you scale from a handful of pages to thousands, the bottleneck shifts from parsing to fetching, because repeated requests from a single IP get throttled or blocked. Routing those requests through rotating proxies spreads the load across many addresses and keeps your extraction running.

Recommended proxy providers

The text extraction is the easy half; reliably fetching the pages is where a good proxy earns its keep. We weigh the options on value and fit rather than marketing.

Beyond our featured value pick, a few established names deserve a fair look:

• Bright Data offers a large network and granular targeting, suited to big teams that need breadth and accept a premium.
• Smartproxy keeps onboarding simple with clear docs, a comfortable middle ground as your scrapers grow.
• Oxylabs handles heavy, high-volume jobs with wide coverage and strong support when reliability is the priority.

Whichever you shortlist, test each against your real targets and weigh success rate against cost before committing.

How to get started

Install lxml with pip, fetch a sample page, parse it with html.fromstring(), and experiment with text_content(), itertext() and a couple of XPath text queries on real markup. Once your extraction is solid on a single page, add a proxy and a polite delay, then scale up gradually while watching your success rate.

Key takeaways

Getting text from lxml comes down to a small toolkit: text_content() for an entire subtree, .text for direct text only, itertext() for fragment-by-fragment control, and XPath or CSS for precise targeting. Clean the whitespace, get the encoding right, and remember that lxml parses but does not fetch. Pair it with a value-focused proxy provider and disciplined bandwidth habits, and you have a fast, affordable text-extraction pipeline.