How to Parse XML with lxml: A Practical Handbook

What you will learn here

XML refuses to go away. Sitemaps, RSS and Atom feeds, SOAP and many API responses, configuration files and countless data exchange formats all speak it, so a scraper or data pipeline sooner or later has to parse some. lxml is the go-to library for this in Python because it is fast, standards-compliant and built on the mature libxml2 engine. This handbook is a focused, practical tour of parsing XML with lxml: loading documents, navigating with XPath, surviving the namespace minefield, and streaming files too big to fit in memory.

We keep the snippets short and real, and finish by connecting parsing to the wider job of fetching XML at scale, where proxies start to matter.

The etree module at a glance

Almost all XML work in lxml goes through lxml.etree. It exposes a tree of elements where each node has a tag, attributes, text and children, and it implements the familiar ElementTree API with many extensions. Whether you load from a string or a file, you end up with the same kind of element tree you can walk, query and modify.

from lxml import etree

That single import is the foundation for everything below.

Loading XML: fromstring versus parse

There are two main entry points, and choosing between them is just about where your data lives:

# XML already in memory
root = etree.fromstring(b"<catalog><book id='1'>Proxy Guide</book></catalog>")

# XML from a file or file-like object
tree = etree.parse("catalog.xml")
root = tree.getroot()

Use fromstring when you have a string or bytes, for instance a downloaded API response. Use parse when reading from a path or stream. Note that parse returns an ElementTree wrapper, so you call getroot() to reach the top element, whereas fromstring hands you the element directly.

Walking the tree without XPath

For simple structures you can navigate with plain iteration and the find and findall helpers:

for book in root.findall("book"):
    print(book.get("id"), book.text)

Iterating an element yields its children, get reads an attribute, and .text reads the node's text. This is enough for shallow documents, but the moment paths get deeper or conditional, XPath is the better tool.

Querying with XPath

XPath is lxml's superpower for XML. The xpath method runs an expression and returns a list of matches, which may be elements, attribute values or text nodes:

root.xpath("//book/@id")        # all id attributes
root.xpath("//book[@id='1']")   # books with id 1
root.xpath("//book/text()")     # the text of each book

Because XPath lets you express conditions, positions and relationships in one string, it replaces a great deal of manual looping and makes your intent obvious to the next reader of the code.

The namespace trap and how to escape it

The single most common XML headache is namespaces. Feeds and many standards declare a default namespace, which silently makes plain tag names fail to match. The fix is to declare the namespace and prefix your tags:

ns = {"a": "http://www.sitemaps.org/schemas/sitemap/0.9"}
urls = root.xpath("//a:loc/text()", namespaces=ns)

Alternatively, when you would rather ignore namespaces entirely, use the local-name() function:

root.xpath("//*[local-name()='loc']/text()")

If an XPath returns nothing while the element is plainly in the document, suspect a namespace before anything else.

Reading attributes and text cleanly

Once you have an element, its data lives in two places. Attributes come from get("name") or the attrib dictionary, and content comes from .text for the direct text or itertext() for everything inside. Knowing which you need keeps extraction predictable, especially in mixed-content documents where text and child tags interleave.

Streaming big files with iterparse

Some XML files are enormous, and loading the whole tree would exhaust memory. iterparse solves this by handing you elements as they are parsed so you can process and discard them:

for event, elem in etree.iterparse("huge.xml", tag="record"):
    process(elem)
    elem.clear()  # free the memory

Calling clear() after each record is the key step; without it the freed elements pile up and defeat the purpose. With it, you can stream through files far larger than your available RAM.

Validating against a schema

When you control the contract, lxml can validate documents against an XML Schema or DTD before you trust them. Loading an XSD and calling its validate method tells you whether the document conforms, which is valuable for catching malformed feeds or partner data early rather than letting bad records flow downstream.

Common real-world targets

• XML sitemaps list a site's URLs, perfect for discovering pages to crawl.
• RSS and Atom feeds deliver structured article metadata.
• SOAP and XML APIs return data that lxml walks cleanly.
• Configuration and data files in countless internal formats.

Who this matters to

Data engineers ingesting partner feeds, SEO teams parsing sitemaps to map a site, researchers harvesting structured open data, and automation builders consuming XML APIs all rely on the same parsing skills. The techniques here scale from a one-off script to a production ingestion pipeline.

A parsing checklist

• Is my XML in memory (fromstring) or a file (parse)?
• Have I inspected and declared the document's namespaces?
• Am I using XPath for anything beyond the shallowest path?
• For large files, am I using iterparse and clearing elements?
• Do I validate against a schema when the contract matters?
• Have I added a proxy before fetching feeds at volume?

Common mistakes to avoid

The recurring pitfalls are ignoring namespaces and then puzzling over empty results, loading giant files fully into memory instead of streaming them, forgetting to call getroot() after parse, treating untrusted XML without considering entity-expansion safety, and assuming lxml fetches documents when it only parses what you supply. Each is straightforward to avoid once you know it exists.

Fetching XML safely and at scale

Parsing is only half the story. To get XML you usually download it with requests or httpx, and when you pull many sitemaps, feeds or API responses on a schedule, requests from a single IP run into rate limits and blocks. Routing those fetches through rotating proxies distributes them across many addresses, so your ingestion keeps running without tripping a target's defences.

Recommended proxy providers

lxml handles the parsing for free; the cost and reliability come from how you fetch. We compare the options on value and fit rather than hype.

Beyond our featured value pick, a few well-known names are worth a fair comparison:

• Bright Data brings a large network and fine-grained controls, a fit for big teams needing breadth at a premium.
• Smartproxy offers approachable setup and clear documentation, a sensible step up as your jobs grow.
• Oxylabs supports high-volume ingestion with broad coverage and strong support when uptime leads.

Whichever you shortlist, trial each on your actual feeds and weigh success rate against cost before deciding.

How to get started

Install lxml with pip, grab a small XML file or a public sitemap, and load it with parse or fromstring. Inspect the root for namespaces, write a couple of XPath queries, and once that works, switch a large file over to iterparse. When you move to fetching feeds in bulk, add a proxy and a polite request rate, then scale up while watching for rate-limit responses.

Key takeaways

Parsing XML with lxml means loading with fromstring or parse, navigating with XPath, taming namespaces with a prefix map or local-name(), and streaming oversized files with iterparse. The library parses but does not fetch, so pair it with a value-focused proxy provider and disciplined request habits to collect sitemaps, feeds and API data reliably and at a price that scales.