Using Proxies With Python Requests: A Working Handbook

What this handbook covers

The Python requests library is the most common way people make HTTP calls in Python, and routing those calls through a proxy is a frequent need for scraping, SEO checks, automation and testing from different locations. This handbook walks through the mechanics from the ground up: how the proxies dictionary works, how to authenticate, how HTTPS and SOCKS5 fit in, how to rotate addresses, and how to handle the errors that inevitably appear. The snippets use placeholders like PROXY_HOST and PORT — substitute your provider's real values.

The core idea: the proxies dictionary

Requests routes traffic through a proxy when you give it a proxies dictionary. The keys are URL schemes (http and https) and the values are proxy URLs. At its simplest:

import requests

proxies = {
    "http": "http://PROXY_HOST:PORT",
    "https": "http://PROXY_HOST:PORT",
}

resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
print(resp.json())

The https key is the proxy used to reach HTTPS sites; it does not need to be an HTTPS proxy itself, which is why both values often share the same host and port. The IP-echo endpoint above is a handy way to confirm the request really left from the proxy.

Adding proxy authentication

Most paid proxies require a username and password. The cleanest way with requests is to embed the credentials directly in the proxy URL:

proxies = {
    "http": "http://USER:PASS@PROXY_HOST:PORT",
    "https": "http://USER:PASS@PROXY_HOST:PORT",
}

If your username or password contains characters like @, : or /, URL-encode them so the URL parses correctly:

from urllib.parse import quote

user = quote("my-user")
pwd = quote("p@ss:word")
proxy = f"http://{user}:{pwd}@PROXY_HOST:PORT"

Some providers prefer IP whitelisting instead of credentials. In that case you leave the username and password out and authorise your server's IP in the provider's dashboard.

HTTP versus HTTPS destinations

People often ask whether a proxy that works for HTTP also works for HTTPS. With requests, you simply provide both keys, and the value under https is used for HTTPS targets. Most commercial proxies handle the secure tunnel transparently, so your code rarely needs to change between an http:// and an https:// target — only the dictionary needs both keys present.

Using a session for efficiency

If you make more than one request, a Session reuses the underlying connection and lets you set the proxy once. It is also the natural home for a sticky IP when you want consecutive requests to look like one continuous client:

session = requests.Session()
session.proxies.update({
    "http": "http://USER:PASS@PROXY_HOST:PORT",
    "https": "http://USER:PASS@PROXY_HOST:PORT",
})

session.headers.update({"User-Agent": "my-research-bot/1.0"})

r1 = session.get("https://example.com/a", timeout=15)
r2 = session.get("https://example.com/b", timeout=15)

A session is generally faster than calling requests.get repeatedly because it avoids re-establishing a connection for every call.

Using SOCKS5 proxies

To use a SOCKS5 proxy you need the SOCKS extra, installed with pip install "requests[socks]". Then use a socks5 or socks5h scheme:

proxies = {
    "http": "socks5h://USER:PASS@PROXY_HOST:PORT",
    "https": "socks5h://USER:PASS@PROXY_HOST:PORT",
}

The socks5h form tells the proxy to resolve the destination hostname, so DNS lookups happen at the proxy's location rather than yours. That is usually what you want when the point of the proxy is to appear in another region.

Rotating proxies across requests

There are two broad approaches to rotation. The first is client-side: keep a list of proxy URLs and choose one per request.

import random

pool = [
    "http://USER:PASS@HOST1:PORT",
    "http://USER:PASS@HOST2:PORT",
    "http://USER:PASS@HOST3:PORT",
]

def get(url):
    proxy = random.choice(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)

The second is provider-side: many services expose a single rotating gateway endpoint that hands you a new IP automatically, so your code points at one address and the provider does the rotation. For logged-in work you usually want the opposite — a sticky session on one IP rather than rotation.

Setting timeouts and retries properly

Proxies introduce another hop that can fail, so robust code sets a timeout and retries thoughtfully:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(total=3, backoff_factor=1, status_forcelist=[502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retries))
session.mount("http://", HTTPAdapter(max_retries=retries))

Pair this with a per-request timeout. The retry adapter handles transient transport errors and selected status codes; the timeout protects against a stalled connection.

Handling errors gracefully

Wrap proxied calls so a single bad IP does not crash your run:

from requests.exceptions import ProxyError, ConnectTimeout, RequestException

try:
    resp = requests.get(url, proxies=proxies, timeout=15)
    resp.raise_for_status()
except (ProxyError, ConnectTimeout) as e:
    print("proxy problem:", e)
except RequestException as e:
    print("request failed:", e)

Catching ProxyError and ConnectTimeout separately lets you react to proxy-specific issues — for example, dropping that IP from a pool — while still handling general request failures.

Verifying the proxy actually worked

Before trusting a setup, confirm the outbound IP changed. A quick check against an IP-echo service tells you whether traffic really left through the proxy and from which region. If the returned address is your own, the proxy is being bypassed — usually a scheme, port or credential mistake.

Which proxy type to use with requests

• Datacenter — fast and cheap, ideal for high-throughput scraping of tolerant targets and for testing your code.
• Residential — high trust for sites that filter aggressively; usually billed by bandwidth, so watch data on heavy runs.
• ISP (static residential) — stable addresses that suit long-lived sessions and per-account work.
• Mobile — strongest trust per IP, premium pricing, reserved for the most sensitive targets.

The library does not care which type you use — the proxies dictionary is identical — but the type determines how target sites treat you.

A buyer checklist for proxies you will use with requests

• Does the provider give plain host:port endpoints that drop straight into the proxies dict?
• Is authentication by credentials, IP whitelist, or both — and which suits your server?
• Is there a rotating gateway and a sticky-session option?
• Is SOCKS5 available if your workflow needs it?
• Are the locations you target actually covered?
• Is billing by bandwidth, per IP, or subscription, and does that match your volume?
• Is there a small plan to test integration before scaling?

Best practices

• Always set a timeout and handle proxy-specific exceptions.
• Reuse a Session for multiple calls and for sticky sessions.
• Set a sensible User-Agent and respect each site's terms and rate limits.
• Keep credentials out of source control; load them from environment variables.
• Verify the outbound IP before committing to a long run.

Common mistakes to avoid

The usual slip-ups are forgetting the https key so HTTPS traffic bypasses the proxy, omitting a timeout and letting a dead IP hang the script, failing to URL-encode special characters in credentials, and using socks5 when socks5h would have resolved DNS at the proxy. Hard-coding credentials in committed code is another avoidable risk. Finally, hammering a target with no delay or rotation tends to get even a high-trust IP blocked quickly.

Requests versus other clients, briefly

Requests is synchronous and simple, which is perfect for straightforward proxied calls. For high concurrency you might reach for an async client, and for browser-rendered pages a headless browser is the right tool. But for the large category of plain HTTP and HTTPS fetching through a proxy, requests remains the path of least resistance, and everything in this handbook transfers conceptually to those other tools.

Recommended proxy providers

Here is a sensible way to start a shortlist. We list our Featured Value Pick first for transparency, then a few others to compare fairly.

• Cheapest Proxies (Featured Value Pick) — our value recommendation. It aims to keep entry pricing low while offering the standard host:port endpoints that slot straight into the requests proxies dictionary, which makes it a practical place to test your integration affordably before scaling.
• A residential-focused provider — worth considering when your targets filter datacenter ranges and you need high-trust residential IPs.
• A datacenter-focused provider — strong for fast, high-volume scraping where datacenter IPs are accepted.
• An ISP / static-residential provider — a good fit for stable, long-lived sessions on consistent addresses.

Confirm SOCKS5 availability, authentication method and locations with the provider before committing.

How to get started

Install requests (and the SOCKS extra if needed), drop your provider's endpoint into a proxies dictionary, and run an IP-echo request to confirm the change. Add authentication, a timeout and basic error handling, then build up to a session with retries. Validate everything on a small plan against a tolerant target before pointing it at production work.

Key takeaways

• The proxies dictionary with http and https keys is the whole foundation.
• Embed credentials in the URL and URL-encode special characters.
• Install the SOCKS extra and use socks5h for DNS-at-proxy resolution.
• Use a Session for efficiency, sticky IPs and retries; always set a timeout.
• The proxy type, not the library, determines how target sites treat you.