Setting up a Scrapy proxy is less about one magic setting and more about matching Scrapy's concurrent, middleware-driven design to a pool that can keep up. Because Scrapy fires many requests in parallel, the way you attach and rotate IPs matters as much as which provider you pick.
This page walks through where proxies plug into a spider, the trade-offs between a rotating gateway and raw IP lists, and what to verify before you scale a crawl.
Using proxies with Scrapy
- Get proxy credentials
Obtain host, port and auth from your provider.
- Configure Scrapy
Set the proxy in your Scrapy request or launch options.
- Rotate & retry
Rotate IPs and handle bans/timeouts for reliable runs.
Which proxy type
| Proxy type | When to use it |
|---|---|
| Residential | IPs assigned by ISPs to real home users; hardest to detect, best for sensitive targets. |
| Datacenter | Fast, cheap IPs from cloud/hosting providers; ideal for tolerant targets and high throughput. |
| ISP | Datacenter-hosted IPs registered under ISPs; residential legitimacy with datacenter speed and stable sessions. |
Our verdict
For value-minded scrapers, start with a rotating residential or datacenter gateway you can drop into HttpProxyMiddleware, then only reach for custom rotation logic if bans persist. Test auth, retry handling and per-request assignment on a small crawl before committing to any provider or plan.
Frequently asked questions
Scrapy ships with HttpProxyMiddleware, so you can set the 'proxy' key on a request's meta or export a proxy environment variable and it will route traffic through it. That works for a single static endpoint, but you will want a rotating gateway or custom middleware once you need many IPs.
A rotating gateway is simpler because one endpoint handles IP cycling for you, which suits high-concurrency crawls. Raw IP lists give more control over which address hits which request but push the rotation, health-checking and ban logic onto your own middleware.
Proxies alone rarely fix bans. Check that you are rotating IPs per request rather than reusing one, respecting AutoThrottle, and sending believable headers. If sites are aggressive, residential or mobile IPs usually hold up better than datacenter ranges, but pair them with sensible request patterns.
Most providers accept credentials in the proxy URL or via a Proxy-Authorization header. Scrapy's built-in middleware reads the userinfo from the URL, so embedding username and password there is the least fragile approach. Confirm the format your provider expects before scaling.
Related proxy guides
Proxy Ranked is an independent comparison resource. Labels are qualitative, not numeric rankings.