Mobile Proxy for Crawl4AI: Setup & Rotation

Crawl4AI takes proxy settings per request through CrawlerRunConfig, supports authenticated endpoints out of the box, and ships a built-in rotation strategy. This guide wires it to a dedicated 4G/5G mobile IP and explains when that single carrier IP does the job better than a rotating pool.

8 min read·Python·Last updated: July 2026

Quick Answer

Pass a ProxyConfig to CrawlerRunConfig(proxy_config=...). It accepts a server plus username / password fields, a plain dict, or a URL string like http://user:pass@host:port. For pools, wrap a list of proxies in RoundRobinProxyStrategy and set proxy_rotation_strategy. With a dedicated mobile IP you keep one endpoint and rotate the IP itself through the provider API instead.

→Proxy config is per request on CrawlerRunConfig, not global on the browser
→ProxyConfig.from_string() and from_env() parse authenticated endpoints for you
→One dedicated carrier IP behind CGNAT often outlasts a cheap rotating pool

Crawl4AI has become a default choice for teams collecting web data for LLMs: it is an open-source, LLM-friendly crawler with roughly 72,000 GitHub stars as of July 2026, at version 0.9.1 on PyPI. It renders pages with a real browser, outputs clean Markdown for RAG ingestion, and runs local-first, which is exactly why its traffic needs a credible egress IP. A crawler run from a datacenter box announces itself by its address range before the first page loads. The fix is the same one we cover across web scraping solutions generally: route the browser through a mobile carrier IP that websites treat as a regular phone subscriber.

Where the proxy config lives

Current Crawl4AI versions attach the proxy to CrawlerRunConfig, so every arun() call can use a different exit. The older pattern of passing proxy_config to BrowserConfig is deprecated in the official docs in favor of this request-level control. proxy_config accepts three equivalent forms: a ProxyConfig object, a dict, or a bare URL string.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig

# Placeholder credentials - substitute your own from the dashboard
run_config = CrawlerRunConfig(
    proxy_config=ProxyConfig(
        server="http://proxy.mobileproxies.org:8000",
        username="u_4a9c",
        password="p_2X7q",
    )
)

# Equivalent dict form:
# run_config = CrawlerRunConfig(proxy_config={
#     "server": "http://proxy.mobileproxies.org:8000",
#     "username": "u_4a9c",
#     "password": "p_2X7q",
# })

async def main():
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url="https://example.com", config=run_config)
        print(result.success, result.url)

asyncio.run(main())

Authenticated endpoint syntax

You do not have to build the object by hand. ProxyConfig.from_string() parses the common formats documented by the project, including credentials embedded in the URL and SOCKS5:

from crawl4ai import ProxyConfig

# user:pass in the URL (HTTP port)
p1 = ProxyConfig.from_string("http://u_4a9c:p_2X7q@proxy.mobileproxies.org:8000")

# SOCKS5 transport instead of HTTP
p2 = ProxyConfig.from_string("socks5://u_4a9c:p_2X7q@proxy.mobileproxies.org:1080")

# ip:port:user:pass shorthand also parses
p3 = ProxyConfig.from_string("203.0.113.10:8000:u_4a9c:p_2X7q")

All three end up as the same server / username / password fields, so pick whichever matches how you store secrets. If a password contains @ or :, prefer the explicit-fields form over the URL form so nothing needs escaping.

Rotation: two different levers

Crawl4AI documents built-in rotation across a list of proxies: RoundRobinProxyStrategy cycles endpoints per request, which pairs naturally with arun_many(). ProxyConfig.from_env() loads the list from a PROXIES environment variable in ip:port:user:pass comma-separated form.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy

async def main():
    proxies = ProxyConfig.from_env()  # reads PROXIES env var
    strategy = RoundRobinProxyStrategy(proxies)

    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        proxy_rotation_strategy=strategy,
    )

    urls = ["https://httpbin.org/ip"] * len(proxies)
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        results = await crawler.arun_many(urls=urls, config=run_config)
        for r in results:
            print(r.success, r.url)

asyncio.run(main())

With a dedicated mobile proxy the second lever is usually better: keep one endpoint and rotate the carrier IP behind it. A POST to the switch endpoint hands you a fresh IP from the carrier while host, port, and credentials stay unchanged, so your ProxyConfig never changes:

import requests

def rotate_ip(slot="us-mob-01", api_key="YOUR_API_KEY"):
    # New carrier IP; the proxy endpoint itself stays the same
    requests.post(
        f"https://buy.mobileproxies.org/api/v1/proxies/{slot}/switch",
        headers={"Authorization": f"Bearer {api_key}"},
        timeout=15,
    )

# e.g. call rotate_ip() between crawl batches, then continue arun_many()

When to pull which lever, and how often, is its own topic - see IP rotation best practices. As a rule: rotate on a block signal or between logical batches, not on every request.

When one carrier IP beats a rotating pool

Mobile IPs sit behind carrier-grade NAT (CGNAT), so one address is shared by many real subscribers at once. That changes the math on both sides. Anti-bot systems cannot cheaply ban or throttle a mobile IP without hitting legitimate phone users: Cloudflare's October 2025 analysis found CGNAT addresses were rate-limited about three times more often than non-CGNAT addresses despite carrying a lower mean bot rate, which is why it built a CGN classifier specifically to reduce that collateral damage. The practical takeaway for a crawler: a single dedicated carrier IP already carries the trust profile that rotating residential pools try to approximate with volume.

One dedicated mobile IP wins when

Your crawl is sequential or lightly concurrent (the default for local-first RAG ingestion), targets a handful of domains, needs stable sessions or logins, or hits sites that fingerprint IP churn. The IP stays coherent; you rotate it deliberately via the switch API when needed.

A rotating pool wins when

You run high-parallelism arun_many() jobs across many domains at once and per-IP rate limits, not blocks, are the bottleneck. Then feed several dedicated endpoints to RoundRobinProxyStrategy and let Crawl4AI spread the load.

Note what CGNAT does to per-IP rate limits from the crawler's side: the target site sees your requests mixed into ordinary subscriber traffic on that address, so pacing matters more than address count. A steady, human-plausible request rate through one mobile IP tends to hold up; a burst will trip the same per-IP thresholds it would anywhere else. This is the sizing logic we use in RAG pipelines with mobile proxies and, at larger corpus scale, in scraping LLM training data.

Pitfalls to watch

•Setting the proxy on BrowserConfig. The docs mark that pattern deprecated; put proxy_config on CrawlerRunConfig so each request controls its own exit.
•Rotating mid-session. If a site set cookies against one IP, switching the carrier IP mid-crawl can invalidate the session. Rotate between batches, or use the strategy's sticky-session methods (get_proxy_for_session / release_session) to pin a proxy per session.
•Cache masking proxy problems. During setup, use CacheMode.BYPASS as in the rotation example above; otherwise cached results can make a dead proxy look healthy.
•Leaking credentials in logs. Load the endpoint from an environment variable (from_env()) and never print the full proxy URL.
•Ignoring robots.txt and terms. A better IP is not permission. Keep volumes reasonable and respect applicable law and each site's terms.

Sources

Related Guides

LLM Data

Point Crawl4AI at a real carrier IP

A dedicated 4G/5G endpoint with user:pass auth that drops straight into ProxyConfig, plus an API to rotate the carrier IP on demand.

Order a dedicated mobile proxy Web scraping solutions →