aiohttp Proxy Setup for Async Web Scraping

The official docs state that aiohttp "supports plain HTTP proxies and HTTP proxies that can be upgraded to HTTPS via the HTTP CONNECT method." In practice that means you always write the proxy URL with an http:// scheme, even when the target site is HTTPS; aiohttp tunnels the TLS connection through the proxy with CONNECT. The proxy is passed to the request method itself:

import asyncio
import aiohttp

# Dedicated mobile proxy slot us-mob-01 (HTTP port)
PROXY = "http://u_4a9c:p_2X7q@proxy.mobileproxies.org:8000"

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://httpbin.org/ip", proxy=PROXY) as resp:
            print(resp.status)
            print(await resp.json())

asyncio.run(main())

Because proxy= is a request-level argument, one session can send some requests through the proxy and others directly, or switch proxies between requests without rebuilding the session. That is a real difference from requests, where a proxies dict usually lives on the session.

Most paid proxies, ours included, use username and password authentication. aiohttp accepts the credentials in three places. The docs confirm that "authentication credentials can be passed in proxy URL", that the classic proxy_auth argument takes an aiohttp.BasicAuth object, and that current versions can send a prebuilt Proxy-Authorization header:

import aiohttp
from urllib.parse import quote

# Option 1: credentials inline in the proxy URL
PROXY = "http://u_4a9c:p_2X7q@proxy.mobileproxies.org:8000"

# URL-encode credentials that contain special characters (@ : / #)
PROXY = f"http://{quote('u_4a9c')}:{quote('p_2X7q')}@proxy.mobileproxies.org:8000"

async def with_proxy_auth(session):
    # Option 2: separate proxy_auth argument with aiohttp.BasicAuth
    # (deprecated since aiohttp 3.14, scheduled for removal in 4.0)
    proxy_auth = aiohttp.BasicAuth("u_4a9c", "p_2X7q")
    async with session.get(
        "https://httpbin.org/ip",
        proxy="http://proxy.mobileproxies.org:8000",
        proxy_auth=proxy_auth,
    ) as resp:
        return await resp.json()

async def with_proxy_headers(session):
    # Option 3: explicit Proxy-Authorization header (aiohttp 3.14+)
    from aiohttp import encode_basic_auth
    proxy_headers = {"Proxy-Authorization": encode_basic_auth("u_4a9c", "p_2X7q")}
    async with session.get(
        "https://httpbin.org/ip",
        proxy="http://proxy.mobileproxies.org:8000",
        proxy_headers=proxy_headers,
    ) as resp:
        return await resp.json()

Whichever option you pick, remember to URL-encode usernames or passwords containing characters like @ or : before embedding them in a URL, otherwise the proxy address parses incorrectly and you get connection errors that look like a dead proxy.

aiohttp's proxy documentation only covers HTTP proxies; SOCKS is not supported natively. The standard solution is the third-party aiohttp-socks package, which describes itself as a "proxy connector for aiohttp" supporting SOCKS4(a), SOCKS5(h), HTTP (CONNECT), and proxy chains. Install it with pip install aiohttp_socks:

import asyncio
import aiohttp
from aiohttp_socks import ProxyConnector

async def main():
    # SOCKS5 port on the same dedicated mobile proxy
    connector = ProxyConnector.from_url(
        "socks5://u_4a9c:p_2X7q@proxy.mobileproxies.org:1080"
    )
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get("https://httpbin.org/ip") as resp:
            print(await resp.json())

asyncio.run(main())

Two behavioral differences matter. First, the proxy now lives on the connector, so every request made through that session goes through the SOCKS proxy; you do not pass proxy= per request. Second, DNS resolution: the ProxyConnector constructor exposes an rdns flag, and the project notes it defaults to True for SOCKS5, meaning hostnames are resolved remotely by the proxy rather than locally - the same distinction as socks5 vs socks5h in other tools. The package also ships a ChainProxyConnector for routing through several proxies in sequence.

If part of your stack is synchronous, the same SOCKS5 endpoint works with the requests library too; our Python requests SOCKS5 proxy example covers that side, including the socks5 vs socks5h DNS behavior in detail.

The docs are explicit that, "contrary to the requests library", aiohttp does not read proxy environment variables by default. If you want HTTP_PROXY / HTTPS_PROXY (and WS_PROXY / WSS_PROXY for websockets, supported since aiohttp 3.8) to apply, opt in with trust_env=True on the session. Per the reference docs, trust_env also picks up proxy credentials from a ~/.netrc file when present.

# Shell: set once, applies to every tool that honors these vars
# export HTTP_PROXY="http://u_4a9c:p_2X7q@proxy.mobileproxies.org:8000"
# export HTTPS_PROXY="http://u_4a9c:p_2X7q@proxy.mobileproxies.org:8000"

import aiohttp

async def main():
    async with aiohttp.ClientSession(trust_env=True) as session:
        async with session.get("https://httpbin.org/ip") as resp:
            print(await resp.json())

This is convenient for containers and CI where the proxy is injected as configuration. The tradeoff is implicitness: a stray environment variable can silently reroute your scraper. For anything production-grade, an explicit proxy= argument is easier to audit.

• •Creating a session per request. The docs suggest one session for the lifetime of the application to benefit from connection pooling. A session per URL throws that away and hammers the proxy with TLS handshakes.
• •Using a socks5:// URL in the proxy argument. Native aiohttp will not handle it; SOCKS must go through the aiohttp-socks ProxyConnector at the session level.
• •Unencoded credentials. Special characters in the username or password break URL parsing; run them through urllib.parse.quote first.
• •Expecting env vars to work. Without trust_env=True, HTTP_PROXY and HTTPS_PROXY are ignored, and requests silently go out over your real IP - verify with an IP echo endpoint before scraping.
• •Unbounded gather. Thousands of coroutines racing through one modem produce timeouts that look like proxy failures. Bound concurrency with a semaphore and set explicit ClientTimeout values.

Sources

• • aiohttp docs - Advanced Client Usage: Proxy support (proxy URL auth, trust_env, proxy_auth deprecation in 3.14)
• • aiohttp docs - Client Reference (proxy / proxy_auth parameters, trust_env, TCPConnector limit, ClientTimeout, session reuse)
• • aiohttp 3.13 docs - proxy_auth with aiohttp.BasicAuth example
• • aiohttp-socks on GitHub - ProxyConnector, supported proxy types, rdns default
• • aiohttp-socks on PyPI - installation and usage