The Agentic Web: Why AI Agents Need Proxies in 2026

According to the Imperva / Thales 2025 Bad Bot Report, automated bots reached 51% of all web traffic — the first time they surpassed human traffic in a decade. Of that, bad bots accounted for 37%, and 44% of advanced bot traffic targeted APIs rather than web pages.

The takeaway is structural: anti-bot vendors are tuning their defenses for a world where most traffic is non-human, and they err toward blocking anything that cannot prove it is a trusted human or a declared, authorized agent.

The browser-agent category matured fast. OpenAI launched Operator in January 2025 and later folded it into ChatGPT Agent, which posts roughly 87% on browser benchmarks. Anthropic shipped Computer Use and a Quick Mode (reported around 3× faster, March 2026). Open-source browser-use became a popular way to wire any model to a real browser.

Not every bet paid off: Google reportedly wound down Project Mariner (around May 4, 2026), with much of the industry pivoting toward API-first agent access. But the agents that do drive real browsers hit exactly the same anti-bot defenses that scrapers always have — TLS fingerprinting, IP reputation, behavioral analysis.

On July 1, 2025, Cloudflare began blocking AI crawlers by default for new sites. Alongside it, Pay Per Crawl revives the long-dormant HTTP 402 Payment Required status code: publishers set a per-request price and choose, per crawler, whether to Allow, Charge, or Block.

HTTP/1.1 402 Payment Required
crawler-price: USD 0.01
# Publisher charges per request; Cloudflare is Merchant of Record

Cloudflare acts as Merchant of Record, settling payments between crawlers and publishers. Major publishers including TIME, Condé Nast, the Associated Press, and The Atlantic aligned behind the model — a signal that paid, declared access is becoming a first-class path for AI traffic.

Identifying agents is moving beyond a self-reported User-Agent string. Google introduced a Google-Agent user-agent (around March 2026), and a coalition including Google, Amazon, Cloudflare, Akamai, and OpenAI is backing Web Bot Auth — a scheme for cryptographically signing agent requests so sites can verify a bot is who it claims to be.

The emerging divide is between declared, signed agents that announce themselves and get allowlisted, and disguised traffic that tries to blend in with human visitors. Each path has very different infrastructure needs.

Signed, paid access works when a publisher has opted into it. But a vast amount of the open consumer web — retail, travel, marketplaces, local listings — has no Pay Per Crawl deal and simply blocks anything that smells automated. Agents that need this live data still need to look like trusted traffic, and high-trust mobile or residential egress remains the most reliable way to get through carrier-grade anti-bot defenses.

The honest, balanced view: the parallel trend toward signed, paid, API-first access could reduce reliance on disguised IPs over time, especially for content publishers who want to be crawled on their own terms. Proxies and declared access are not enemies — they serve different slices of the agentic web, and most serious operators will end up using both.

Sources

• • Thales / Imperva — 2025 Bad Bot Report
• • Cloudflare — Introducing Pay Per Crawl
• • Cloudflare — Blocking AI crawlers by default
• • OpenAI — Introducing Operator / ChatGPT Agent
• • Anthropic — Computer Use announcement