August 28, 2025
TL;DR
The bot vs. anti-bot arms race is over for legitimate automation. Anchor Browser now speaks Cloudflare’s language for verified bots and optional Pay Per Crawl. That means stable access for builders, policy-enforced control (and revenue) for site owners, and faster, cheaper automation for end users. The Cloudflare Blogpost
Why this matters (now)
For two decades, most bots were treated as adversaries. That forced automation teams to juggle CAPTCHAs, rotating proxies, and brittle workarounds—while websites paid the performance and security tax. With AI agents becoming first-class web citizens, the web is shifting to identify and reward “good bots” rather than blanket-blocking everything. Cloudflare formalized this with Verified Bots and a monetization lane via Pay Per Crawl.
What’s new
Cloudflare-verified browser agent: Anchor Browser signs bot traffic so origins can cryptographically verify who’s calling—no spoofing, no guesswork. (Cloudflare extended its Verified Bots program with message signatures.)
Pay-as-you-crawl (optional): Sites can allow, charge, or block crawlers. If payment is required, origins advertise price via HTTP 402; compliant bots pay and proceed.
How it works
Identify: Anchor Browser includes the headers Cloudflare expects for verified bots and signs requests per their policy. Origins can filter on Cloudflare’s “verified bot” signal.
Authorize: Site owners set rules (allow/charge/block) and, if enabled, a per-request price. Cloudflare enforces this at the edge.
Settle: If a 402 is returned, the bot sees the crawler-price in response headers and completes payment before retrying.
Who benefits
Automation builders:
Stable, CAPTCHA-free sessions; fewer flaky retries.
Lower ops overhead (fewer rotating proxies, fewer brittle scripts).
Better latency (less proxy hop-scotch).
Website owners:
Verified-only access; granular controls per crawler.
A new revenue stream via Pay Per Crawl.
Clear bot analytics (Cloudflare Radar lists verified bots).
Automation users:
Faster outcomes and lower cost—no “proxy tax.”
Get started
If you run bots (Anchor customers): enable Cloudflare verification for your browser agents in your Anchor environment and route traffic through your Cloudflare-protected targets.
If you run a site: enable Verified Bots policies and enroll in Pay Per Crawl to control (and, if you want, monetize) AI/automation access.
Where this goes next
The web is moving toward policy-aware, identity-verified automation—and payments-aware access (via mechanisms like HTTP 402). Expect richer categories for “good bots,” better cryptographic attestations, and cleaner monetization rails. We’ll keep shipping so Anchor customers get these capabilities natively as they land.
References
Cloudflare Verified Bots & categories; cryptographic message signatures; and Pay Per Crawl (including HTTP 402 and crawler-price headers) are documented in Cloudflare’s docs and launch posts. The Cloudflare Blog