When you're prototyping a browser agent, one session is enough. But shipping to production almost always means volume: dozens of pages to scrape, hundreds of accounts to monitor, thousands of records to process.
If your agent handles tasks one at a time, the math gets ugly fast. A 30-second task run sequentially over 100 items takes nearly an hour. Run them concurrently and you're done in under a minute.
The Sequential Bottleneck
Most browser agents start with a simple loop:
from anchor_browser import AnchorClient
client = AnchorClient()
results = []
for url in urls:
session = client.sessions.create()
try:
result = extract_data(session, url)
results.append(result)
finally:
session.close()
This works. It's also exactly the pattern that breaks at scale — not just because it's slow, but because the architectural assumption (one task at a time) doesn't match real-world workloads.
The fix isn't just concurrency for speed. Each Anchor session has its own browser context, cookies, and fingerprint. Running tasks in parallel means complete isolation between jobs — no shared state, no cookie leaks between accounts, no fingerprint correlation.
Async Session Pool with asyncio
Anchor's Python SDK is async-native. Here's the core parallel pattern:
import asyncio
from anchor_browser import AsyncAnchorClient
client = AsyncAnchorClient()
async def process_url(semaphore: asyncio.Semaphore, url: str):
async with semaphore:
session = await client.sessions.create()
try:
return await extract_data(session, url)
finally:
await session.close()
async def run_batch(urls: list[str], max_concurrent: int = 10):
semaphore = asyncio.Semaphore(max_concurrent)
tasks = [process_url(semaphore, url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
successful = [r for r in results if not isinstance(r, Exception)]
failed = [r for r in results if isinstance(r, Exception)]
print(f"Done: {len(successful)}/{len(urls)} succeeded, {len(failed)} failed")
return successful
urls = [f"https://example.com/product/{i}" for i in range(50)]
results = asyncio.run(run_batch(urls, max_concurrent=10))
The asyncio.Semaphore is the critical piece. It caps how many sessions are open simultaneously regardless of how many tasks you queue. Set it too high and you risk rate-limiting; set it too low and you leave performance on the table.
Staggering Session Start Times
For some targets, launching 10 sessions at exactly the same millisecond looks suspicious. A small stagger fixes this without meaningfully slowing things down:
async def run_staggered(urls: list[str], max_concurrent: int = 10, stagger_ms: int = 200):
semaphore = asyncio.Semaphore(max_concurrent)
async def delayed_process(i: int, url: str):
await asyncio.sleep(i * stagger_ms / 1000)
return await process_url(semaphore, url)
tasks = [delayed_process(i, url) for i, url in enumerate(urls)]
return await asyncio.gather(*tasks, return_exceptions=True)
With stagger_ms=200 and max_concurrent=10, the first 10 tasks start across 2 seconds — natural enough for most bot-detection systems to treat as organic traffic.
Retry Logic for Transient Failures
At scale, some sessions will fail — target pages go down, bot detection triggers, timeouts fire. The return_exceptions=True flag prevents one failure from killing the entire batch. Add exponential backoff for transient errors:
async def process_with_retry(semaphore: asyncio.Semaphore, url: str, max_retries: int = 3):
last_error = None
for attempt in range(max_retries):
try:
return await process_url(semaphore, url)
except Exception as e:
last_error = e
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt) # 1s, 2s, 4s
raise last_error
Production-Ready Wrapper
Here's a complete pattern that ties everything together:
import asyncio
import logging
from dataclasses import dataclass
from typing import Any
from anchor_browser import AsyncAnchorClient
@dataclass
class TaskResult:
url: str
data: Any = None
error: str | None = None
@property
def ok(self) -> bool:
return self.error is None
async def run_agent_batch(
urls: list[str],
max_concurrent: int = 10,
stagger_ms: int = 150,
max_retries: int = 3,
) -> list[TaskResult]:
client = AsyncAnchorClient()
semaphore = asyncio.Semaphore(max_concurrent)
async def run_one(i: int, url: str) -> TaskResult:
await asyncio.sleep(i * stagger_ms / 1000)
async with semaphore:
for attempt in range(max_retries):
session = await client.sessions.create()
try:
data = await extract_data(session, url)
await session.close()
return TaskResult(url=url, data=data)
except Exception as e:
await session.close()
if attempt == max_retries - 1:
logging.warning(f"Failed after {max_retries} attempts: {url} — {e}")
return TaskResult(url=url, error=str(e))
await asyncio.sleep(2 ** attempt)
tasks = [run_one(i, url) for i, url in enumerate(urls)]
return await asyncio.gather(*tasks)
# Usage
async def main():
urls = [f"https://shop.example.com/product/{sku}" for sku in range(1, 101)]
results = await run_agent_batch(urls, max_concurrent=15, stagger_ms=100)
succeeded = [r for r in results if r.ok]
failed = [r for r in results if not r.ok]
print(f"Processed {len(urls)} URLs: {len(succeeded)} OK, {len(failed)} errors")
for r in failed:
print(f" ✗ {r.url}: {r.error}")
asyncio.run(main())
Tuning Concurrency for Your Workload
A few rules of thumb for setting max_concurrent:
- Task duration matters: Short tasks (<5s) benefit from high concurrency (15–25). Long tasks (>30s) need fewer slots to keep memory and timeout management clean.
- Target site tolerance: Most sites handle 5–10 concurrent requests from distinct fingerprints without triggering rate limits. Beyond that, staggering becomes more important than reducing concurrency.
- Leave headroom for retries: If your plan allows 20 concurrent sessions, set
max_concurrentto 16–17 so retried tasks can acquire slots without waiting behind new ones.
What's Next
Parallel sessions are the foundation. Once you have this pattern solid, the natural next step is adding observability — knowing which tasks are running, which failed, and why. Anchor's session replay and tracing features make that straightforward to layer on top of exactly this architecture.
Start with 5–10 concurrent sessions, measure your throughput and error rates, then tune from there. The bottleneck is almost never the session pool — it's usually the target site or your own parsing logic.
