A page can look complete and still be almost empty. Fetch the URL with a plain HTTP request, and you might get a shell of HTML—a few <div> tags, a script bundle, none of the product listings, prices, or comments you actually wanted. Open the same URL in a browser, and everything is there. The gap between those two views is JavaScript rendering, and knowing which side of it you're on is the difference between a scraper that works and one that silently returns nothing.
The Quick Diagnostic Test
Before reaching for any tooling, check what the server actually sends versus what the browser shows. Fetch the raw response and search it for the text or field you need:
import requests
resp = requests.get("https://example.com/products")
raw_html = resp.text
print("price found in raw HTML:", "$" in raw_html and "product-price" in raw_html)If the data is missing from raw_html but visible when you open the page normally, it's being injected by client-side JavaScript after the initial load—via a framework like React or Vue hydrating the DOM, or a fetch call populating a template. A static request will never see it, no matter how you tweak headers or retry.
Check for the Underlying API First
Most JavaScript-rendered pages aren't generating data out of nowhere—they're calling an internal API and rendering the JSON response into HTML. Open your browser's network tab, reload the page, and filter for XHR/fetch requests. If you find the endpoint the page itself uses, call it directly. You get structured JSON instead of markup to parse, no rendering overhead, and a contract that's more stable than the DOM layout sitting on top of it. This is worth ten minutes of investigation before you write a single line of browser automation, because it eliminates the rendering problem entirely when it works.
When You Actually Need a Real Browser
Sometimes there's no clean API to call, or the data only exists after real interaction. That's the case when:
- Content is injected by client-side routing (single-page apps built on React, Vue, or similar) with no separate data endpoint you can isolate.
- The page requires scrolling, clicking, or waiting on a timer—infinite scroll feeds, "load more" buttons, modals that fetch on open.
- Content is gated behind an authenticated session, and the API calls themselves depend on cookies or tokens set up during login.
- The layout or values are computed client-side (canvas-rendered elements, WebSocket-fed live state) rather than delivered as static markup or a fetchable payload.
In these cases you need something that executes JavaScript the way a real user's browser would, waits for the right elements to appear, and then hands you the fully rendered DOM.
Rendering With a Real Session
Anchor gives you a cloud browser session you can drive with Playwright, so you're reading the DOM after it's actually rendered instead of guessing at timing with a headless script you have to babysit:
import os
from anchorbrowser import Anchorbrowser
from playwright.sync_api import sync_playwright
anchor_client = Anchorbrowser(api_key=os.getenv("ANCHOR_API_KEY"))
session = anchor_client.sessions.create()
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp(session.data.cdp_url)
page = browser.new_page()
page.goto("https://example.com/products")
page.wait_for_selector(".product-card")
rendered_html = page.content()If what you actually want is structured data rather than raw HTML to parse yourself, Anchor's agent tasks take a schema instead of a selector list, so the extraction survives a layout change instead of breaking on the next redesign:
from pydantic import BaseModel
class Product(BaseModel):
name: str
price: str
result = anchor_client.agent.task(
"Extract the product name and price from this page",
task_options={
"output_schema": Product.model_json_schema(),
"url": "https://example.com/products/123"
}
)The decision isn't "browser vs. no browser" as a blanket rule—it's diagnosing, page by page, whether the data lives in the initial response or gets built afterward. Static requests are faster and cheaper, so use them whenever the raw HTML or an underlying API already has what you need. Reach for a real browser only when the content genuinely doesn't exist until JavaScript runs. Anchor handles the session, the rendering, and the schema-first extraction for that half of the problem. Get an API key and try it against a page you already know is JavaScript-heavy.
