Scrape JavaScript-rendered pages without running a browser

Recognize an empty application shell

Start by searching the downloaded HTML for text that is clearly visible in a browser. If the page title exists but repeated cards, prices, or article text do not, the first response may be only a shell.

Other clues include a nearly empty root element, many script tags, serialized application state, and placeholders that are replaced after load. Do not assume every missing selector is a rendering problem. The page may have redirected, returned an access page, or changed its markup.

Why an HTML parser cannot fix missing content

Beautiful Soup and Cheerio parse a document that already exists. They do not execute JavaScript. Waiting before calling a parser also changes nothing because no browser is running in that process.

The fix belongs in retrieval: obtain the rendered document first, then send that HTML to the same parser and selectors used for a static page.

Choose Auto or Hard mode

Use mode: "auto" when targets vary. Auto tries Easy first and can fall back to Hard when the first result is not usable. Use mode: "hard" when you have confirmed that a target consistently requires browser-level retrieval.

Easy is less expensive at $0.10 per 1,000 successful requests. Hard is $0.20 per 1,000 successful requests. Auto charges only the successful route, never Easy plus Hard for the same final success.

Request rendered HTML

The example uses explicit Hard mode to make the rendering requirement visible in code.

const response = await fetch("https://web-bypass.dev/v1/fetch", {
  method: "POST",
  headers: {
    Authorization: "Bearer " + process.env.WEB_BYPASS_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/app", mode: "hard" }),
  signal: AbortSignal.timeout(90_000),
});

const data = await response.json();
if (!response.ok || !data.success) {
  throw new Error(data.error?.code ?? "FETCH_FAILED");
}
console.log(data.access.used_mode, data.final_url, data.content.length);

On success, parse data.content. Keep data.final_url, data.status_code, and data.request_id with diagnostics so redirects and failures remain traceable.

Validate the rendered result

Browser-level retrieval can still return a login wall, regional notice, consent screen, or a page with different data than expected. Validate one or two fields that prove the useful page loaded.

• Check the expected page heading or record container.
• Reject known access-page titles and empty result shells.
• Confirm the final URL remains within the expected public site.
• Record selector failures separately from retrieval failures.

Avoid running a browser fleet in your app

Self-managed browser workers need process isolation, browser updates, memory limits, crash recovery, navigation timeouts, and concurrency control. An API moves that retrieval layer behind one HTTP contract. Your application still owns target selection, parsing, validation, scheduling, and responsible use.

If you need full browser interaction, authenticated sessions, clicks, uploads, or custom navigation logic, a simple fetch endpoint may not match the job. Define those requirements before choosing infrastructure.

Keep the target within the public-web boundary

Rendered access does not change what you are permitted to collect. Use public HTTP or HTTPS pages, respect site rules and request rates, and do not attempt to enter accounts or bypass authentication. Review the acceptable-use policy.