Web scraping in JavaScript with Node.js and Cheerio

Install Cheerio

Modern Node.js versions provide fetch(). Cheerio supplies the HTML parser and a selector API that resembles jQuery.

npm install cheerio

Store the API key in WEB_BYPASS_API_KEY on the server. Never place it in client-side JavaScript, a public repository, or application logs.

Fetch a public page

Send a JSON body to POST /v1/fetch. Auto mode is the default for most jobs because it tries Easy first and uses Hard only when needed.

const response = await fetch("https://web-bypass.dev/v1/fetch", {
  method: "POST",
  headers: {
    Authorization: "Bearer " + process.env.WEB_BYPASS_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com", mode: "auto" }),
  signal: AbortSignal.timeout(90_000),
});

const data = await response.json();
if (!response.ok || !data.success) {
  throw new Error(data.error?.code ?? "FETCH_FAILED");
}
if (data.content_encoding !== "utf8") throw new Error("Expected HTML text");

The timeout protects the worker from waiting forever. Use the error code and status to decide whether the request should be retried.

Parse the returned HTML with Cheerio

Load data.content, not the complete API response. The response is JSON that also contains billing and retrieval metadata.

import * as cheerio from "cheerio";

const $ = cheerio.load(data.content);
const title = $("title").text().trim();
const description = $('meta[name="description"]').attr("content")?.trim() ?? "";
const links = $("a[href]")
  .map((_, element) => new URL($(element).attr("href"), data.final_url).href)
  .get();

console.log({ title, description, links: links.slice(0, 10) });

Resolve relative links against data.final_url. The original target may redirect, so using the submitted URL as the base can produce incorrect links.

Put retrieval and parsing in one function

This version returns a small record and preserves the request ID in errors. That is safer than returning an empty object when retrieval fails.

import * as cheerio from "cheerio";

const API_URL = "https://web-bypass.dev/v1/fetch";

export async function scrapePage(url) {
  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      Authorization: "Bearer " + process.env.WEB_BYPASS_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, mode: "auto" }),
    signal: AbortSignal.timeout(90_000),
  });
  const data = await response.json();
  if (!response.ok || !data.success) {
    const code = data.error?.code ?? "HTTP_" + response.status;
    throw new Error(code + " request_id=" + (data.request_id ?? "unknown"));
  }
  if (data.content_encoding !== "utf8") throw new Error("Expected HTML text");
  const $ = cheerio.load(data.content);
  return {
    title: $("title").text().trim(),
    heading: $("h1").first().text().trim(),
    finalUrl: data.final_url,
    statusCode: data.status_code,
    usedMode: data.access.used_mode,
    chargeUsd: data.billing.charge_usd,
  };
}

Choose selectors that survive small page changes

Prefer semantic elements, stable IDs, documented attributes, and structured metadata. Avoid selectors such as div:nth-child(7) when a record has a stable attribute or heading relationship.

• Trim extracted text and normalize whitespace.
• Validate required fields before storing a record.
• Keep the source URL and retrieval time for traceability.
• Test selectors against more than one representative page.

Use Auto until the target proves otherwise

Easy mode costs $0.10 per 1,000 successful requests and suits ordinary HTML. Hard mode costs $0.20 per 1,000 successful requests and handles browser-dependent pages. Auto attempts Easy first and charges only the successful path. A final failed request costs $0.

If a target consistently needs browser rendering, explicit Hard mode can make intent clearer. Otherwise, Auto keeps the integration simple. See current pricing and billing rules.

Do not turn failures into empty data

A parsing function should never convert every exception into an empty array. That makes a timeout indistinguishable from a page with no matching items. Preserve the stable error code and request ID, then apply a bounded policy from the retries and errors guide.