Handle web scraping API retries, timeouts, and errors

Classify errors before retrying

Retrying every failure wastes time and can amplify load. Invalid URLs, rejected targets, invalid API keys, and insufficient balance require a change in input or account state. Repeating the same request will not repair them.

Concurrency limits, retrieval failures, timeouts, and transient network errors may recover. Even then, use a small attempt limit and increasing delay.

Use a clear retry matrix

Implement bounded exponential backoff

This Node.js example makes at most three attempts. It retries temporary HTTP statuses and transport errors, adds jitter, and includes the request ID when it gives up.

const RETRYABLE = new Set([429, 502, 504]);
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

export async function fetchPage(url, maxAttempts = 3) {
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    let response;
    try {
      response = await fetch("https://web-bypass.dev/v1/fetch", {
        method: "POST",
        headers: {
          Authorization: "Bearer " + process.env.WEB_BYPASS_API_KEY,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ url, mode: "auto" }),
        signal: AbortSignal.timeout(90_000),
      });
    } catch (error) {
      if (attempt === maxAttempts) throw error;
      await sleep(500 * 2 ** (attempt - 1));
      continue;
    }

    const data = await response.json();
    if (response.ok && data.success) return data;
    if (!RETRYABLE.has(response.status) || attempt === maxAttempts) {
      const code = data.error?.code ?? "HTTP_" + response.status;
      throw new Error(code + " request_id=" + (data.request_id ?? "unknown"));
    }

    const jitter = Math.floor(Math.random() * 250);
    await sleep(500 * 2 ** (attempt - 1) + jitter);
  }
}

Keep the maximum attempts low. A worker that retries indefinitely can consume its own capacity, delay healthy jobs, and make an external incident harder to recover from.

Set timeouts at every boundary

The HTTP client needs a timeout, but the queue and job also need deadlines. A 90-second request inside a worker with a 60-second job limit will be terminated before the client can report its own error.

• Set the client timeout below the worker or serverless execution limit.
• Limit total attempts and total elapsed time.
• Cancel abandoned work when the caller no longer needs the result.
• Track timeout rates by target group instead of hiding them in one average.

Treat rate limits as a queueing signal

RATE_LIMITED means too many requests are active at once. Retrying immediately creates the same condition. Reduce worker concurrency or delay the job before trying again.

The default account concurrency is 10. Applications should still control their own queue so a burst does not turn into synchronized retries.

Keep failure handling aligned with billing

A final failed retrieval is charged $0. Successful requests include billing.charge_usd and billing.balance_remaining_usd. Do not estimate charges with JavaScript floating point or infer billing from the HTTP status alone. Read the response fields and use the published billing rules.

Log enough to debug without leaking credentials

Record the request ID, stable error code, attempt count, elapsed time, requested mode, and target hostname. Do not log the Authorization header, API key, full private query strings, or complete page content.

Separate retrieval failures from parser failures. A healthy API response followed by a missing selector is a content or parser issue, not a network retry case.