Web scraping in Python with an API and Beautiful Soup

What Beautiful Soup does, and what it does not

Beautiful Soup parses a document that you already have. Give it HTML and it can find headings, links, tables, metadata, or any element matched by a CSS selector. It does not open a browser, execute page scripts, solve access problems, or fetch the document for you.

That distinction explains a common failure. A script downloads a page with Requests, passes the response to Beautiful Soup, and finds no product cards or article text. The parser is working. The server simply returned HTML that expects JavaScript to fill in the page later.

Install Requests and Beautiful Soup

This example uses Requests for the API call and Beautiful Soup 4 for HTML parsing.

python -m pip install requests beautifulsoup4

Keep the API key in a server-side environment variable. Do not paste it into source code, browser JavaScript, screenshots, or a public repository.

# macOS or Linux
export WEB_BYPASS_API_KEY="your_api_key"

# Windows PowerShell
$env:WEB_BYPASS_API_KEY="your_api_key"

Fetch a page through the web scraping API

Send the target URL to POST /v1/fetch. Auto mode is the practical default: it starts with the lower-cost path and uses the browser-capable path only when the first result is not usable.

import os
import requests

api_key = os.environ["WEB_BYPASS_API_KEY"]

response = requests.post(
    "https://web-bypass.dev/v1/fetch",
    headers={"Authorization": f"Bearer {api_key}"},
    json={
        "url": "https://example.com",
        "mode": "auto",
    },
    timeout=90,
)

data = response.json()

if not response.ok or not data.get("success"):
    error = data.get("error", {})
    raise RuntimeError(f"Fetch failed: {error.get('code', response.status_code)}")

if data["content_encoding"] != "utf8":
    raise RuntimeError("Expected a text response, but received binary content")

html = data["content"]
print(f"Fetched {data['final_url']} with {data['access']['used_mode']}")

The response is JSON. For an HTML page, the HTML is in content. The response also records the final URL after redirects, the HTTP status from the target, which access mode succeeded, the request charge, and the remaining account balance.

Parse the returned HTML with Beautiful Soup

Build the soup from data["content"], not from the API response body itself. The API response body is a JSON envelope around the page content.

from bs4 import BeautifulSoup
from urllib.parse import urljoin

soup = BeautifulSoup(html, "html.parser")

title = soup.title.get_text(strip=True) if soup.title else "Untitled"
links = [
    urljoin(data["final_url"], anchor["href"])
    for anchor in soup.select("a[href]")
]

print(title)
for link in links[:10]:
    print(link)

urljoin() converts relative links such as /docs into absolute URLs. Use final_url as the base because the target may have redirected before the page was returned.

A complete Python scraper

The complete version below returns a small, useful record instead of dumping a large HTML document to the terminal. It extracts the page title, meta description, links, final URL, status code, access mode, and charge.

import os
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

API_URL = "https://web-bypass.dev/v1/fetch"
API_KEY = os.environ["WEB_BYPASS_API_KEY"]


def scrape_page(url: str) -> dict:
    response = requests.post(
        API_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"url": url, "mode": "auto"},
        timeout=90,
    )
    data = response.json()

    if not response.ok or not data.get("success"):
        error = data.get("error", {})
        code = error.get("code", f"HTTP_{response.status_code}")
        raise RuntimeError(f"Web scraping API error: {code}")

    if data["content_encoding"] != "utf8":
        raise RuntimeError("This scraper expects an HTML text response")

    soup = BeautifulSoup(data["content"], "html.parser")
    title = soup.title.get_text(" ", strip=True) if soup.title else "Untitled"
    description_tag = soup.select_one('meta[name="description"]')
    description = description_tag.get("content", "").strip() if description_tag else ""
    links = [
        urljoin(data["final_url"], anchor["href"])
        for anchor in soup.select("a[href]")
    ]

    return {
        "title": title,
        "description": description,
        "links": links,
        "final_url": data["final_url"],
        "status_code": data["status_code"],
        "used_mode": data["access"]["used_mode"],
        "charge_usd": data["billing"]["charge_usd"],
    }


result = scrape_page("https://example.com")
print(result)

Choose Auto, Easy, or Hard mode

Most applications should keep mode set to auto. Direct mode selection is useful when you already know how the target behaves.

A final failed request costs $0. See the pricing page for the current billing rules.

Handle API errors without hiding the cause

Do not catch every exception and return an empty list. That makes a blocked page look like a page with no results. Log the request ID and stable error code instead. Never log the Authorization header.

Text pages normally use content_encoding: "utf8". Binary data may be returned as Base64. This tutorial stops on Base64 because Beautiful Soup expects text, not an image, PDF, or other binary file.

Use web scraping responsibly

A working request is not permission to collect or republish data. Respect the target site's terms, robots instructions, copyright, privacy obligations, and rate limits. Do not use the service for private pages, credentialed sessions, account access, or attempts to bypass authentication.

The API accepts public HTTP and HTTPS targets. Local, private, internal, and metadata-network addresses are rejected. Read the acceptable-use policy before putting a scraper into production.