Coins5
EN ES
Back to Blog
Data Engineering Python Automation Supabase

Resilient Web Scraping: High-Performance Pipelines

• Written by Marlon

The Fragility of Web Extraction: Moving Beyond Basic Scripts

Many businesses begin their data extraction journey with a simple script: a basic Python tool importing BeautifulSoup and requests. While this works perfectly in a local development environment running against a dozen static pages, it fails catastrophically in production.

When you scale from a hundred pages to millions of records, the web becomes hostile. HTML structures change without warning, servers deploy rate-limiters, IP addresses get blacklisted, and network latency spikes. A basic scraper will crash, corrupt your database, or leave you with incomplete and duplicate datasets.

For modern startups and enterprise platforms, web scraping python implementations cannot afford to be fragile. They must be treated as production-grade software: reliable, automated, and built to survive edge cases.

Architecting for Resilience: Evading Blocks and Handling Failures

To build a resilient data extraction pipeline, your architecture must assume failure is the default state. Handling network fluctuations, rate limits, and server-side defensive blocks requires three main pillars:

1. Advanced Proxy Rotation and Fingerprinting Evasion

Relying on a single IP address is a fast track to getting blocked. A professional pipeline integrates residential or mobile proxy pools. Furthermore, modern bot detection looks beyond IP addresses to TLS fingerprints, HTTP/2 settings, and header consistency. Rotating User-Agents must be paired with realistic headers and browser-like behaviors.

2. Concurrency and Rate Limiting

Flooding a target server with requests is both unethical and self-defeating. You must implement concurrency control using token buckets or semaphores, ensuring your scraper stays within safe, human-like request thresholds.

3. Automated Retries with Exponential Backoff

When a server returns a 429 Too Many Requests or 503 Service Unavailable, a naive script immediately retries or crashes. A resilient pipeline employs exponential backoff with jitter, delaying subsequent retries to give the target server time to recover.

Here is how to implement a resilient, asynchronous HTTP client in Python using tenacity and httpx:

import asyncio
import httpx
from tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_exception_type

# Define resilient retry policy with exponential backoff and jitter
@retry(
    wait=wait_random_exponential(min=1, max=10),
    stop=stop_after_attempt(5),
    retry=retry_if_exception_type(httpx.HTTPError),
    reraise=True
)
async def fetch_page_with_retry(client: httpx.AsyncClient, url: str, headers: dict) -> str:
    response = await client.get(url, headers=headers, timeout=10.0)

    # Raise an exception for HTTP errors (4xx/5xx) to trigger the retry decorator
    response.raise_for_status()
    return response.text

async def main():
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
        "Accept-Language": "en-US,en;q=0.9"
    }

    async with httpx.AsyncClient(proxies="http://your-proxy-pool.com:8000") as client:
        try:
            html_content = await fetch_page_with_retry(client, "https://api.target.com/data", headers)
            print("Successfully extracted data.")
        except Exception as e:
            print(f"Failed to fetch data after multiple attempts: {e}")

if __name__ == "__main__":
    asyncio.run(main())

High-Throughput Ingestion: Database Optimization

Extracting data is only half the battle. Storing millions of records efficiently is where many data pipeline automation flows bottle-neck. Making a single database connection and performing single-row inserts for every record will exhaust your database pool, cause lock contention, and crash your server.

When inserting millions of data points into a supabase database or any PostgreSQL instance:

  1. Bulk Inserts (Batching): Accumulate records in memory or a memory-optimized queue (like Redis) and write them in chunks of 500 to 1,000 rows. This reduces network roundtrips drastically.
  2. Upserts (ON CONFLICT DO UPDATE): Web scraping often encounters duplicate data. Use PostgreSQL’s UPSERT capabilities to insert new rows while updating existing ones in a single query.
  3. Connection Pooling: Use transaction-level connection pools (such as Supabase’s built-in PgBouncer/Supavisor) to prevent client script concurrency from overwhelming database connection limits.

Here is an example of batching and upserting data efficiently into Supabase/PostgreSQL:

from supabase import create_client, Client
import math

SUPABASE_URL = "https://your-project.supabase.co"
SUPABASE_KEY = "your-service-role-key"
supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)

def batch_upsert_data(records: list, batch_size: int = 1000):
    total_records = len(records)
    batches = math.ceil(total_records / batch_size)

    print(f"Starting ingestion of {total_records} records in {batches} batches...")

    for i in range(batches):
        start_idx = i * batch_size
        end_idx = start_idx + batch_size
        batch = records[start_idx:end_idx]

        try:
            # High-performance bulk upsert mapping to primary key
            response = supabase.table("extracted_products").upsert(
                batch,
                on_conflict="sku" # Unique key constraint to prevent duplicates
            ).execute()
            print(f"Batch {i+1}/{batches} successfully written.")
        except Exception as e:
            print(f"Error writing batch {i+1}: {e}")
            # In production, route failed batches to a dead-letter queue (DLQ) for recovery

# Example payload
scraped_items = [{"sku": f"PROD-{idx}", "price": 29.99, "stock": idx} for idx in range(5000)]
batch_upsert_data(scraped_items)

The Senior + AI Advantage: Scalable Data Infrastructure

Building large-scale data systems requires structural discipline. Applying Clean Architecture to data ingestion ensures your parsing logic is decoupled from your network client, making it easy to swap crawler engines or adapt to target changes in minutes.

At Coins5, Marlon applies over a decade of software engineering expertise (solidifying Clean Architecture, PostgreSQL optimization, and robust async patterns) combined with AI-powered development workflows. By utilizing AI to write comprehensive integration tests and validate edge cases rapidly, we build bulletproof pipelines up to 3x faster than standard development teams, keeping your operational costs low and your data clean.

Need to automate data extraction for your business?

  • Schedule a Call: Book a Call to discuss your data architecture, Supabase setup, and scalability needs during a 15-minute discovery session.
  • Get a Direct Quote: Let’s discuss your targets, scope, and timeline directly on WhatsApp.