How often does a web fetch really work?

Run of 2026-11-02 · 166 pages · 134 domains · 6 site types. Data: benchmark-2026-11-02.csv (one row per page × route) and the target list. Licence: CC BY 4.0.

Results

Share of pages that loaded and passed verification. "Either" is what you get when a router tries plain HTTP first and falls back to the browser.

Why pages failed on every route

blocked:datadome 13 · too_short 11 · blocked:cloudflare 8 · blocked:perimeterx 7 · provider_error 7 · http_403 6 · blocked:aws_waf 4 · http_429 2

What this says

• Single-page apps: plain HTTP passed 22%, a browser 48%. The plain response is usually a 200 with an empty shell, a failure an agent can't see from the status code.
• Retail and anti-bot-protected sites: even with a browser fallback only 40% passed from a datacenter IP. Those need an unblocking provider with residential IPs.
• Status codes mislead: 33 failed fetches came back with HTTP 2xx (block pages, empty shells, missing content). Only a content check catches them.

Pages no route could load

Linked for reproducibility. These sites block automated datacenter traffic; we don't try to get around login or paywalls.

Method

• Targets: 166 public URLs in six types: simple sites, docs, news, single-page apps, retail, and sites behind anti-bot protection. Picked once and kept fixed so runs compare. List: targets.json.
• Routes: plain HTTP (Node HTTP client with Chrome-like headers) and headless Chromium (Playwright; waits for the DOM, then up to 8 s for the network to go quiet). Each page is fetched once per route per run. No retries, no proxies, no CAPTCHA solving.
• Where from: one Fly.io machine in Ashburn, Virginia (datacenter IP). Residential IPs would pass more protected pages; that's the job of unblocking providers, which aren't in this table yet.
• Pass: HTTP 2xx, no block or challenge page (signatures for Cloudflare, DataDome, PerimeterX, Akamai, Imperva, Kasada and others), at least 200 characters of real text (or a complete script-free page), and the expected text for that page when the target lists one.
• Load: at most 3 fetches at a time and never two at once to the same domain. Timeouts: 25 s plain, 45 s browser.
• Limits: one region, one run per day, about 25–30 pages per type. Treat differences of a few points as noise.

Want your site removed from the target list? Email hello@fetchrouter.dev. Live per-domain numbers from all traffic: Reliability.