Which Websites Block Headless Chrome? A 200-Site Study

Disclosure: I build RenderScreenshot, a screenshot API. This study used plain Puppeteer on my own machine, not our API, and the last section explains why our production renders behave differently.

Short answer: in October 2026, 19 of the 125 top sites that served a page or a block (15.2%) blocked or challenged headless Chrome running with Puppeteer's defaults. Changing only the user agent string cut that to 6 of 125 (4.8%). Hiding navigator.webdriver on top of that changed nothing measurable.

This page has the method, the per-profile results, what the block pages looked like, the raw CSV, and the script, so you can rerun it on your own list. It is part of our engineering notes on building RenderScreenshot.

Data from the October 2026 run. Last verified: October 15, 2026.

Method box

The three profiles

Each domain was loaded three times. Only one thing changes between profiles, so a difference in outcome points at that signal.

No profile used a stealth plugin, a proxy, a CAPTCHA solver or stored cookies. I wanted to measure what a typical screenshot script meets, not how far evasion can go.

How a load was classified

The rules are crude on purpose. They are written in the script, so you can check every row against them.

Which domains count

A Tranco top-200 list is not a list of 200 websites. It includes CDN hosts, ad and telemetry domains and API hostnames that never serve a homepage. Of the 200 domains:

• 75 served no website in any profile. 50 did not resolve at all on their bare domain (for example ad-serving and asset hosts), 10 returned only a 404, 400 or 418 status, 5 returned an empty document, 4 presented a TLS certificate for another name, and 6 refused, reset or timed out the connection.
• 125 served a page, a block or a challenge in at least one profile. These are the sites in every percentage below.

Results

Outcome by profile (125 responding sites)

Blocked or challenged, share of 125 sites
headless_default        ███████████████▏ 15.2%
headless_ua             ████▊             4.8%
headless_ua_webdriver   █████▋            5.6%

The two error rows in the user agent profiles are navigation timeouts on sites that loaded in the default profile, and the extra blocked row in the last profile is a single 503 from a site that served the page in the other two. With one run per profile, treat differences of one or two sites as noise.

Three groups of sites

The 19 sites that turned away the default profile fall into clear groups:

The last row is the surprise. One site served the default headless browser and challenged the same browser once it claimed to be regular Chrome. A plausible reading is a consistency check: the user agent said Chrome, while other signals still looked headless. A mismatch can score worse than an honest headless user agent.

What the block pages looked like

For the 14 sites that reacted to the user agent, the default profile got these responses. All 14 returned HTTP 403.

The signature tells you who made the decision, which tells you who to talk to if the site is yours: the WAF or bot product in front of it, not the application.

Did the webdriver patch matter?

No. The user agent change explained the whole drop from 19 to 6. Patching navigator.webdriver produced the same 117 successful loads. That does not mean navigator.webdriver is never checked. It means none of these homepages made a different decision on it once the user agent was fixed, on this network, on this day.

What this means for screenshot jobs

1. Check the result, not the status code. Every challenge in this run came back as a page with a title and some text. A screenshot pipeline that checks only "did the browser load something" stores 15% challenge pages for a list like this. Check the status, the title and, for important targets, the image itself.
2. Do not retry a challenge in a loop. The same browser with the same signals gets the same answer. One retry catches a transient 503. More retries cost time and, on a paid API, money.
3. Know which group a failing site is in. If it reacts to the user agent, a normal user agent fixes it. If it blocks every profile from your network, nothing in the browser settings will. Our bot protection guide covers the options when the site is yours.
4. Expect a datacenter IP to do worse. This run used a residential connection. Bot products weigh IP reputation heavily, and cloud egress ranges start with a lower score.

Timeouts are a separate problem from blocks. If a page loads but never settles, see how long a screenshot takes for where the seconds go.

How our production renders differ

RenderScreenshot does not render on a residential connection with a patched user agent. Our renderer runs in Cloudflare's Browser Run service, and Cloudflare documents that it adds non-configurable headers to those requests and signs them with Web Bot Auth, so they are always identified as automated. A custom user_agent in our API does not change that. We do not try to hide it, and we do not solve CAPTCHAs; the bot protection guide has the policy and the sources.

So the 4.8% figure for the rewritten user agent is not a prediction for our API. For sites behind a bot product, expect our renderer to be treated like the default profile or worse. When we measure that directly, it will go into the screenshot API benchmark with its own method box.

Limits of this study

• One run, one day, one network. Bot decisions change with time, IP reputation and load. A rerun next week will move individual sites.
• Homepages only. Many sites protect search, login and product pages harder than the homepage.
• Top-ranked domains only. The long tail of smaller sites has less bot protection on average, so these percentages are likely higher than for a random list of URLs.
• Text rules, not visual review. A soft block that shows the real layout with missing content is counted as ok. Conversely, a short legitimate page could be counted as empty.
• Chrome 148 only, in Puppeteer's new headless mode. The older chrome-headless-shell binary has a different fingerprint and was not tested.

Rerun it on your own URLs

The script loads each domain in the three profiles and prints one CSV row per load. Put your domains (or a Tranco CSV) in a file and run it.

npm install [email protected]
node headless-study.mjs domains.txt 200 > results.csv

// headless-study.mjs: load each site's homepage in three browser profiles and
// record whether the response looks like a block or a bot challenge.
// Node 22+, puppeteer@24.  Usage:
//   node headless-study.mjs domains.txt 150 > results.csv
import { readFile } from 'node:fs/promises';
import puppeteer from 'puppeteer';

const [, , listFile = 'domains.txt', limitArg = '100'] = process.argv;
const LIMIT = Number(limitArg);
const CONCURRENCY = 4;
const NAV_TIMEOUT_MS = 30_000;
const SETTLE_MS = 3_000; // let challenge scripts swap the page in

// Profiles: what changes between them is the only thing a site could react to.
const PROFILES = {
  // 1. Puppeteer defaults: the user agent says "HeadlessChrome".
  headless_default: { patchUa: false, patchWebdriver: false },
  // 2. Same browser, user agent rewritten to the matching desktop Chrome string.
  headless_ua: { patchUa: true, patchWebdriver: false },
  // 3. Rewritten user agent and navigator.webdriver hidden.
  headless_ua_webdriver: { patchUa: true, patchWebdriver: true },
};

const TITLE_PATTERNS = /just a moment|attention required|access denied|are you a robot|captcha|verify you are human|pardon our interruption|request unsuccessful|security check|unusual traffic|bot detection|forbidden/i;

// DOM markers of common challenge vendors. Counted only as "challenge", never as "blocked".
const MARKER_SELECTORS = {
  cloudflare: 'iframe[src*="challenges.cloudflare.com"], #challenge-form, #cf-challenge-running',
  datadome: 'iframe[src*="captcha-delivery.com"]',
  perimeterx: '#px-captcha',
  imperva: 'iframe[src*="_Incapsula_Resource"]',
  recaptcha: 'iframe[src*="recaptcha/api2/anchor"]',
  hcaptcha: 'iframe[src*="hcaptcha.com"]',
};

function csv(value) {
  const s = String(value ?? '');
  return /[",\n]/.test(s) ? `"${s.replaceAll('"', '""')}"` : s;
}

function classify({ status, title, markers, textLength, error }) {
  if (error) return 'error';
  if (markers.length > 0 || TITLE_PATTERNS.test(title)) return 'challenge';
  if ([401, 403, 429, 503].includes(status)) return 'blocked';
  if (status >= 400) return 'http_error';
  if (textLength < 50) return 'empty';
  return 'ok';
}

async function visit(browser, domain, profile, desktopUa) {
  const context = await browser.createBrowserContext();
  const page = await context.newPage();
  const started = performance.now();
  const row = { domain, profile: profile.name, status: 0, title: '', markers: [], textLength: 0, finalUrl: '', error: '' };
  try {
    await page.setViewport({ width: 1280, height: 800 });
    if (profile.patchUa) await page.setUserAgent(desktopUa);
    if (profile.patchWebdriver) {
      await page.evaluateOnNewDocument(() => {
        Object.defineProperty(Navigator.prototype, 'webdriver', { get: () => false });
      });
    }
    const response = await page.goto(`https://${domain}/`, { waitUntil: 'domcontentloaded', timeout: NAV_TIMEOUT_MS });
    await new Promise((r) => setTimeout(r, SETTLE_MS));
    row.status = response?.status() ?? 0;
    row.finalUrl = page.url();
    row.title = (await page.title()).slice(0, 120);
    row.textLength = await page.evaluate(() => document.body?.innerText.length ?? 0);
    for (const [vendor, selector] of Object.entries(MARKER_SELECTORS)) {
      if (await page.$(selector)) row.markers.push(vendor);
    }
  } catch (err) {
    row.error = err.message.split('\n')[0].slice(0, 160);
  } finally {
    row.ms = Math.round(performance.now() - started);
    await context.close().catch(() => {});
  }
  row.outcome = classify(row);
  return row;
}

const domains = (await readFile(listFile, 'utf8'))
  .split('\n')
  .map((line) => line.trim().split(',').pop())
  .filter((d) => d && !d.startsWith('#'))
  .slice(0, LIMIT);

const browser = await puppeteer.launch({ headless: true });
let exitCode = 0;
try {
  // Build the "normal" desktop UA from the real browser version, minus the Headless marker.
  const desktopUa = (await browser.userAgent()).replace('HeadlessChrome', 'Chrome');
  console.log('domain,profile,outcome,status,markers,title,text_length,final_url,ms,error');

  const jobs = domains.flatMap((domain) =>
    Object.entries(PROFILES).map(([name, p]) => ({ domain, profile: { name, ...p } })),
  );
  let next = 0;
  async function worker() {
    while (next < jobs.length) {
      const { domain, profile } = jobs[next++];
      const r = await visit(browser, domain, profile, desktopUa);
      console.log([r.domain, r.profile, r.outcome, r.status, r.markers.join('|'), r.title, r.textLength, r.finalUrl, r.ms, r.error].map(csv).join(','));
    }
  }
  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
} catch (err) {
  console.error(err);
  exitCode = 1;
} finally {
  await browser.close();
}
process.exitCode = exitCode;

Tested on October 15, 2026 with Node.js 22.18.0 and Puppeteer 24.43.1 (Chrome 148) on macOS; this exact script produced the published CSV. The published file drops the ms column: the machine was shared with other jobs during the run, so the timings are not a fair speed measure.

Two notes before you rerun it on a large list. Load homepages politely: one request per site per profile is enough, and the script never retries. And keep the SETTLE_MS pause: several challenge pages in this run replaced the document a second or two after domcontentloaded.

Related

• How we built RenderScreenshot: the architecture these notes come from.
• Bot protection and CAPTCHAs: what blocked captures look like in our API, and how site owners can allow them.
• How long does a screenshot take?: the latency side of the same pipeline.
• Screenshot API benchmark: method and data for comparing hosted APIs.

If you need captures of your own pages, the free plan is enough to test the allow-list options in the guide above.