We spent a long time trying to get a headless browser through Cloudflare's interstitial. It never works, and the reason is narrower and more interesting than the folklore suggests. It is not the webdriver flag, and it is not that you waited the wrong number of seconds.
The folklore
Search for this problem and you get the same advice everywhere: patch navigator.webdriver, spoof the user agent, add plugin and language properties, randomise the viewport, wait longer before reading the DOM. There is a whole genre of stealth plugins built on that list.
We tried all of it. On the easy tier it helps. On the hard tier — the one that shows "Just a moment…" and then keeps showing it — none of it moves the needle at all.
What actually gives you away
The tell is Runtime.enable.
Automation libraries talk to Chrome over the DevTools Protocol, and to evaluate JavaScript in a page most of them enable the Runtime domain. Doing so is observable from inside the page. The challenge script checks for it, and once it sees it, no amount of fingerprint patching matters: you have already announced that a debugger is attached.
This reframes the problem. Every fix in the folklore is cosmetic, aimed at what the page can read about the environment. The real signal is structural, and it comes from how your automation is driving the browser at all.
What passes
Four things together, and it is genuinely all four:
- A driver that does not enable
Runtime— patchright rather than stock Playwright. - Headful. A real window. Headless never passes, with any driver.
- Real Chrome, not the bundled Chromium.
- A persistent profile, so the clearance cookie has somewhere to live.
With those, the challenge clears in two to three seconds. Drop any one of them and you are back to watching "Just a moment…" forever.
Headless never passes. Not slowly, not sometimes, not with better waits. If a tool claims headless Cloudflare bypass on the hard tier, that claim is doing a lot of work.
What this means for a scraper
It means the honest architecture has tiers, and the expensive tier is genuinely expensive.
Most pages need nothing special. Scrape sends a plain request with sensible client hints and gets the page. A middle tier uses a real Chrome channel rather than bundled Chromium, which handles a lot of the softer checks. Only the hard tier needs a visible window on your actual desktop, and that is a resource you cannot run a thousand of in parallel.
A hosted scraper has to solve this on a server with no display, for many customers at once, which is precisely the case that does not work. Running on your own machine is not only a privacy position; on this specific problem it is the only configuration that passes.
Two things that cost us time
Worth writing down, because neither is obvious.
The window has to be on-screen. Parking it at negative coordinates to keep it out of the way is enough to fail the check.
Trust the DOM over a stale status code. After clearance lands you can still be holding a 403 from the earlier response. Reload and read the page rather than believing the number.
The general lesson
The published advice on this problem optimises the wrong variable, and it does so confidently and at length. Everyone is patching what the page can see about the browser, when the detection is watching how the browser is being driven.
It is worth asking, whenever a workaround has stopped improving no matter how much effort goes in, whether the thing being measured is the thing you are adjusting.
Build a list while this is fresh
Free plan includes 500 actions a month, unlimited tables and rows. No card.