Why AI browser agents get blocked, and what actually helps

Cloudflare gives the requests it screens a bot score from 1 to 99. Its bot score documentation says a 1 means it is “quite certain the request was automated” and a 99 means it is quite certain a human sent it. If your browser agent keeps landing on challenge pages, it is scoring near the bottom, and the model driving it has almost nothing to do with why.

Anti-bot systems can’t see your prompt. They see a connection from an IP address, a TLS handshake, a browser that does or doesn’t behave like a normal one, and a pattern of requests over time. Each of those gives something away. Stack enough of it up and you get a challenge, a 403, or a quiet block that looks like the site is broken.

This piece goes through the signals roughly in the order a request meets them, then covers what legitimately improves things. It assumes the agent is doing legitimate work: acting for a user who asked it to, testing your own site, or collecting data within a site’s terms. If a site’s terms rule out automation, the honest move is to ask for permission or use their API.

The IP address and who owns it

The first thing any edge network sees is your IP, before a byte of HTML. From the IP it can look up the network (the ASN) that announces it, and that information is public. AWS, Google Cloud, Azure, Hetzner and DigitalOcean ranges are all easy to identify as hosting providers.

Traffic from a hosting range isn’t automatically bad. Plenty of legitimate crawlers and APIs live there. But real people rarely shop or bank from a cloud VM, so a datacenter IP starts with less benefit of the doubt. If your agent runs in a cloud container with default egress, every site it visits sees exactly that.

Reputation also builds up per address. An IP that ran someone else’s scraper last week can arrive already flagged, and you inherit the history.

Residential and mobile IPs sit at the other end. Mobile carriers put many subscribers behind carrier-grade NAT, so one mobile IP can front a lot of real people at once. Blocking one hard risks blocking paying customers, which is why many sites handle mobile ranges more carefully. Treat that as a tendency. A mobile IP carrying abusive traffic still gets challenged.

The TLS handshake

Before any HTTP happens, your client sends a TLS Client Hello listing the cipher suites, extensions and versions it supports, in a particular order. Cloudflare’s docs describe JA3 and JA4 fingerprints as identifying “TLS clients based on how they initiate connections.” Real Chrome has a recognisable fingerprint. So does Python’s requests library, and the two look nothing alike.

This is why an agent that sends a Chrome user-agent header through a plain HTTP library gets caught so fast. The header says Chrome and the handshake says Python. Agents that drive a real Chromium through Playwright or the DevTools protocol mostly avoid that mismatch, because the actual browser does the handshake.

Headless and automation tells

Once a page loads, JavaScript can inspect the browser. The obvious check is navigator.webdriver. MDN documents that in Chrome it returns true when the --enable-automation or --headless flag is used, or when remote debugging is opened on port 0. It exists so that pages can tell they’re being automated.

Headless mode used to be a bigger tell than it is. Chrome’s old headless mode was a separate implementation; since Chrome 132 it only ships as a standalone chrome-headless-shell binary, and the current --headless shares its code with normal Chrome. Closer is still not identical, and detection scripts probe rendering, fonts, plugins and window sizes for the differences.

My position: don’t play the patching game. Stealth plugins that overwrite navigator.webdriver and friends are an arms race you lose on the vendor’s schedule, and on a lot of sites they cross the line from being a well-behaved client into deliberately defeating a protection. Run a real browser, headed where you can, and be honest about what you are.

Behaviour and timing

Cloudflare’s docs describe machine-learning scoring over headers, session characteristics and browser signals, plus JavaScript detections aimed at headless browsers. Agents have a recognisable shape. They load a page, sit still for however long the model takes to think, then act instantly and precisely. No scrolling, no mouse path, a click in the dead centre of an element, then another long pause. Sessions often start cold on a deep URL with no cookies and no referrer.

I wouldn’t try to fake human mouse movement. What you can do is stop doing obviously strange things. Keep cookies between runs so every task isn’t a brand-new visitor, start from pages a person would start from, and don’t hammer one endpoint.

Rate

The simplest signal, and the one teams ignore most. An HTTP 429 means you sent too many requests in a given time, and the server may include a Retry-After header saying how long to wait. Agents that fan out across 50 workers on one IP, or retry the instant something fails, walk into rate limits and then into longer blocks.

What legitimately helps

In the order I’d rely on them:

1. Use the official API if there is one. It’s faster, cheaper per task, and nobody is trying to stop you. An agent clicking through a web form that the company also exposes as an API is solving the wrong problem.
2. Ask for access. Many sites will allowlist a known partner by IP or by credential. There’s also a newer route. Cloudflare’s Web Bot Auth lets bots sign their HTTP requests using message signatures, so a site can verify who sent them, and in August 2025 Cloudflare added a signed agents category for agents acting on behalf of users, naming ChatGPT agent, Goose, Browserbase and Anchor Browser among the first. If you run an agent platform, read both posts.
3. Respect robots.txt and the terms. RFC 9309 is explicit that robots rules “are not a form of access authorization”, so they don’t lock anything. Ignoring them is still how you end up on the wrong side of a site owner.
4. Rate-limit yourself. Obey Retry-After, back off exponentially with jitter, and cap concurrency per domain.
5. Run a real browser and keep your sessions. Persist cookies and storage per identity, so a returning visit looks like a returning visit. The mechanics are in the browser agent production checklist.
6. Put legitimate traffic on a better-reputation IP. If your agent runs in a datacenter and the task is legitimate, a residential or mobile exit means the site judges it on behaviour instead of on where it’s hosted. We run Singapore Mobile Proxy, which provides real Singapore mobile IPs on SingTel, StarHub and M1 with sticky sessions, so a task keeps one IP for its whole run. It is Singapore only. That fits agents whose work doesn’t depend on country, or that serve Southeast Asia. If your task needs a US IP, it’s the wrong tool.

What doesn’t help, or doesn’t help for long:

• Rotating to a fresh IP on every request. It breaks sessions, and a login that hops IPs every minute looks worse than one that stays put.
• Spoofing a browser user agent over a non-browser HTTP client. The TLS fingerprint gives it away.
• Stealth patches, for the reasons above.
• CAPTCHA-solving services. A CAPTCHA is the site asking whether a human is present, and the right answer is to hand that step to the human the agent works for. Anthropic’s computer use docs make a related point: ask a person to confirm actions that need affirmative consent, such as agreeing to terms of service.

What nobody outside the vendor knows

The detection models are proprietary and they change constantly. The signals above are the ones vendors document publicly. How much weight each carries on a given site on a given day is not something anyone outside the vendor knows, us included. When a guide quotes a precise block rate for some setup, ask where the number came from.

To see which of these signals your own agent is tripping, you need a trace and a screenshot of each failure, which is what observing AI agents: traces, replays and cost per task covers. If your agent drives a phone instead of a browser, the picture changes again; see phone-use agents on real Android. Everything else is in the article index.

Written by Xavier Fok

disclosure: The Agent Ops Report is published by the team that runs Singapore Mobile Proxy and cloudf.one. Any link to those two products is ours and is marked sponsored. No vendor paid for a mention in this article. Last reviewed by Xavier Fok on 2026-09-11.