Ominous, right?

Sounds like I'm about to teach you something you'll need a lawyer for.

…We'll come back to the lawyer in a bit.

Here's a little bit of background…

So…

I was getting ready for this talk -- and OpenAI threatened to ban my account ...

Just for the work that I was doing in preparing it…

I checked my email the other day, and I found this.

What a shocker this was!

So -- what does one have to do to earn the banhammer?

For cyber abuse?

With a web browser?

This. This is what I was doing.

Every one of these is being solved by an AI agent using a web browser

... and in 15 minutes or so, you'll understand exactly how.

I want my agents to be able to use the web the way that I do -- book the things, send the emails, fill in the forms -- so I don't have to.

But, the moment something that isn't a person starts clicking, many webpages fight back.

So -- this is a talk about winning that fight. And it starts with one slightly unconventional idea.

Here it is. The premise of the whole talk.

'A CDP browser is just like a meatbag with a mouse.'

...

At least as far as Google, Cloudflare, and the rest can tell

No joke -- if you have your agent drive a browser through the Chrome DevTools Protocol, your agent's clicks and keystrokes travel the exact same path inside Chrome that yours do.

That's the big idea. The rest of the talk is how you pull it off. And that comes down to three things.

A CLI -- not an MCP.

The Chrome DevTools Protocol -- which is where a tool I built, called chrome-agent comes in.

And a loop, run on a ladder.

So, let's start with the one that picks a fight.

First thing: give your agent a command line interface (shell-based tools), not an MCP server. And before some of you knuckle-up on this debate, there are specific reasons for this.

It's worth noting that capability is a wash -- CLI and MCP both got to the right answer about 83% of the time in a recent study by Arize AI.

HOWEVER, a CLI beats an MCP in reuse, speed and cost.

Reuse first. A CLI sequence can be programmed: write it once, run it a thousand times, without a model in the loop. Whereas MCP hits the model on every turn.

The CLI is FASTER for a similar reason -- because there's no model in the middle of every step. In that same study -- MCP took 71 round-trips and 8 minutes for the same task that was completed with just 7 calls and under a minute using the CLI. Hold onto this one -- speed comes back at the very end.

And token cost -- Anthropic reported that executing code instead of running MCP can be 75x cheaper in terms of token usage.

So -- what are we actually running on the command line?

We drive the browser with the Chrome DevTools Protocol -- the second thing your agent needs to appear human.

You already know of this protocol, even if you've never heard its name. That developer panel that opens in Chrome when you hit F12 drives the browser using it. Your agents can speak it, too, using the chrome-agent tool. Chrome Agent also makes it easy for your agent to write code to replay CDP interactions.

The surface area of CDP is enormous. 57 domains, hundreds of methods and events, covering everything the browser can do. I've grouped these domains into 8 buckets to make it easier to hold in your head.

But the good news is that you don't need all 57 domains. To interact with a page like a human, you usually need just a small subset of these.

It's easiest to think about the subset of CDP domains you'll frequently use in terms of the 'digital senses' they provide your agent.

You SEE the page -- read its structure from the DOM, its semantics from the accessibility tree, or just take a screenshot when you want the pixels.

You HEAR what the page reports about itself -- the network data, its console, and its logs.

And you OPERATE the page -- with clicks, keystrokes, and navigation.

The third and final thing you need for your agent to appear human when using the browser is what I call "a loop on a ladder."

So here's the loop:

Sense, act, verify -- and you repeat it until the page gives in.

Sense -- perceive the page through one or more channels -- the DOM, the accessibility tree, a screenshot.

Then Act -- do ONE thing -- click something, type something, select something.

And then Verify -- sense again, but through a different channel than the action. e.g. If you clicked something, don't ask the click whether it worked -- look at the screen, or the network instead.

Sense where you are. Make one move. Confirm it landed. Then iterate again.

And when the loop won't close -- so when you sense, act, verify, and the page still won't do what you want -- that's the page fighting back and telling you to climb the ladder.

The Meatbag Ladder, that is! This is a ladder of techniques that are increasingly more human as you climb it.

It has 3 rungs and you climb only as high as the page forces you. Said differently, you climb to the lowest rung that works.

On Rung 1 you don't act human at all. If you can just call the API exposed within the page, or fire a synthetic JavaScript click, do that -- it's easy, it's free, it's instant, and it's the right default.

Climb to Rung 2 when faking it stops working: e.g. when you need a real click using CDP's Input domain. This is agent input that the page cannot tell apart from your own.

Climb to Rung 3 when you need human input plus human behavior -- a real mouse path, a little dwell and some jitter, or vision to actually see and interpret things.

So you start cheap, and climb one rung at a time -- only when the page makes you. Climb the ladder, run the loop on each rung, then write down the path that worked.

And that's how you arrive at fully automated AI agent-driven browsing.

First you explore -- you run the loop by hand, climbing rungs, until the thing actually works.

And then you automate -- you write that solution down so you never have to discover it again.

And you write it down as CODE, as an AGENT SKILL or, quite often, as both.

So let me show you how this all comes together -- starting with a word from my attorney.

I told you we'd come back to the lawyer.

Everything you're about to see is real -- all of this comes from real agent browser use, in the wild.

HOWEVER, On the advice of counsel, everything you're about to see runs only on infrastructure and accounts I own.

NOW ...

Who's ready to see some agents getting BUSY with the browser?

Alright, so here's some simple, everyday use. Let's send a batch of personalized emails -- each one different -- from your Outlook web client.

We've got some lovely pseudocode here to make it easy to get the general sense of how this all works in code.

Outlook's compose box has nothing to defeat, so this is Rung 1 of the Meatbag Ladder -- you don't act human at all. A synthetic click opens compose, you fill it in programmatically, and a synthetic click sends it.

And then you just … let it rip!

The reason this is going so smoothly is that it's executing a program. You capture the sequence once then loop it -- 20 emails, or 200, from one command. Solve it once, run it forever.

The agent is riffing on the content to personalize it, whilst running a program for all the interactions.

You may ask "Why drive the web UI at all -- why not use the API?" Because in a corporate environment the official API needs an app registration and an admin's sign-off -- which, as an employee, you often can't get. HOWEVER, The logged-in web session needs nothing but the login you already have. The web UI becomes a universal API in this pattern.

That's Rung 1 ... But what happens when the page pushes back?

So -- let's say you're shopping at your favorite online megastore. Let's just call them … Demazon. They're a crafty bunch over there at Demazon, and their pages don't quite care for your bots. Run that exact same fake click that just worked on Outlook, point it at an 'Add to Cart' button, and … nothing. No error, no complaint. The page just ignores it.

Because the page is checking: did this click come from a real person, or from JavaScript? Chrome stamps every event with that answer -- whether an input is trusted, or untrusted. A click you fire from code is stamped untrusted, and, in this case, the page quietly drops that input.

BUT ... No worries -- you just climb the Meatbag Ladder! On Rung 2 you run the click through Chrome's Input domain which uses the same input path your actual mouse uses. Now it's stamped trusted, the page can't tell the difference, and the item drops into the cart.

Now, we've got quite the inside view of Demazon.com. Watch the bottom panel and let's imagine that's what the server's logs might look like. Every fake click is rejected, but the real ones are accepted.

As a heads-up -- when you see the mouse pointer in these screen recordings, that has been added programmatically to show what mouse inputs the agent is simulating using chrome-agent / CDP. The agent isn't actually using my mouse.

Rung 2 is where the real meatbag inputs begin BUT … that, alone, is not enough to let your agent replace you in the browser.

So we climb to the top of the Meatbag Ladder. Rung 3 -- this is the narrow frontier where pages are actively hunting for bots. There are a variety of techniques we deploy here, though, so let's talk through a few of them.

I'm sure this guy looks familiar. This is Cloudflare Turnstile and it appears deceptively simple, but it's the hardest target we've hit yet. You can't easily get a hold of that checkbox via typical web automation programming.

It's buried behind three nested boundaries -- a closed shadow root, then a cross-origin iframe, then another shadow root inside that. To every cheap trick, that checkbox simply does not exist. There is no element to grab.

SOOOO ... you stop trying to grab it! You ask the browser where the iframe sits on screen, do a bit of maths to calculate the position of the checkbox, and fire a trusted click at that point on the glass. And Chrome does the rest!

A real click lands right on the checkbox.

No human in the loop -- all agent. That's one level, cleared, and mostly the trick here was just figuring out how to interact with it. The next levels make you prove you can actually see.

So here's MTCaptcha -- remember these ones? These types are still around ...

The agent has to actually read this guy. So it simulates what you'd do -- it takes a screenshot of the challenge and looks at it, and its own vision capability picks the characters out of the noise.

Then it types the answer back using real, trusted keystrokes, routed into the challenge's cross-origin iframe, one character at a time. The same inputs your keyboard sends.

And the server agrees: text correct, token issued!

Now there's one more level before the final boss -- and this one is won or lost based on how you move like a meatbag.

So this one's by Lemin -- it's a little jigsaw puzzle where you have to spot where the piece belongs, and drag it into the gap -- and there's an entire class of CAPTCHAs like this.

This one's tricksy in different ways. There's no shadow root and no cross-origin iframe to pierce -- the piece is sitting right there in the page. The hard part is the drag itself.

When you drop the puzzle piece, these types of CAPTCHAs sample the mouse movement into a trail of points the whole way across, including jitter and changing speed and all. So it's not just solving the puzzle, but solving it with Moves like Jagger!

SOOOO … the agent drags the way a hand drags -- eased in, gently curved, a small overshoot and then settles. Just like a meatbag with a mouse! It uses Vision to see the gap and human-like motion to cross it.

So that's Turnstile. MTCaptcha. and Lemin -- three gates built to keep agents out, and we've just beat each of them. Which leaves only one boss standing ...

And here he is -- The final boss of The Internet. reCAPTCHA v2.

The little checkbox, the grid of blurry traffic lights and crosswalks -- everyone in this room has squinted at one.

BUT we've got the whole kit for how to beat this guy, too!

The digital senses … the loop to deploy them in … the Meatbag Ladder … everything you need to take down this cheeky bastard!

So let's go!

Here's the whole machine, and it comes in two halves.

On one side we have The Solver. Pure code -- no agent, no model. It does everything programmatically:

• the trusted click on the checkbox,
• piercing into the challenge iframe,
• and then every round it screenshots the tile grid;
• if a round expires, it just re-arms and goes again.

This bit is deterministic, fast, free.

There's one step in that loop the code can't do and that's look at a grid of fuzzy photos and know which ones have a bus in them.

That's vision and thinking, and it needs eyes and a brain.

So that's the only job the agent gets. And we call that The Operator. The solver taps the agent on the shoulder -- and the agent:

• takes one look at the grid
• picks the tiles with the thing in them
• hands the answer back to the Solver
• then stands by until the next lap.

This is really the entire talk, running as one system. Code does the deterministic driving; the agent does only the part that takes eyes and a brain.

Who wants to see it go?

Allllllright!

So while that's playing, by the way, some of these are really hard! The agent spotted bicycles in photos that I didn't!

But look at that -- it's solved and server-verified and IT'S FAST!

And fast is the whole point! This big bad boss is on the clock! Every round expires, and a full solve can be many rounds back to back. An agent that round-trips a model on every click and every look BURNS THAT CLOCK and loses -- the challenge resets before it ever finishes.

The only thing I've found that defeats this is what you're watching: deterministic code running at machine speed, with one quick AI look per round.

Remember when I told you to hold onto speed? This is why.

And this is why it had to be a CLI, on CDP -- never a model sitting in the middle of every single interaction step.

In case you're wondering, solving this wasn't a fluke …

Here it is, again and again ...

A reliable, repeatable solution, every time.

But HERE's what I actually want you to walk out of here with …

The big takeaway is the methodology enabling this. The CAPTCHAs were just a tricksy test of it.

This came down to careful, disciplined engineering -- it's the engineering that enabled an agent to do something it simply could not do on its own.

And the method is simple:

• Give your agent a CLI, so you can program it.

• Drive the whole browser through CDP -- using its digital senses.

• Run it as a loop on the Meatbag Ladder -- climbing only as high as the page forces you.

• Then Explore until you solve it and write the solution down.

That's what makes this durable and truly useful: Figure it out once. Do it forever.

Which brings us all the way back … to the crew at OpenAI and the impending banhammer.

After a quiet word, they kindly rescinded the threat.

So, I've still got access to Codex, which is nice!

So, you, too, can use Chrome Agent. It's installable in the Python ecosystem.

It's also open-source and available on GitHub: github.com/captivus/chrome-agent

That's the tool I wrote to do all of this with, and I use it all day, every day!

OR ... build your own! We live in the age of unbounded, personalized software.

Please do follow me on X and let's chat -- I want to hear how you're automating the web with AI.

Happy hacking!