BlogAI agents

Browser and computer-use agents: automating work that has no API

Browser and computer-use agents are models that operate software the way a person does: they look at a page or a screen, choose the next click or keystroke, and read what happened. They are worth using for work stuck behind interfaces with no API, such as supplier portals, old back-office systems and exploratory QA. Wherever a decent API exists they are the wrong tool, because every step costs a model call and none of them is guaranteed.

What these agents are

A computer-use agent runs a loop. Your code captures the screen and sends it to a model with the task. The model replies with an action: click at these coordinates, type this text, press this key, scroll, wait. Your code performs the action, captures the screen again and sends it back. The loop ends when the model says the task is done, or when your step limit says it has had enough.

A browser agent runs the same loop inside a web browser, usually with a better view of the page. Instead of pixels alone it can read the page’s structure: the accessibility tree, the visible text, the interactive elements with references it can point to. It can then say “click element 14” where a pixel-based agent says “click at 512, 380”. Structure is cheaper to process and less error-prone than pixels, so use it when you have it. Pixels are for the cases where structure does not exist: a canvas, a remote desktop session, a twenty-year-old Windows client.

One point gets lost in demos. The model never touches your systems. It proposes actions, and your program executes them in a browser or virtual machine that you run. So you decide which actions are allowed, which need a person’s approval and which sites the browser can reach. Most of the engineering in a production browser agent is in that control loop, not in the prompt.

What is available now

We checked the providers’ documentation while writing this. Names and versions in this area change every few months, so treat this as a snapshot and follow the links.

  • Anthropic’s computer use tool is listed as stable on the Claude API. It gives the model mouse, keyboard, screenshot and zoom actions, all executed by your application. A separate browser toolset works from page structure and falls back to screenshots.
  • OpenAI’s computer use guide describes a computer tool in the Responses API that returns batches of interface actions. The same guide documents a second style, in which the model writes short scripts (JavaScript with Playwright, or Python) and your runner executes them in a browser that stays open between steps.
  • Google’s Computer Use tool in the Gemini API covers browser, mobile and desktop environments. A proposed action can carry a safety decision that tells your code to ask the user before going ahead.

The open-source tooling matters as much, because it is what you build the control loop from. Playwright MCP, from Microsoft, exposes a browser to any model through the Model Context Protocol using accessibility snapshots, so no vision model is needed. browser-use is an MIT-licensed Python library whose 1.0 version has the agent write Python against the Chrome DevTools Protocol. Stagehand lets you mix natural-language steps with ordinary browser code in one script, which is the shape we think most production automations should have.

Approach How it sees the screen Good for Weak at
Computer use (pixels) Screenshots and coordinates Desktop clients, remote desktops, canvas apps Speed, token cost, small click targets
Browser agent (structure) Accessibility tree, page text, element references Web portals, forms, looking things up Canvas, embedded frames, heavily customised widgets
Model-written scripts The model writes browser code and reads the results Multi-step web tasks, extraction at volume Needs a sandbox to run generated code
Script with model steps Code drives, the model handles named steps Repeatable processes with a few variable screens Open-ended tasks nobody has mapped

The same capability is sold to individuals as browser extensions and AI browsers. Those help one person with their own tasks while they watch. This article is about agents that run inside a business process with nobody watching each click.

Where they are worth it

Our test is simple: a person does the task through a screen today, there is no API, and writing a conventional script for it is uneconomic.

Legacy portals. Supplier, customer, carrier, insurer and government portals are built for people, and few offer an API to a mid-size customer. If your team logs in to thirty of them, each laid out differently, thirty hand-written scrapers is a maintenance job nobody wants. An agent that can read an unfamiliar page copes with the long tail.

Back-office systems with no API. An old ERP with a desktop client, a hosted application reached only through remote desktop, a terminal screen. Computer use is slow here, but sometimes it is the only door.

QA. Agents make good exploratory testers. Describe a user journey in plain language, let the agent attempt it against a staging build and keep the trace. They can also repair tests that broke because a selector changed, which is where test suites lose a lot of time.

Take a wholesale distributor whose credit-control team logs in to 35 customer procurement portals every week to check invoice status and download remittance advices. No two portals look alike, and the volume on each is small. The work is read-only, the answer can be checked against the ledger, and a mistake costs a re-run. That is close to an ideal first task.

Where are they the wrong tool?

Anything with a decent API. An API call takes milliseconds, costs almost nothing and returns the same answer every time. Before building an agent, look for the API, then for a scheduled export, then for read access to the database. We have seen “no API” turn out to mean “nobody asked the vendor”.

Our order of preference:

  1. An API or webhook.
  2. A file export, scheduled report or read replica of the database.
  3. A deterministic browser script, where the pages are few and stable.
  4. A browser agent, where pages vary or change often.
  5. Pixel-based computer use, where nothing else reaches the application.

Agents are also wrong for high-volume work where each task must finish in seconds, for irreversible actions with no approval step, and for sites whose terms forbid automated access. A CAPTCHA is the site owner saying no. Ask for an API or a service account instead of looking for a way round it.

If your starting point is an existing estate of screen-scraping bots, our comparison of agentic workflows and RPA covers when to keep them.

How reliable and how fast are they?

Slower than you expect, and less reliable over long tasks than a demo suggests.

Speed first. Each step is a round trip to a model, often with an image attached. A task that takes a person forty clicks takes the agent forty model calls or more, so it runs for minutes where a script runs for seconds. Batched actions and structure-based tools help. Nothing makes it as fast as code.

Errors compound. As plain arithmetic, if each step goes right 98 times in 100, a forty-step task finishes without a single wrong step less than half the time (0.98 to the power of 40 is about 0.45). That is an illustration, not a benchmark. The point is that long chains punish small error rates, and that short tasks, checkpoints and retries are how you raise the task-level figure.

The vendors document the failure modes themselves. Anthropic’s documentation warns that the model sometimes behaves as though an action worked without checking, and that dropdowns and scrollbars are awkward to operate by mouse. Google’s warns that unexpected pop-ups and layout changes can confuse the model. Our own additions from practice: sessions that time out mid-task, cookie banners, date pickers, and pages that go on loading data after they look finished.

Cost has the same shape. Anthropic’s documentation puts one screenshot at roughly 1,000 to 1,800 input tokens, and earlier screenshots are sent again on every step unless you shrink, prune or cache them.

What we do about it:

  • Verify outcomes in code. Never accept the model’s word that a file was downloaded. Check that the file exists and parses.
  • Do the login and the fixed navigation in ordinary code, and hand over to the agent at the part that varies.
  • Set a step cap and a time budget per task. A task that exceeds either goes to a person with the trace attached.
  • Fix the viewport size, and prefer direct URLs and keyboard shortcuts to mouse travel.
  • Make tasks safe to repeat, so a retry does no harm.

Security: credentials, sandboxes and approvals

A browser agent reads content written by strangers and then takes actions. That is prompt injection in its most direct form: text on a page can try to instruct the model. All three providers warn about it in their documentation, and OpenAI’s guide says plainly to treat what is on the screen as untrusted. Provider-side classifiers reduce the risk without removing it, so design as if an injection will sometimes succeed.

  • Isolate the browser. A dedicated container or virtual machine, a fresh profile with no saved sessions, and nothing else of value on the machine.
  • Allowlist domains at the network layer, and check again after redirects.
  • Keep credentials out of the prompt. Ordinary code fetches the secret from a vault and performs the login. The model never sees the password.
  • Give the agent its own account on each system, with the least privilege that does the job. Read-only if the task is read-only, and never a member of staff’s login.
  • Require approval for consequential actions: payments, deletions, form submissions, outgoing messages, accepting terms. Enforce this in your code at the point where the action is executed, not by asking the model to be careful.
  • Record every run: the action log and the screenshots. When something goes wrong, this is how you find out what the agent saw.

Multi-factor authentication is the awkward case. A one-time code sent to someone’s phone cannot be automated without weakening it. Either a person completes the login and the agent continues in that session, or you ask the portal’s owner for a service account.

Running isolated browsers at any volume is as much an infrastructure job as an AI one: container images, network policy, secrets, session recording. Plan it with whoever runs your cloud and platform engineering. For the wider picture on permissions and injection, see our guide to guardrails for AI agents.

How to pilot one

  1. Pick one task that is read-mostly, done at least weekly, and has an answer you can check against another system.
  2. Prove there is no API or export. Email the vendor, and read the site’s terms on automated access.
  3. Record 20 to 30 real runs by the people who do the task now: the steps, the time taken, the odd cases. This becomes your test set.
  4. Build it in a sandbox with a dedicated low-privilege account: scripted login, the agent on the variable pages, code checks on every output.
  5. Run it in shadow for two weeks next to the person, and compare the outputs field by field.
  6. Measure four things: tasks completed without help, tasks that finished with a wrong result, minutes per task and cost per task.
  7. Set the thresholds before the shadow run, and decide who owns the failures.

The number to fear is the second. A task that fails loudly costs a retry. A task that finishes with the wrong invoice status and says nothing costs trust.

In the first week, do steps one to three and nothing else. If the recorded runs show the same six clicks every time on three stable portals, write a script and stop there. If they show thirty layouts and weekly surprises, you have a case for an agent and a test set to hold it to. Building and hardening agents of this kind is part of our AI agents work.

Questions

Asked often.

What is the difference between a browser agent and a computer-use agent?

A browser agent works inside web pages and usually reads the page structure, such as the accessibility tree, so it can act on named elements. A computer-use agent works from screenshots and coordinates, so it can operate any desktop application, including remote desktops and old Windows clients, at the price of more tokens and more missed clicks.

Are browser agents reliable enough for production work?

For narrow, read-mostly tasks with a checkable answer, yes, provided your own code verifies the result and a person approves anything irreversible. Long tasks with many steps still fail often enough to need step limits, retries and a fallback queue. Treat reliability as something you measure on your own tasks, not as a vendor claim.

Is it safe to give an AI agent my login details?

Do not put credentials in the prompt. Let ordinary code log in with a secret from a vault, give the agent its own account with the least privilege that does the job, run the browser in an isolated container with a domain allowlist, and require human approval for payments, deletions and submissions.

Should I use a browser agent if the system has an API?

No. An API call is faster and cheaper, returns the same answer every time and does not break when a button moves. Use a browser or computer-use agent only for the part of the work that has no API, export or database access, and keep everything else in ordinary code.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.