What is computer use in AI?
Computer use is the ability of an AI agent to operate software through the screen, keyboard, and mouse, the same way a person does. The agent works toward a goal one action at a time, reading the screen and then clicking, typing, or scrolling.
At each step, the agent runtime sends a screenshot to a vision-language model with the goal and the actions so far. The model returns the next action and where on the screen to perform it. The runtime carries it out inside a virtual machine or browser, and a new screenshot starts the next turn. Browser agents work only inside a web browser and can also read the page structure, which is faster and more precise than reading pixels.
Why computer use matters
Many systems that businesses rely on have no usable API, including older desktop applications, government filing sites, and supplier portals. The established way to automate them is robotic process automation, whose scripts expect fixed screens and break when a layout changes. A computer use agent reads the screen fresh at every step, so a moved button rarely stops it.
The approach has three costs. Each step is a model call on an image, so a task that an API finishes in a second can take minutes. Long tasks fail more often than short ones, because one wrong click can derail the rest. The agent also reads everything on the screen, including text planted to manipulate it, which makes prompt injection a practical risk. Where an API or a Model Context Protocol (MCP) connector exists, it remains the better choice.
Computer use in practice
A third-party logistics provider books delivery appointments in the scheduling portals of dozens of retail distribution centers. None of the portals offer an API, and each has its own login and layout. A computer use agent runs in an isolated virtual machine with a separate account for each portal. For each load in the transportation management system, it books the earliest slot that fits the load’s ready date and records the confirmation number. When nothing fits, a scheduler gets the load with a screenshot of the options the agent saw.
What to watch for
- Run the agent in a sandboxed virtual machine or browser with its own credentials rather than on a person’s logged-in desktop.
- Require human confirmation before irreversible actions such as submitting a payment, sending a message, or deleting a record.
- Check each site’s terms of use, because some portals prohibit automated access.
- Keep screenshots of each step for the audit log, and protect them, since they can capture personal data.