Vercel agent-desktop: Computer Use Without Screenshots

Vercel Labs released agent-desktop on October 31, 2026, and it reached 6.9K GitHub stars within days. The premise is simple: AI coding agents can now control your entire desktop — not just the browser — without relying on screenshots or a vision model. Built in Rust, licensed Apache-2.0, and designed to work with Claude Code, Codex, Cursor, or any agent that can run shell commands.

Vercel CTO Malte Ubl described it as “the same token-efficient computer use you know and love from agent-browser but for the entire desktop operating system.” If you use Vercel’s agent-browser for web automation, agent-desktop extends that approach to native apps on macOS, Windows, and Linux.

Why Screenshot-Based Computer Use Is Expensive and Fragile

The dominant approach to AI agent “computer use” — from OpenAI Operator to Anthropic’s CUA — relies on taking screenshots and feeding them to a vision model. It works. But it costs.

A single screenshot fed to a vision model consumes roughly 50,000 tokens. An accessibility tree snapshot of the same screen costs around 4,000. On a tested 52-step task on Apple Maps, switching from screenshots to tree-based interaction reduced total token consumption from 198,000 tokens to 31,000 — an 84% reduction. At current API pricing of $2–10 per million tokens, that difference is real money for teams running heavy agentic workflows. Snapshot speed is also notably faster: accessibility tree reads take 50-200ms; screenshot vision takes 3-8 seconds.

Screenshot-based approaches also break under DPI scaling, theme changes, and multi-monitor setups. Accessibility trees are structured data — deterministic, compact, and already exposed by every OS for screen reader support. They don’t care what your color theme is.

How agent-desktop Works

agent-desktop reads the operating system’s accessibility tree through platform-specific APIs: AX API on macOS, UI Automation on Windows, and AT-SPI on Linux. It assigns a stable reference (“ref”) to every actionable UI element and returns them in a JSON envelope. The workflow is a three-step loop:

1. Snapshot — capture the current UI state, receive refs for every interactive element
2. Act — perform an action using a ref (click, type, scroll, and more)
3. Verify — take another snapshot to confirm the action succeeded

Refs like @s8f3k2p9:e8 are stable within a snapshot. If the UI changes after the snapshot (a dialog opens, a window moves), the ref goes stale and the tool returns a STALE_REF error with a disposition field telling the agent whether retrying is safe. This explicit error design means agents can handle UI changes gracefully rather than silently acting on a wrong element.

Getting Started in Three Commands

npm install -g agent-desktop
agent-desktop version
agent-desktop permissions --guide  # macOS only — grants Accessibility access

To integrate with your existing coding agent as a skill:

npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

After that, the agent can call agent-desktop snapshot --app "Finder" -i to get a list of interactive elements, act on a ref, and snapshot again to verify. No new model subscription. No new API key. Any agent that can run shell commands can use it immediately.

56 Commands, Cross-Platform, Background Operation

The CLI ships 56 operational commands across observe (snapshot, find, get, screenshot, list-surfaces), act (click, type, set-value, scroll, toggle, check, expand), app management (launch, close-app, list-apps, list-windows), clipboard (get, set, clear), and keyboard/mouse (press, hover, drag, mouse-wheel). Four additional low-level commands require daemon mode.

Agents run headlessly in the background by default — your cursor doesn’t jump, your windows don’t lose focus. Multiple agents can run in parallel on the same machine, each with its own colored overlay cursor on supported platforms. Supported systems: macOS 13+, Windows 10 (v1809+), and Ubuntu 24.04+ with AT-SPI in a graphical session.

Tradeoffs to Know Before You Deploy

agent-desktop is not a sandboxed VM. Agents interact with your actual desktop. Snapshots, clipboard reads, and screenshots can expose sensitive content — do not pipe this output to public logs or shared environments without filtering.

The tool is only as reliable as the target app’s accessibility support. Most mainstream apps — browsers, terminals, Electron apps like VS Code and Notion, standard productivity tools — expose accessibility trees properly. Some proprietary software, heavily customized UIs, and games may not. Run a quick snapshot test before committing to a workflow. For web page automation, use agent-browser instead — agent-desktop does not handle web pages.

On macOS, the Accessibility permission grant is required and must be done manually in System Settings. macOS sessions also have a 15-minute idle timeout; long-running automated tasks need periodic activity to stay alive.

Where This Fits in the Agentic Stack

As of 2026, agents initiate 40% of GitHub pull requests. Browser automation is mostly solved. Reliable, cost-effective, model-neutral desktop automation has been the gap. agent-desktop fills it without requiring you to adopt a new model or provider.

The 6.9K stars and 504 forks in under two weeks suggest the developer community has been waiting for exactly this. If you are building agentic coding workflows, or want Claude Code or Codex to handle desktop tasks you would otherwise do manually, this is worth an afternoon to evaluate. Start with a simple task — snapshot an app, read its state, perform one action — and verify whether the accessibility tree for your target app is rich enough to build on. The growth trajectory suggests the ecosystem will keep up.