Agents reading an app’s accessibility tree instead of screenshots, parallel agents each with their own cursor, and claimed lower token use: that’s the approach described for agent-desktop.
Chris Tate reports that Vercel Labs has open-sourced the tool, describing it as a way for any coding agent to operate macOS, Windows and Linux apps.
How it works
The operating system’s accessibility tree represents an app’s interface as structured information. The announced approach lets an agent work through that structure instead of interpreting a screenshot—a route for coding agents to interact with desktop apps beyond writing code.
Tate’s post also describes parallel agents each getting their own cursor and claims “far lower token use.” Tokens are the units of information a model processes; the claimed reduction concerns how much information the agent needs to process.
What still needs testing
Tate’s post provides no measurements or comparison details for the token claim, and no task results, failure rates or tests across the three operating systems. The announced capability therefore leaves reliability on real tasks unproven.
For teams considering desktop automation, the useful next step is a bounded trial in the actual app: check whether the agent completes the intended task correctly, handles interruptions and leaves a result that can be verified.