Agent S

• Automating a repetitive task in a desktop app that has no API, such as moving data between a legacy Windows program and a spreadsheet
• Embedding a computer-use agent in a Python application through the `AgentS3` class and your own screenshot-and-execute loop
• Comparing main models and grounding models on the same GUI tasks by swapping providers in the engine parameters
• Reproducing or extending published computer-use results with the repository's OSWorld, OSWorld V2, ScienceBoard, and WindowsAgentArena setup guides
• Running Agent S4's interface evolution to improve an open-weight model on OSWorld V2 or ScienceBoard tasks

Agent S is one of the more credible places to start if you want to understand or build a computer-use agent from open code. It covers all three desktop operating systems, lets you choose any major model provider, and the research behind it is published and peer reviewed. Read the headline numbers with care, though. The 72.6% OSWorld result depends on Behavior Best-of-N, which the CLI does not run, and the new S4 figures measure an evolution pipeline on AWS rather than the agent you install with pip. Plan for the operational work: hosting a grounding model, running on a single monitor, and supervising an agent that holds your mouse and keyboard. Teams that just want the outcome should compare the effort with Simular's hosted Sai.

About Agent S

Agent S is a research framework for agents that use a computer through its screen, mouse, and keyboard rather than through app APIs. The current desktop agent is Agent S3, shipped in the `gui-agents` Python package (version 0.3.2). You install it with `pip install gui-agents`, plus Tesseract for OCR, and run it either from the `agent_s` command line or from Python through the `AgentS3` class. Each step takes a screenshot. A main model you choose (OpenAI, Anthropic, Gemini, Azure OpenAI, OpenRouter, or a local model served by vLLM) decides what to do, and a separate grounding model maps that decision to screen coordinates. The output is pyautogui code that runs on your machine. The grounding model is required: Simular recommends UI-TARS-1.5-7B on a Hugging Face Inference Endpoint or another host, paired with gpt-5-2025-08-07 as the main model. An optional local coding environment lets the agent run Python and Bash for jobs such as spreadsheet edits, which is faster than clicking through a GUI but means it executes arbitrary code with your user permissions. The repository reports that Agent S3 reached 72.6% on OSWorld with Behavior Best-of-N (December 2025), which Simular describes as the first result above human-level performance there. The bundled CLI runs Agent S3 without that multi-rollout selection. Agent S4, released on October 20, 2026 with the paper "Evolving Agent Interfaces", is a different kind of tool. It is a separate `agent-s4` package that runs on a Linux host and uses a meta-agent to write and revise the Python tool libraries a computer-use agent works through, then scores the variants on OSWorld V2 or ScienceBoard tasks running on AWS VMs. The paper reports that Qwen3.8-27B with evolved interfaces reached 21.3% on OSWorld V2, versus 13.0% for GPT-5.5 and 20.6% for Claude Opus 4.8, at about 4.3% and 1.5% of their respective costs. For people who want to skip setup, the README points to Simular Cloud, which describes itself as powered by Agent S. Simular also sells Sai, a hosted computer-use agent built on this research, and the Simular API, which exposes Sai over HTTP and MCP. The open-source framework itself has no paid tier: you pay your model providers and whoever hosts the grounding model. The tradeoff is that this is a research codebase, so expect to wire up model endpoints, keep to a single monitor, and supervise an agent that has real control of your machine.