Microsoft’s Mage-VL-Cowork lets one small model choose between clicks and code

Editing a spreadsheet can involve clicking through a window or changing the file with code. Microsoft’s newly public Mage-VL-Cowork research model puts both routes inside one policy, letting it choose a graphical action or a terminal command at each turn.

The team describes Mage-VL-Cowork as a 4B-class computer-use model. Its weights are available on Hugging Face under Apache 2.0, with implementation and evaluation documentation on GitHub. This is a research release that developers can examine and run, rather than an announcement that a new assistant has arrived on everyone’s desktop.

The distinction is practical. Microsoft’s documentation assigns visual layouts and live application state to screenshot-based mouse and keyboard interaction. Structured changes to documents, spreadsheets and presentations can instead use code, command-line tools and application libraries. The model selects between those channels without a separate external routing agent.

That makes tool selection part of what the model learns. A command can manipulate structured data directly; a graphical action can work with something visible on screen. The useful question is whether the agent can choose an appropriate route and still produce the requested result.

Training the choice

The published recipe builds the capabilities separately before combining them: visual grounding and graphical execution, then coding and terminal work, followed by hybrid supervised training and reinforcement learning. The team also refined the teacher’s tools and execution environment before collecting demonstrations. Simply handing an agent more tools was not the whole method.

Microsoft reports 66.0% on OSWorld-Verified with a 50-step budget. Its documented evaluation configuration uses 361 tasks, excluding Google Drive tasks. Those conditions belong beside the number: it is a developer-reported result on a particular task set, not an independently reproduced measure of success across all computer work.

The benchmark’s own documentation explains why the task count matters. OSWorld includes 369 tasks but permits excluding eight Google Drive tasks affected by network setup. It uses configured starting states and execution-based evaluation scripts. Its site also points readers to OSWorld 2.0, available since June; Mage’s reported number is labeled OSWorld-Verified and should not be silently transferred to that newer benchmark.

The release makes the approach inspectable, while leaving deployment questions open. Microsoft’s model card specifies isolated, disposable environments and recommends explicit human confirmation for irreversible actions. A model that can issue shell commands needs a carefully bounded place to act.

For researchers, the most useful next comparison is between the agent’s choices and its actual outputs: when did a click help, when did code help, and did the saved file or application state satisfy the task? Public weights and evaluation code make that investigation possible. The headline score alone cannot answer it.