01OBSERVE
Understand what is actually on screen.
Capture the visible desktop, windows, cursor, focus, and every trustworthy structured signal without assuming one privileged interface.
COMPUTER USE · REAL SOFTWARE · REAL OUTCOMES
The computer-use
agent for
real work.
Tell mouse the outcome. It sees the screen, operates the software you already use, verifies the result, and recovers when the interface fights back.
ARBITRARY SOFTWAREPixels are the universal interface.
VERIFIED OUTCOMESSuccess means the final state is correct.
BUILT FOR CONTROLInterrupt, redirect, approve, or take over.
mouse — MISSION 001
Software should not need an API before an agent can use it.
01 OBSERVE THE VISIBLE STATE02 ACT WITHIN AUTHORITY03 VERIFY · RECOVER · FINISH
COMPUTER USE IS THE PRODUCT
See. Act.
Verify.
mouse works through the same visible interfaces people already use. It can take advantage of accessibility, OCR, APIs, and shortcuts, but it never assumes they exist.
REAL-WORK TASK · RUN-0042
Prepare the weekly operations report
APPS3
BUDGET5 min
MODESupervised
STATEVerified
01Observed state
3 SURFACES GROUNDED- XLSOperations workbookTotals repaired · saved locallyVerified
- WEBMetrics dashboardReporting range and source values matchedMatched
- PDFWeekly reportExport completed · final artifact presentReady
02Guarded next action
APPROVAL BOUNDARYAttach the verified PDF in the mail client and confirm that the draft contains the correct recipient, subject, and file.
01 / CAPTURE THE STATE02 / CHOOSE A GUARDED ACTION03 / VERIFY WHAT CHANGED04 / RECOVER OR FINISH
WHY MOUSE
The long tail of software
is still manual.
Real work does not stop at the browser—or at software with a clean API.
Teams depend on browsers, desktop apps, native dialogs, remote sessions, legacy tools, and private business software. mouse starts where that work already happens.
02ACT
Operate the software people already use.
Use mouse, keyboard, shortcuts, menus, files, and application signals to complete work across native, web, Electron, and legacy software.
03VERIFY
Notice failure. Recover. Finish.
Check the resulting state, stop guarded action sequences when the screen diverges, repair focus and timing failures, and escalate honestly.
09:41scene.capturedSpreadsheet · Browser · Mail visible✓09:42action.executedWorkbook formula repaired✓09:43state.verifiedTotals recomputed and matched✓09:44transition.completeReport exported to PDF✓09:45next_action.guardedOutlook draft ready for approval◫runs/run-0042/manifest.jsonPASS · 03:42
+ final artifact matched grader · 0 human interventions · full trajectory preservedTHE SCOREBOARD
Grade the outcome.
Not the clicks.
The north star is held-out end-to-end task completion under a fixed time and model budget. A task passes only when the final user-visible result is correct.
- 01Reset the machine to a known state
- 02Fix task, model, action, and time budgets
- 03Grade the external final state
- 04Replay every success and failure
UNIVERSAL BY DESIGN
Pixels are the floor.
Every signal can help.
Structured interfaces accelerate the agent when they are trustworthy. The visual path preserves coverage when they are absent, incomplete, or wrong.
Capture
Displays, windows, pixels, cursor, DPI, focus, and efficient frame deltas.
Actuation
Mouse, keyboard, scroll, drag, shortcuts, launch, and explicit focus semantics.
Controller
Model adapters, observation packing, planning, grounding, and guarded batching.
Verification
Visual change detection, goal checks, adaptive waits, and honest completion.
Recovery
Unexpected-state detection, focus repair, bounded retries, replanning, and escalation.
Trajectory
Synchronized frames, actions, model I/O, timing, errors, and human corrections.
REAL SOFTWARE COVERAGE
Known apps.
Unknown interfaces.
The benchmark spans familiar productivity software and the poorly instrumented long tail. Optimizing only for “supported apps” misses the product.
ADESKTOP SOFTWARE
System surfaces, file managers, settings, dialogs, and productivity tools.
NATIVE · OFFICE · SYSTEM UI
BMODERN WORK
Browsers, Electron, downloads, uploads, and app transitions.
WEB · ELECTRON · MULTI-APP
CTHE LONG TAIL
Legacy, custom-rendered, canvas, remote, and unfamiliar interfaces.
PIXELS · NATIVE INPUT · RECOVERY
CONTROL AT EVERY BOUNDARY
Delegate without
losing control.
See what mouse sees, what it intends to do, what changed, and whether the outcome is correct. Interrupt, redirect, approve, or take over whenever the task crosses your comfort boundary.
VISIBLEObservation + intended actionEXPLAINED
SCOPEDApps + accounts + permissionsBOUNDED
INTERRUPTIBLEPause + redirect + take overANYTIME
# observe · approve · act · verify · recoverAUTONOMY EXPANDS ONLY WHEN PERFORMANCE HOLDSBRING US A REAL TASK
Show us where
today's agents fail.
We'll put it on the scoreboard.
Bring the software, the expected outcome, and the edge cases that make the work real.
Bring us a task ↗