AskUI

AskUI is a vision-powered automation framework that controls real computers by looking at the screen — no selectors, no DOM hooks, no accessibility trees. Its PTA-1 (Prompt-to-Action) model converts natural-language instructions into clicks, typing, scrolling, and drag-and-drop across Windows, macOS, Linux, Android, iOS, and Citrix environments. You can script precise single-step commands RPA-style or give the agent a goal (agent.act("search for a flight from Berlin to Paris")) and let it navigate autonomously.

The stack has two halves: Agent OS, a device controller that takes screenshots and drives mouse/keyboard (including in-background automation on Windows via a second session), and the VisionAgent Python SDK with pluggable model backends — use AskUI-hosted models, Anthropic, OpenRouter, or bring your own, and retrain on your own data on-premise. That enterprise posture (secure deployment, model hot-swapping, retraining infrastructure) is why it shows up in QA and RPA conversations rather than just hobbyist ones.

The honest caveat: vision-based automation is inherently slower and flakier than API automation — treat it as the tool for UIs with no API, not a replacement for one. Linux currently needs XOrg (no Wayland support for Agent OS). The SDKs are MIT-licensed open source, but the hosted models and suite are commercial.

1. Install Agent OS on the target machine (download the installer for your OS from the docs; Linux needs XOrg).
2. pip install askui and set your model credentials (e.g. ANTHROPIC_API_KEY).
3. Write a script: with VisionAgent() as agent: agent.act("your goal in plain language").
4. For step-level control, use agent.click("the login button"), .type(), .get() to read screen state.