Skip to content

feat: multi Agent OS server support in AskUiControllerClient#276

Open
mlikasam-askui wants to merge 15 commits into
mainfrom
SOLENG-360-refactor/askui-controller-multi-target
Open

feat: multi Agent OS server support in AskUiControllerClient#276
mlikasam-askui wants to merge 15 commits into
mainfrom
SOLENG-360-refactor/askui-controller-multi-target

Conversation

@mlikasam-askui
Copy link
Copy Markdown
Contributor

Summary

AskUiControllerClient can now manage multiple Agent OS servers (local
and/or remote) at once. One is active at a time and receives the agent's
actions; callers can switch at runtime or scope a switch to a with block.

New API:

  • AgentOsServer (base), LocalAgentOsServer (replaces the old
    AskUiControllerServer — owns the local controller subprocess; auto-detects
    the Windows AskuiCoreService and switches to its port), RemoteAgentOsServer
    (already-running remote controller, no process management).
  • AgentOsServerManager — enforces invariants: at most one local, unique
    session GUIDs / computer_ids / remote addresses.
  • On AskUiControllerClient and exposed through AgentOs /
    ComputerAgentOsFacade: add_agent_os_server, add_remote_agent_os_server,
    list_agent_os_servers, get_active_agent_os_server,
    switch_agent_os_server, reset_agent_os_servers, temporary_select.
  • AndroidAgentOs / PpadbAgentOs get a sibling temporary_select(device_sn).
  • ComputerAgent auto-registers three new act() tools so the LLM can drive
    multi-target flows: ComputerListAgentOsServersTool,
    ComputerSwitchAgentOsServerTool, ComputerGetActiveAgentOsServerTool.
  • LocalAgentOsServer and RemoteAgentOsServer are re-exported from the
    top-level askui package.
  • Version bumped to 0.33.0.

New unit tests cover the server classes, the manager, the multi-target client,
and the new computer tools. The e2e test was updated to the new constructor.

Breaking changes

  1. AskUiControllerServer removed. Use LocalAgentOsServer instead (same
    responsibility).

  2. AskUiControllerClientSettings removed. The env vars
    ASKUI_CONTROLLER_CLIENT_SERVER_ADDRESS and
    ASKUI_CONTROLLER_CLIENT_SERVER_AUTOSTART no longer have any effect.
    Configure explicitly:

    AskUiControllerClient(
        agent_os_servers=[LocalAgentOsServer(address="127.0.0.1:24000")],
    )

    For the old "do not autostart" behavior, use RemoteAgentOsServer (never
    starts a process) or LocalAgentOsServer(is_service=True).

  3. AskUiControllerClient.__init__ signature changed. Removed
    controller_server and settings; added
    agent_os_servers: list[AgentOsServer] | None. connect() now requires at
    least one registered server.

  4. ComputerAgent.__init__ signature changed. Removed tools: AgentToolbox;
    added agent_os_servers: list[AgentOsServer] | None. Callers that injected
    a custom toolbox must subclass ComputerAgent or build the toolbox + agent
    themselves.

  5. AgentOs gained 7 abstract-ish methods (default raises
    NotImplementedError). Non-AskUI backends (Playwright, custom) that don't
    override them will raise when the new computer tools call
    get_active_agent_os_server — which they do for screenshot,
    get_mouse_position, get_system_info, list_displays, and
    retrieve_active_display.

  6. Computer tool output prefix changed. The five tools above now return
    "[Server with id '<computer_id>']: ...". Anything parsing the raw strings
    needs updating.

  7. ToolCollection.get_agent_os_by_tags keyword renamed: tags
    required_tags. Positional calls are unaffected.

  8. Exception messages changed.

    • AskUiControllerOperationTimeoutError default message rewritten; when
      raised by _run_recorder_action it now includes the active server's
      description, session GUID, timeout window, retry count, and action class
      id.
    • AskUiControllerInvalidCommandError message reworded; details lines
      prefixed with Server details:.

    Tests asserting on exact message strings need updating.

mlikasam-askui and others added 13 commits May 8, 2026 12:17
Replace single-controller config with a TargetComputer / TargetComputerManager
abstraction. AskUiControllerClient now manages a list of local and remote
controller servers, opens a gRPC connection per target on connect(), and routes
agent-os actions to a single active target. Add tools for the agent to list,
switch, and inspect the active target computer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant