feat: multi Agent OS server support in AskUiControllerClient#276
Open
mlikasam-askui wants to merge 15 commits into
Open
feat: multi Agent OS server support in AskUiControllerClient#276mlikasam-askui wants to merge 15 commits into
AskUiControllerClient#276mlikasam-askui wants to merge 15 commits into
Conversation
Replace single-controller config with a TargetComputer / TargetComputerManager abstraction. AskUiControllerClient now manages a list of local and remote controller servers, opens a gRPC connection per target on connect(), and routes agent-os actions to a single active target. Add tools for the agent to list, switch, and inspect the active target computer.
… target session GUID
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AskUiControllerClientcan now manage multiple Agent OS servers (localand/or remote) at once. One is active at a time and receives the agent's
actions; callers can switch at runtime or scope a switch to a
withblock.New API:
AgentOsServer(base),LocalAgentOsServer(replaces the oldAskUiControllerServer— owns the local controller subprocess; auto-detectsthe Windows
AskuiCoreServiceand switches to its port),RemoteAgentOsServer(already-running remote controller, no process management).
AgentOsServerManager— enforces invariants: at most one local, uniquesession GUIDs /
computer_ids / remote addresses.AskUiControllerClientand exposed throughAgentOs/ComputerAgentOsFacade:add_agent_os_server,add_remote_agent_os_server,list_agent_os_servers,get_active_agent_os_server,switch_agent_os_server,reset_agent_os_servers,temporary_select.AndroidAgentOs/PpadbAgentOsget a siblingtemporary_select(device_sn).ComputerAgentauto-registers three newact()tools so the LLM can drivemulti-target flows:
ComputerListAgentOsServersTool,ComputerSwitchAgentOsServerTool,ComputerGetActiveAgentOsServerTool.LocalAgentOsServerandRemoteAgentOsServerare re-exported from thetop-level
askuipackage.New unit tests cover the server classes, the manager, the multi-target client,
and the new computer tools. The e2e test was updated to the new constructor.
Breaking changes
AskUiControllerServerremoved. UseLocalAgentOsServerinstead (sameresponsibility).
AskUiControllerClientSettingsremoved. The env varsASKUI_CONTROLLER_CLIENT_SERVER_ADDRESSandASKUI_CONTROLLER_CLIENT_SERVER_AUTOSTARTno longer have any effect.Configure explicitly:
For the old "do not autostart" behavior, use
RemoteAgentOsServer(neverstarts a process) or
LocalAgentOsServer(is_service=True).AskUiControllerClient.__init__signature changed. Removedcontroller_serverandsettings; addedagent_os_servers: list[AgentOsServer] | None.connect()now requires atleast one registered server.
ComputerAgent.__init__signature changed. Removedtools: AgentToolbox;added
agent_os_servers: list[AgentOsServer] | None. Callers that injecteda custom toolbox must subclass
ComputerAgentor build the toolbox + agentthemselves.
AgentOsgained 7 abstract-ish methods (default raisesNotImplementedError). Non-AskUI backends (Playwright, custom) that don'toverride them will raise when the new computer tools call
get_active_agent_os_server— which they do for screenshot,get_mouse_position,get_system_info,list_displays, andretrieve_active_display.Computer tool output prefix changed. The five tools above now return
"[Server with id '<computer_id>']: ...". Anything parsing the raw stringsneeds updating.
ToolCollection.get_agent_os_by_tagskeyword renamed:tags→required_tags. Positional calls are unaffected.Exception messages changed.
AskUiControllerOperationTimeoutErrordefault message rewritten; whenraised by
_run_recorder_actionit now includes the active server'sdescription, session GUID, timeout window, retry count, and action class
id.
AskUiControllerInvalidCommandErrormessage reworded; details linesprefixed with
Server details:.Tests asserting on exact message strings need updating.