Add unified CUA template with multi-provider fallback#143
Open
masnwilliams wants to merge 15 commits intomainfrom
Open
Add unified CUA template with multi-provider fallback#143masnwilliams wants to merge 15 commits intomainfrom
masnwilliams wants to merge 15 commits intomainfrom
Conversation
Contributor
|
🔧 CI Fix Available |
Consolidates the separate anthropic-computer-use, openai-computer-use, and gemini-computer-use templates into a single "cua" template that supports all three providers with automatic fallback. - TypeScript and Python templates with identical structure - Provider selection via CUA_PROVIDER env var - Optional fallback chain via CUA_FALLBACK_PROVIDERS - Shared browser session lifecycle with replay support - Each provider adapter is self-contained and customizable - Registered as "cua" template in templates.go Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
masnwilliams
force-pushed
the
hypeship/unified-cua-template
branch
from
99891de to
73255f9
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Provider resolution at module load crashes during Hypeman's build/discovery phase when env vars aren't available. Use lazy initialization so providers are resolved on first invocation instead. Also fix TS type errors: narrow candidate.content in Gemini provider, cast input items in OpenAI provider, simplify computer_call_output construction. Made-with: Cursor
…odel inputs - Bump all TS and Python deps to latest versions - Fix Anthropic computer use: use computer_20251124 with computer-use-2025-11-24 beta flag (claude-sonnet-4-6 requires the newer tool version) - Fix OpenAI: add missing screenshot action handler - Fix Python: correct SDK API (kernel.App), fix session.delete call, add missing openai dependency - Restore provider and model as per-request payload overrides (were dropped in rewrite). Provider uses a typed enum (anthropic | openai | gemini). Made-with: Cursor
… API, session delete - Add missing screenshot action handler in Python OpenAI provider - Use Part.from_function_response() instead of FunctionResponsePart() in Python Gemini provider (pydantic extra_forbidden in google-genai >=1.71) - Fix session cleanup: use delete_by_id() instead of delete() Made-with: Cursor
…n Gemini - TS Anthropic: move SYSTEM_PROMPT to getSystemPrompt() function so the date is computed per-request instead of freezing at module load - Python Gemini: include screenshot data as inline_data Part alongside function responses so the model can see action results - Remove unused PREDEFINED_ACTIONS list from Python Gemini Made-with: Cursor
…eout) Add optional `browser` field to CUA payload for per-request browser session configuration. Supports proxy_id, profile (id/name/save_changes), extensions, and timeout_seconds. Viewport and stealth remain deploy-time defaults since CUA providers depend on consistent viewport dimensions. Made-with: Cursor
When session_id is provided in the payload, the CUA task uses that existing browser session directly instead of creating a new one. The caller is responsible for the session lifecycle. This lets users pre-configure browsers with any settings and reuse sessions across tasks. Made-with: Cursor
When an external session_id is provided, retrieve the browser's real viewport dimensions via browsers.retrieve() instead of hardcoding 1280x800. This ensures coordinate mapping is correct regardless of how the browser was created. Made-with: Cursor
dprevoznik
reviewed
Apr 9, 2026
| // Shared interface every provider adapter must implement. | ||
| export interface TaskOptions { | ||
| query: string; | ||
| model?: string; |
Contributor
There was a problem hiding this comment.
Each provider hardcodes model-specific API features that will break if you swap in a different model. Sharing Cursor's analysis, which is pretty aligned with what i experienced building other templates. I think it would be smart to at least have in the template which models from which providers are compatible. I don't think we need to lock it down to specific models, particularly if we have defaults.
Cursor Reco
The model field should either be removed from the public payload (keep it as an env var only) or the template should validate model compatibility per provider before calling the API. At minimum, the README and input description should warn that only specific computer-use-capable models work. Right now a user seeing that free-text field in the dashboard will absolutely try claude-haiku-3-5 or gpt-4o and get a confusing 400 error.
dprevoznik
reviewed
Apr 9, 2026
| break; | ||
| } | ||
| case 'scroll_document': | ||
| case 'scroll_at': { |
Contributor
There was a problem hiding this comment.
The Gemini provider in both ts-cua and python-cua is not likely optimized for scroll behavior. In the standalone templates, we have logic in place (though not perfect) to handle the fact that Gemini reports back a magnitude as the value in pixels.
Cursor summary of issue:
It's missing the magnitude ÷ 60 pixel-to-notch conversion and the max(1, min(17, ...)) clamp. When Gemini asks to scroll with its default magnitude (~400 pixels), the unified template will fire 400 wheel notches instead of 7. Both TS and Python have the same bug — they're consistent with each other, but both wrong compared to the standalone template.
Getting API exhaustion errors with gemini right now, but wanted to surface this. I put in the % 60 and clamp logic in the standalone templates. It likely could be improved
Contributor
There was a problem hiding this comment.
The behavior in anthropic and openai providers matches the standalone templates right now.
dprevoznik
reviewed
Apr 9, 2026
Contributor
dprevoznik
left a comment
dprevoznik
left a comment
There was a problem hiding this comment.
Left some comments. All three providers working on both ts and python versions. Though Gemini api exhaustion error made it so I couldn't test the scroll logic comment I made.
Prefer PageUp and PageDown in the provider prompts so long-page navigation is more reliable across the unified CUA templates. Made-with: Cursor
Bring in main branch template updates and resolve the registry overlap by keeping both the unified CUA and Tzafon template entries. Made-with: Cursor
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
dprevoznik
approved these changes
Apr 16, 2026
Contributor
dprevoznik
left a comment
dprevoznik
left a comment
There was a problem hiding this comment.
Other models look good. Everything working.
Gemini feedback:
System-prompt level Scroll behavior change makes sense. We may still want to change the default scroll action handler logic as I suggested since if the model chooses to use regular scroll it will be large jumps currently. Thoughts?
## Summary Stacks on top of the unified CUA template branch. Fixes the Gemini scroll handler bug Danny flagged in review. Gemini's computer-use API reports scroll `magnitude` in **pixels** (default ~400), but `computer.scroll`'s `delta_x` / `delta_y` expects **wheel notches**. The cua adapter was passing `magnitude` through unchanged, so a default Gemini scroll fired ~400 notches instead of ~7. The standalone `gemini-computer-use` template already does the right thing — this just brings the unified adapter in line: - default magnitude: `3` → `400` (pixels, matching Gemini's spec) - divide by `PX_PER_NOTCH` (60) and clamp to `MAX_NOTCHES_PER_ACTION` (17) - applied symmetrically in TS (`providers/gemini.ts`) and Python (`providers/gemini.py`) The `anthropic` and `openai` adapters already match their standalone equivalents — no changes needed there. ## Test plan - [ ] `go build ./...` passes (verified locally) - [ ] `go test ./pkg/create/...` passes (verified locally) - [ ] Deploy CUA template with Gemini provider, ask it to scroll a long page; confirm scroll distance is page-sized, not catastrophic - [ ] Repeat for Python template Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
4 tasks
…155) ## Summary Stacks on top of the unified CUA template branch. Addresses three of the outstanding Cursor Bugbot findings on PR #143. ### Gemini argument-name mismatches Both bugs cause the affected actions to be silently/structurally broken — fixed in TS and Python in lockstep with the standalone `gemini-computer-use` templates. - **`key_combination`** read from `args.key_combination` → now reads `args.keys`. Gemini's schema sends the combo in `keys` (a single `+`-joined string), so the previous code always saw an empty combo and silently dropped every shortcut. - **`drag_and_drop`** read from `start_x / start_y / end_x / end_y` → now reads `x / y / destination_x / destination_y`. Gemini's schema uses the latter; the previous names always resolved to `0`, so every drag went `(0,0) -> (0,0)`. ### Session cleanup ordering (TS + Python) `session.stop()` previously placed the state-reset (`_sessionId = null`, …) **after** the `try / finally` that performs replay-stop + browser-delete. If `stopReplay()` threw, the `finally` deleted the browser, the exception then propagated past the cleanup lines, and a follow-up `stop()` from the caller's error path would attempt to delete the already-destroyed session — masking the original error. Moved the state-reset into the `finally`, so a second `stop()` is a safe no-op regardless of how the first attempt unwound. ### Bugbot findings I checked but did *not* change These are flagged on the PR but already fixed in the current branch (probably resolved between scans): - Anthropic system prompt date freezing — already a `getSystemPrompt()` function that reads `new Date()` per task. - Python `session.py` using `browsers.delete` — already uses `browsers.delete_by_id`. - Python OpenAI provider missing `screenshot` action — already returns `[]` for `screenshot`. - Python Gemini dropping screenshots — already appends an `inline_data` `Part` per response when `result["screenshot"]` is set. - `PREDEFINED_ACTIONS` unused — leftover constant; harmless. Left as-is to keep the diff focused. ## Test plan - [x] `go build ./...` - [x] `go test ./pkg/create/...` - [ ] Deploy CUA template with Gemini provider, ask it to perform a drag, then a keyboard shortcut (e.g. ctrl+a, ctrl+l) — confirm both succeed - [ ] Force a replay-stop failure (e.g. invalid replay state) and confirm session.stop() can be called twice without crashing <!-- CURSOR_SUMMARY --> --- > [!NOTE] > <sup>[Cursor Bugbot](https://cursor.com/bugbot) is generating a summary for commit 1c6b554. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
3 tasks
## Summary Stacks on top of the unified CUA template branch. Addresses the new High-Severity Cursor Bugbot finding on PR #143's latest scan: `stop()` throws on repeated call instead of being a no-op. ## Background PR #155 moved the session-state reset (`_sessionId = null`, …) into the `finally` so that a thrown replay-stop or browser-delete error wouldn't leave stale state behind. That fix was correct, but it exposed a latent bug: `stop()` opens with `const info = this.info` (TS) / `info = self.info` (Py), and the `info` getter delegates to the `sessionId` property, which raises when `_sessionId` is null. So once PR #155 reliably cleared `_sessionId` on the first call, the caller's error-path retry would hit the throwing getter and mask the original exception — exactly the failure mode PR #155 was meant to prevent. ## Fix `stop()` now: 1. Short-circuits at the top with a sentinel `SessionInfo` when no session is active — never touches the throwing getter. 2. Builds the live-session `info` from local fields directly so the body never depends on `this.info` / `self.info` either. Symmetric in TS (`session.ts`) and Python (`session.py`). ## Test plan - [x] `go build ./...` - [x] `go test ./pkg/create/...` - [ ] Force a replay-stop failure (e.g. delete the replay out-of-band, or pass an invalid replay id) and confirm calling `session.stop()` twice from the caller's error path no longer raises and the original error surfaces. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches session shutdown/cleanup logic in both Python and TypeScript templates; while small, mistakes could lead to leaked browser sessions or masked errors during teardown. > > **Overview** > `KernelBrowserSession.stop()` in both `pkg/templates/python/cua/session.py` and `pkg/templates/typescript/cua/session.ts` is updated to be **idempotent**. > > It now short-circuits when no active session exists (returning a sentinel `SessionInfo` without reading `info`/`sessionId`), and builds the returned `SessionInfo` directly from internal fields so teardown can’t fail due to accessing a getter after state has been cleared. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit db0bdea. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c684ca7. Configure here.
…used openai dep (#157) ## Summary Two bugbot findings on commit \`c684ca7\`: 1. **Medium** — Python Gemini provider sent screenshots as a separate \`Part(inline_data=...)\` entry in the user content after the \`FunctionResponse\` part. With multiple function calls per turn the model can't bind a screenshot to its originating call. The standalone \`python/gemini-computer-use\` template and the TS unified template both nest the screenshot as a \`FunctionResponsePart\` inside \`FunctionResponse.parts\`. This PR matches that structure and adds the predefined-actions allowlist that gates screenshot inclusion. 2. **Low** — \`openai\` was listed in \`pyproject.toml\` but never imported. The OpenAI provider uses raw \`httpx\` against the Responses API. Removed. ## Test plan - [ ] Smoke run python cua with Gemini against a multi-call turn and confirm screenshot binds to the originating call - [ ] \`uv sync\` after dep change <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Moderate risk because it changes the structure of Gemini tool-call response parts, which could affect how multi-call turns are interpreted by the model or SDK. Dependency removal is low risk but may impact downstream installs if they relied on the extra package. > > **Overview** > **Gemini Python CUA now nests screenshots inside each tool call response.** Instead of sending a standalone `Part(inline_data=...)` after the `FunctionResponse`, screenshots are attached as `FunctionResponse.parts` (as `FunctionResponsePart`/`FunctionResponseBlob`) so multi-call turns can reliably associate images with the correct action; screenshot inclusion is gated by a `PREDEFINED_ACTIONS` allowlist. > > **Template deps cleanup.** Removes the unused `openai` dependency from `pyproject.toml`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ee48a5c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cuatemplate (TypeScript + Python) that consolidates the separateanthropic-computer-use,openai-computer-use, andgemini-computer-usetemplates into a single multi-provider templateCUA_PROVIDERenv var, with automatic fallback viaCUA_FALLBACK_PROVIDERStemplates.gofor both TypeScript and PythonStructure
Test plan
go build ./...passesgo test ./pkg/create/...passeskernel createshows "Unified CUA" template for both TS and Python🤖 Generated with Claude Code
Note
Medium Risk
Adds a sizable new template with provider-selection/fallback logic and browser session lifecycle/replay handling, which may affect new-user flows and external API integrations. Existing templates are mostly untouched aside from template registry/sorting updates.
Overview
Adds a new
cua(“Unified CUA”) template for both TypeScript and Python that runs a computer-use agent against Anthropic/OpenAI/Gemini withCUA_PROVIDERselection and optionalCUA_FALLBACK_PROVIDERSautomatic fallback.Registers the new template in
pkg/create/templates.go(including deploy/invoke samples and template ordering), and introduces new template projects underpkg/templates/{typescript,python}/cuawith provider-specific adapters, a shared browser session manager (including optional replay recording), and accompanying.env.example/README/dependency files.Reviewed by Cursor Bugbot for commit 51b69fb. Bugbot is set up for automated code reviews on this repo. Configure here.