Add cost-tracker workflow for agent spend observability#319
Add cost-tracker workflow for agent spend observability#319dsyme merged 4 commits intogithubnext:mainfrom
Conversation
|
Thank you @adamhenson! |
|
|
||
| Aggregate token counts by model across all lines. Use this pricing table (USD per 1M tokens): | ||
|
|
||
| | Model | Input | Output | Cache write | Cache read | |
There was a problem hiding this comment.
Re-reviewing - hardwiring this model cost table is likely not so great. But I'll leave it as is for now
There was a problem hiding this comment.
Pull request overview
Adds a new sample “Cost Tracker” agentic workflow to provide per-run spend observability by consuming token-usage.jsonl produced by the gh-aw firewall, then reporting costs back to the related PR (or opening an issue when no PR is associated).
Changes:
- Add
workflows/cost-tracker.mdworkflow that triggers onworkflow_run: completed, downloadsagent-artifacts, computes per-model/total cost, and posts a report. - Add
docs/cost-tracker.mdwith install/config guidance and a Mermaid flow diagram. - Update
README.mdto list Cost Tracker under “Fault Analysis Workflows”.
Show a summary per file
| File | Description |
|---|---|
| workflows/cost-tracker.md | New workflow prompt that downloads artifacts, parses token-usage.jsonl, calculates costs, and posts results via safe-outputs. |
| docs/cost-tracker.md | Documentation for installing and configuring Cost Tracker, including data source and optional high-spend alert. |
| README.md | Adds Cost Tracker to the workflow catalog. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 4
| high-spend alert issue when a single run exceeds a configurable threshold. | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: ["agent-implement", "agent-pr-fix"] # Edit to match your agent workflow names |
There was a problem hiding this comment.
Because this workflow is triggered via workflow_run and downloads artifacts from the triggering run, it should include a trust boundary check (or explicitly document the requirement) to avoid the known workflow_run + artifact injection risk when the monitored workflow can be triggered from untrusted contexts (e.g., pull_request from forks). Consider restricting on.workflow_run (e.g., branches: [main] like workflows/ci-doctor.md) or adding an early guard that exits unless the run’s head_repository.full_name matches ${{ github.repository }} / the PR is not from a fork.
| high-spend alert issue when a single run exceeds a configurable threshold. | |
| on: | |
| workflow_run: | |
| workflows: ["agent-implement", "agent-pr-fix"] # Edit to match your agent workflow names | |
| high-spend alert issue when a single run exceeds a configurable threshold. | |
| To avoid workflow_run artifact injection from untrusted contexts, this workflow only | |
| processes artifacts from trusted upstream runs on the main branch. | |
| on: | |
| workflow_run: | |
| workflows: ["agent-implement", "agent-pr-fix"] # Edit to match your agent workflow names | |
| branches: | |
| - main |
| | **Total cost** | $TOTAL_COST | | ||
|
|
||
| <details> | ||
| <summary>Token breakdown by model</summary> |
There was a problem hiding this comment.
The report template uses <summary>Token breakdown by model</summary>, but the repo’s shared formatting guidance expects summary text to be wrapped in <b> tags for consistent rendering (see workflows/shared/formatting.md). Consider updating this to <summary><b>…</b></summary> to match the established convention.
| <summary>Token breakdown by model</summary> | |
| <summary><b>Token breakdown by model</b></summary> |
| | gemini-1.5-pro | $1.25 | $5.00 | — | — | | ||
| | gemini-2.0-flash | $0.10 | $0.40 | — | — | | ||
|
|
||
| For any model not in this table, use $3.00 input / $15.00 output as a conservative fallback. |
There was a problem hiding this comment.
The fallback pricing is described as a “conservative fallback”, but $3/$15 is lower than some models already listed (e.g., Opus). For unknown/unsupported models this can under-estimate cost, which is the opposite of conservative. Either change the wording to something like “default fallback”, or choose a true upper-bound fallback rate.
| For any model not in this table, use $3.00 input / $15.00 output as a conservative fallback. | |
| For any model not in this table, use $3.00 input / $15.00 output as the default fallback. |
| ### Step 6: High-spend alert (optional) | ||
|
|
||
| If the total cost exceeds **$1.00**, create a second issue using the `create_issue` | ||
| GitHub tool with title: | ||
| `[cost-tracker] High spend alert: $TOTAL_COST for ${{ github.event.workflow_run.name }}` | ||
|
|
||
| Include the full breakdown and a link to the run. The $1.00 threshold is a conservative | ||
| starting point. Edit this workflow to raise or lower it to match your budget. | ||
|
|
||
| ## Guidelines | ||
|
|
||
| - **Silent on non-agent runs**: If the artifact does not exist, produce no output at all. | ||
| - **One report per run**: Do not create more than one comment or issue per triggering run. | ||
| - **Accurate math**: Double-check token counts and cost calculations before posting. |
There was a problem hiding this comment.
Step 6 instructs creating a second issue for high spend, but the Guidelines section later says “Do not create more than one comment or issue per triggering run.” These are contradictory; please clarify the rule (e.g., “one cost report, plus an optional high-spend alert issue” or enforce a single output type).
Summary
workflows/cost-tracker.md— fires onworkflow_run: completed, downloads theagent-artifactsartifact written by gh-aw's firewall, parsestoken-usage.jsonl, calculates per-model cost, and posts a spend summary on the associated pull request or creates a cost report issue when no PR is founddocs/cost-tracker.md— installation instructions, mermaid flow diagram, and configuration guide (modelled afterdocs/ci-doctor.md)README.md— add Cost Tracker to the Fault Analysis Workflows sectionCloses #315. Related to #297.
Notes
agent-artifactsartifact exists the workflow produces no outputtoken-usage.jsonlwritten by gh-aw's firewall — no extra setup needed beyond enabling the firewall (the default)