StructOutput-15

StructOutput-15 is an official BenchLocal Bench Pack for deterministic structured outputs. It measures whether a model can produce valid, constrained, machine-readable responses under practical schema and formatting requirements.

A Bench Pack is an installable benchmark package that runs inside the BenchLocal desktop app. BenchLocal provides the shared app experience for provider setup, model selection, verifier lifecycle, sampling controls, run histories, and side-by-side comparison across benchmark packs.

This repository contains the benchmark source: structured-output scenarios, scoring logic, methodology, a BenchLocal adapter, a CLI runner, and an optional verifier runtime.

Run With BenchLocal

1. Download BenchLocal from the latest BenchLocal release.
2. Open BenchLocal and install StructOutput-15 from the official Bench Pack registry.
3. Add one or more models, select StructOutput-15, and start a run.

Bench Pack Structure

lib/                    Benchmark core, scoring, and model runtime
benchlocal/             Thin BenchLocal SDK adapter
cli/                    Non-UI runner
verification/           Optional verifier runtime for exact validation
scripts/                Local helper scripts for verifier development
benchlocal.pack.json  Static install/discovery manifest
METHODOLOGY.md          Published benchmark methodology

BenchLocal Adapter

• benchlocal/index.ts is the only place that imports @benchlocal/sdk.
• lib/ stays framework-agnostic and is shared by the CLI and BenchLocal.
• benchlocal.pack.json is the canonical Bench Pack metadata manifest used for install, inspection, and runtime metadata.
• Verifier lifecycle belongs to BenchLocal; the pack only declares verification/ and its manifest metadata.

Development

• BenchLocal build: npm run build:benchlocal
• CLI runner: npm run cli
• Methodology: METHODOLOGY.md

Local Verifier

npm install
cp .env.example .env

Terminal 1:

npm run verify:sandbox:serve

Validation

npm run typecheck
npm run build:benchlocal
npm run verify:canonical