ReasonMath-15

ReasonMath-15 is an official BenchLocal Bench Pack for practical reasoning and math. It measures whether a model can solve grounded reasoning tasks, perform careful calculations, and avoid plausible but incorrect shortcuts.

A Bench Pack is an installable benchmark package that runs inside the BenchLocal desktop app. BenchLocal provides the shared app experience for provider setup, model selection, sampling controls, run histories, and side-by-side comparison across benchmark packs.

This repository contains the benchmark source: reasoning scenarios, scoring logic, methodology, a BenchLocal adapter, and a CLI runner for local development.

Run With BenchLocal

1. Download BenchLocal from the latest BenchLocal release.
2. Open BenchLocal and install ReasonMath-15 from the official Bench Pack registry.
3. Add one or more models, select ReasonMath-15, and start a run.

Bench Pack Structure

lib/                    Benchmark core, scoring, and model runtime
benchlocal/             Thin BenchLocal SDK adapter
cli/                    Non-UI runner
benchlocal.pack.json  Static install/discovery manifest
METHODOLOGY.md          Published benchmark methodology

BenchLocal Adapter

• benchlocal/index.ts is the only place that imports @benchlocal/sdk.
• lib/ stays framework-agnostic and is shared by the CLI and BenchLocal.
• benchlocal.pack.json is the canonical Bench Pack metadata manifest used for install, inspection, and runtime metadata.
• Per-pack default sampling belongs on the manifest, not on global host settings.

Development

• BenchLocal build: npm run build:benchlocal
• CLI runner: npm run cli
• Methodology: METHODOLOGY.md

Validation

npm run typecheck
npm run build:benchlocal