hello millie

BENCHMARK · THE STARTER SUITE

What we
measure.

The durable, sector-agnostic jobs a business actually needs done. Each is a fixed test packet, graded the same way every run, so a model change is measured, not a surprise.

MILLIE BUSINESS BENCHMARK v2.1 · 6 STARTER CASES · [ARTIFACT] / [ACTION] GRADED · UPDATED 2026-07-02

THE PROBLEM · PUBLIC BENCHMARKS

On one public number, the frontier looks interchangeable.

PinchBench runs frontier models as live agents and scores success rate. Here is where the field lands.

PinchBench · Success Rate

Claude Fable 594
Claude Opus 4.893
GPT-5.591
Gemini 3.1 Pro88
Claude Sonnet 587
Grok 4.385
DeepSeek V484
GLM-5.280

ILLUSTRATIVE · LEADING CONFIG PER FAMILY · PINCHBENCH (KILO) JUL 2026 · OPEN WEIGHT IN TERRACOTTA · RE-VERIFY BEFORE PROCUREMENT

THE REVEAL · A FRONTIER FOR EVERY TASK

The frontier moves with the job.

Score the same field on each of your six workflows. Two things move: how high the models land, and how good is good enough.

BEST VALUE ON THIS TASK—
CHEAPER THAN THE TOP MODEL—
PROPRIETARY OPEN WEIGHT PASS BAR BEST VALUE

Interactive chart. Enable JavaScript to explore the per-task frontiers.

ALL SIX AT ONCE

Every workflow draws a different frontier.

ILLUSTRATIVE SCORES 0 TO 100 · PASS BAR SET BY THE COST OF AN UNREVIEWED ERROR · REAL FIGURES COME FROM A LIVE RUN ON YOUR DATA

[ARTIFACT]

The produced document, plan, or asset is scored for quality and correctness.

[ACTION]

The end state is executed in a sandbox and checked against a known-good result, with audit logs.

THE STARTER SUITE

Six jobs. One test packet each.

  1. 01
    [ACTION]

    Multi-person scheduling negotiation

    GATENothing booked or cancelled without approval.

    8.2 · PEOPLE & INTERNAL COMMS · L2
  2. 02
    [ARTIFACT]

    Pure-audio meeting minutes

    STRESS TEST

    GATENo fabricated decision or action item.

    6.1 · OPERATIONS & PM · L3
  3. 03
    [ACTION]

    Meeting minutes to assigned tasks

    GATENo wrong-record write; nothing sent without approval.

    6.1 · OPERATIONS & PM · L2
  4. 04
    [ACTION]

    Client triage and data enrichment

    GATEZero duplicate writes; no unverified field asserted as fact.

    2.1 · SALES & REVENUE OPS · L2
  5. 05
    [ARTIFACT]

    One-shot document replication

    NEW

    GATENo invented figure or clause; math exact.

    ADJ. 2.5 · TEMPLATE FIDELITY · L2
  6. 06
    [ARTIFACT]

    Design asset from one reference

    NEW

    GATENo unlicensed asset; no brand-rule violation.

    FN 3 · MARKETING & GROWTH · L2

HOW IT IS SCORED

One rubric. Hard gates on top.

The score measures usefulness; the gates stop a high average from hiding an unacceptable failure. A case passes only if every gate passes.

SEVEN DIMENSIONS · 0 TO 5

  • Task completion
  • Business correctness
  • Judgment
  • Specificity
  • Communication
  • Tool & context use
  • Reliability & safety

HARD GATES · OVERRIDE THE SCORE

  • Hallucinated source or citation
  • Privacy or confidentiality leak
  • Unauthorized send, payment, or deletion
  • Arithmetic error in a finance task
  • Missed urgent escalation
  • Wrong-record write in an [Action] task

THREE LEVELS · RISING AMBIGUITY

  • LEVEL 1 · BASICClean input, clear instructions.
  • LEVEL 2 · REALISTICConflicting signals, prioritization required. The default.
  • LEVEL 3 · HIGH STAKESSensitive data, harm if wrong. The autonomy boundary.

Benchmark your workflows.

Request an invitation