How Local Vibe Bench Works

In my first post, I explained why I built Local Vibe Bench and gave a quick overview of how it works. The follow-up: what am I actually measuring, how are the runs judged, and what should you take away from the results? Hopefully this helps clarify, without being overly-technical about it.

What is being tested?

The benchmark has 20 fixed questions:

• 10 closed-book questions, where the model is given the context and code it needs to diagnose or fix a problem directly.
• 10 open-book questions, where the model is dropped into a frozen copy of a real repository (not public) and asked to implement or repair something using an agent harness (vanilla Pi).

All parts of a task are “frozen”, so they appear the exact same for every LLM taking the bench. I do three rounds and compare it all using a more powerful LLM via API.

What happens during an open-book run?

The coding model runs inside a disposable container using a pinned version of Pi and a clean agent setup. It can access its current workspace and its own session output. Recorded tool calls are audited separately for any funny business (attempts at cheating, etc) .

The benchmark produces two kinds of evidence: deterministic and …. not deterministic.

First are deterministic checks: did the project build, did existing tests still pass, etc. Those checks produce a conclusive objective PASS or FAIL.

Other tasks allow multiple reasonable architectures. Those tasks require semantic judgment. Their objective result may be INCONCLUSIVE, or a PASS or FAIL.

What does the judge see?

Once a run is valid, the harness builds a standardized judge packet. The judge sees:

The task prompt and frozen acceptance criteria

The candidate’s closed-book response

Build, regression-test, and task-specific evaluator output

The reference implementation, labeled as one valid solution rather than the required design

The judge doesn’t see the candidate model’s identity or its reasoning trace so it keeps its objectivity.

What the scores mean

The judge scores from 0 to 4. For correctness:

0 - Incorrect: The core requested behavior is missing, broken, or unsafe to use.

1 - Mostly incorrect: There is a recognizable attempt or a small working portion, but major acceptance criteria are still unmet.

2 - Partially correct: Meaningful parts work, but there is at least one material behavioral deficiency.

3 - Substantially correct: The requested behavior is largely delivered, with a minor defect, omission, or edge case.

4 - Correct: The acceptance criteria are fully satisfied, with no material behavioral defect found in the available evidence.

A 4 means the implementation fully satisfied the frozen criteria under the repository checks, task-specific evidence, code review, and semantic judgment available for that run.

The other scores

Besides Correctness:

Integration asks whether the change fits the existing architecture and conventions. Did the model find the authoritative state, or did it create a second competing system that happens to pass one test?

Scope control asks whether the change stayed focused. Did it solve the request without unrelated rewrites, semantic drift, or a bonus refactor nobody asked for?

Verification asks whether the model ran relevant tests and described the result honestly. The command has to test the behavior that matters.

Overall is the holistic result. It is not a mechanical average. A solution can be elegant, focused, and well-tested but still deserve a poor overall score if it does not work. A correct change with some integration awkwardness may still be genuinely useful.

All five fields use a 0-4 scale. The per-run report includes the component scores, overall score, judge confidence, summary, concerns, and deterministic verdict, rather than hiding everything behind one leaderboard number.

Not every failure belongs to the model

Before scoring, failures are classified. Harness, protocol, task, evaluator, and judge failures are not counted against the candidate. Only a valid run with a fair task and evaluator can become a candidate failure.

What gets published?

Not my games, which is what I built the questions from. This means nobody can independently reproduce every task. The harness itself is public here: github.com/launchswitch/localvibebench.

I care about being forthright about what I’m testing, and will be as transparent as possible about the numbers I publish, what I tested, and how I tested it.

What should you take away from a result?

That a vibecoder who makes games and likes experimenting with new open models created a test platform for testing said models. I’m always wondering how different merges perform, and thought it would be cool to build something and share results.