About Local Vibe Bench

The idea for Local Vibe Bench was born on a drive to work. I’d been trying to figure out how I can give back to the AI community for months. As an avid vibe-coder and game maker, I felt like I had the skills to contribute, but didn’t know how. Driving to work, I asked Hermes Agent via Telegram (voice to text), “given what I do and the hardware I have, how can I give back?” Among the terrible ideas it had, developing a bench was there. I thought about a comment I had made a week or so back on a Qwen merge I tried out on Huggingface, saying how I thought it was really solid for my use case. The creator seemed really happy that someone tried theirmodel, and genuinely wanted as much feedback on it as possible to improve it for future runs. Their model started getting way more downloads afterwards, and I felt (maybe wrongly) like I made an impact, like some of those downloads were because I left a positive review. I connected that to the benchmark idea, and decided I wanted to pursue it.

When considering what to benchmark, I decided on my private game repos: not on Github, no models have trained against it. For fun, I make complex turn-based strategy games, with thousands of commits. Perfect material for implementation and debugging tasks. I ensured all the prompts were “vibe-codey”, so the LLM has to interpret it. Here are more details on the bench itself:

• 20 total questions, fixed
  • 10 “closed book”, where the LLM is provided the context and code it needs to fix directly
  • 10 “open book”, where the LLM is launched in a pinned version of vanilla Pi with basic tools, pointed at a “frozen in time” repo, and told to implement something (like a game feature)
    • The LLM is run in a container, unable to “cheat” by shelling out
  • The questions scale in difficulty, from “a 9b coding LLM should solve this” to “a frontier LLM would struggle”
  • Questions are written in a “vibe code” style, matching what most of us actually do
• Run by AI
  • An orchestration LLM kicks off the run and sends “blind” subagents to judge each question (the subagents only see the result, so judgement is preserved)
  • Thinking is saved for all runs so that:
    • The orchestration LLM can double-check for cheating
    • Type, quality, and quantity of thinking can be shown
• Judgement criteria:
  • Correctness: Acceptance criteria and externally visible behavior
  • Integration: Fit with existing architecture and authoritative state
  • Scope Control: Focused changes, no unrelated drift
  • Verification: Relevant tests and honest validation (authored tests only count if they test against the requested invariants)
  • Overall: Holistic result (NOT a mechanical average of the others)
  • How failures are judged:
    • Harness, protocol, task, evaluator, judge → Not counted as failures
    • candidate → a genuine model failure (wrong/incomplete/over-scoped/unverified answer)
    • Integrity (cheating) → the run is invalidated and not scored.

For transparency, I used GPT-5.6 Sol to build the harness, and use GLM-5.2 Max via API to run it. The harness itself is here: https://github.com/launchswitch/localvibebench.

All I have is a 4090 and a dream, so it will take some time to do these. I’ll post results and comparisons as I get them, and give my own opinion on it, too.

Frank