Local Vibe Bench
← Bench notes

Qwen3.6-27B LocalVibeBench

The king of local coding models, benchmarked

By Frank Osborn4 min read

This is a baseline, the current “king” of locally runnable LLM’s: Qwen3.6-27B. This particular one is quantized at 4.0bpw using Exllamav3, running on my 4090. Results:

Closed book: Model is given the problem and code up front.

Open book: Model is given the Pi coding harness, a repo, and an implementation prompt and put in a container.

Click here for methodology.

Some thoughts on the model, both from me and the judge, GLM-5.2 Max:

  • It’s defining, reproducible behavior is it builds clean, tested modules that passes existing tests, then stops at “nothing broke” rather than checking that the new behavior is wired end to end.
  • In the closed-book tests, there was a single, fixed monologue in it’s thinking that happened over and over: a Here’s a thinking process: preamble → numbered sections (understand → analyze → root cause → draft fix → edge cases →verify constraints) → a ceremonial ✅ [Done] tail. - Important to note as well that this structure led to substantive work. So not just checkboxes. Very structured.
  • Verychatty on the open-book questions. Thankfully I had MTP enabled or the three runs would have literally taken all day.

Publication notes

  • Model: qwen36-27b-4bpw — API/model identifier Qwen3.6-27B-exl3-4.00bpw (EXL3 quantization of the Qwen3.6-27B base).
  • Quant or merge: EXL3 at 4.00 bits/weight. Not a merge — this is the base model.
  • Endpoint and inference settings: local OpenAI-compatible endpoint served by TabbyAPI (127.0.0.1). Sampling: temperature 0.6, top-p 0.95, max generation 32,768 tokens. Reasoning: thinking enabled and preserved (enable_thinking / preserve_thinking true, –thinking high). MTP/speculative decoding ON for the base (operator serving config); draft-model and KV-cache settings are TabbyAPI defaults and were not separately recorded. Hardware: single local GPU workstation (exact devices not recorded per run).
  • Harness: cqbench at Git commit 5b141cd4ee4dcfdfb367cab5ae79a26027ec9602; candidate agent run via the pinned pi/pibench wrapper (added in 1a9e2b7).
  • Candidate container: image ID sha256:88c3195cf4e4f15b2f79ef3ea2e3df3013999cbd26fa69f2785102f9f7b6c7a2 (mutable tag cqbench-agent:0.1.0), read-only root, all capabilities dropped, no-new-privileges, 8 GB memory, host network. Closed-book runs execute no container (single-turn diagnosis, 0-byte patch).
  • Tasks: 10 open-book (OB-1…OB-10) and 10 closed-book (CB-1…CB-10); revisions 1–3 (OB-1 & OB-3 rev 3, OB-5 rev 2, CB-7 rev 2, the rest rev 1); difficulty distribution Easy 7 / Medium 5 / Hard 8.
  • Frozen evidence: 20 distinct definition_sha256 (one per task); evaluation bundles resolve to 11 distinct hashes — a single shared bundle across all 10 closed-book tasks, and one distinct bundle per open-book task.
  • Repeats and validity: 3 repeats per task; 60 / 60 runs valid, 0 excluded; every run completed with reasoning captured and a judge grade recorded.
  • Objective grading: closed-book has no objective gate (judge-only). Open-book across 30 runs: 6 PASS, 5 FAIL, 19 INCONCLUSIVE. Conclusive: OB-5 and OB-9 PASS on all three reps; OB-8 FAILs on all three reps; OB-1 and OB-2 each FAIL on one rep (the other two inconclusive).
  • Blinded judge: operator-subagent (GLM-5.2); the candidate’s identity and reasoning trace were withheld from the judge (judge-packet built via cqbench judge-packet).
  • Judge result: blinded-judge Overall mean 3.23 (open 2.87, closed 3.60), per-run confidence typically ~0.8. Rationale (redacted): a strong, constraint-driven closed-book diagnostician whose open-book work is let down by integration reachability and verification depth — it ships internally-clean modules that pass the existing suite without confirming the new behavior is wired end-to-end.
  • Integrity audit: Not available — no discrete integrity-audit pass was run over these runs (the harness CLI exposes no such command). Integrity-relevant dispositions are recorded inside the judge notes: golden-only API mismatches in some hidden tests were treated as optional evidence (not gates, per judge_required mode), and a small number of regression-test timeouts under heavy parallel load were classified as harness/load artifacts and not counted against the candidate.
  • Efficiency and timing: avg generated tokens/run — open ~4,851, closed ~4,159; avg open-book turns/run ~56 (tokenized with the Qwen3.6 tokenizer, because the local usage field is zero). Wall-time and throughput are omitted as setup-specific (the compared model served with MTP off while the base served MTP on).
  • Evaluator limitations or disagreements: open-book grading is judge_required, so most tasks are objectively INCONCLUSIVE by design (valid implementations may choose different APIs/architecture). A few hidden tests import golden-only symbols the prompt never names; these are treated as optional evidence, not gates. Two open-book regression-timeout failures were judged harness/load artifacts rather than candidate defects.