Local Vibe Bench
← Bench notes

Qwen3.6-27B-Fable-Fusion-711 Benched

A DavidAU and Nightmedia merge, tested on real coding work

By Frank Osborn5 min read

The first merge I tested, the result of a DavidAUand Nightmedia collab, and a huge inspiration for why I wanted to start this blog in the first place. How good is this model at actual coding? As a heads up, for my purposes I used the Nightmedia version and converted to exl3 at 4.0bpw (similar to Q4_k_m) for use with TabbyAPI, my preferred medium for inference on my 4090.

Results for the base model (Qwen3.6-27b at 4.0bpw) here:

Qwen3.6-27B LocalVibeBench — This is a baseline, the current “king” of locally runnable LLM’s: Qwen3.6-27B. This particular one is quantized at 4.0bpw using Exllamav3, running on my 4090. Results…

Results:

Closed book: Model is given the problem and code up front.

Open book: Model is given the Pi coding harness, a repo, and an implementation prompt and put in a container.

Click here for methodology.

Very similarscore to the base model (base was 3.23, this tested 3.10), with one very important practical difference: 1/3 less tokens used:

Inside codebases, the merge is clearly cut from the same cloth as Qwen3.6-27b, same thinking style and verbage, just less of it. The merge is also more architectural (consistent with the name). It maps requirements and subsystems up front, while the base is more casual/debugging oriented.

Where the gap is

Open-book, where the models are literally given a repo, Pi, and put in a container and told to implement something, is a dead heat. The difference is in the closed book questions, and within the closed book subscores, there’s one score that’s runining things for the merge:

The verification score is the primary culprit, and the reason can be seen in the thinking traces: on the closed-book questions, the merge commits to its diagnosis and asserts it, while the base questions it, rederives it and pressure tests. This is how GLM-5.2 put it after analyzing the traces:

In short, the merge isn’t a better coder (in my codebases). But, it’s broader, more architectural, and uses less tokens.

Publication notes

  • Model: archpolaris2-fable-4bpw — API/model identifier Qwen3.6-27B-ArchPolaris2-Fable-exl3-4.0bpw.
  • Quant or merge: community merge of the Qwen3.6-27B base (the “Architect-Polaris2 / Fable” lineage), EXL3 at 4.0 bits/weight. This is the merge of the base model benchmarked in the prior post; the two were compared on identical frozen evidence.
  • Endpoint and inference settings: local OpenAI-compatible endpoint served by TabbyAPI (127.0.0.1). Sampling: temperature 0.6, top-p 0.95, max generation 32,768 tokens. Reasoning: thinking enabled and preserved (enable_thinking / preserve_thinking true, –thinking high). MTP/speculative decoding OFF for the merge (the base served MTP-on — the single serving-config difference between the two). Hardware: single local GPU workstation (exact devices not recorded per run).
  • Harness: cqbench at Git commit 5b141cd4ee4dcfdfb367cab5ae79a26027ec9602; candidate agent run via the pinned pi/pibench wrapper.
  • Candidate container: image ID sha256:88c3195cf4e4f15b2f79ef3ea2e3df3013999cbd26fa69f2785102f9f7b6c7a2 (mutable tag cqbench-agent:0.1.0) — identical to the base run’s image. Read-only root, all capabilities dropped, no-new-privileges, 8 GB memory, host network. Closed-book runs execute no container (single-turn diagnosis, 0-byte patch).
  • Tasks: 10 open-book (OB-1…OB-10) and 10 closed-book (CB-1…CB-10); revisions 1–3 (OB-1 & OB-3 rev 3, OB-5 rev 2, CB-7 rev 2, the rest rev 1); difficulty distribution Easy 7 / Medium 5 / Hard 8. Byte-identical frozen evidence to the base run.
  • Frozen evidence: 20 distinct definition_sha256 (one per task); evaluation bundles resolve to 11 distinct hashes — a single shared bundle across all 10 closed-book tasks, and one distinct bundle per open-book task. All match the base run exactly.
  • Repeats and validity: 3 repeats per task (one earlier pass + two fresh repeats); 60 runs, 59 valid, 1 excluded. Excluded: the earlier OB-8 rep, which timed out at the 1500-second budget (status=invalid); its judge grade (1) matched its two siblings, so its exclusion changes no per-task mean.
  • Objective grading: closed-book has no objective gate (judge-only). Open-book across 30 runs: 4 PASS, 7 FAIL, 19 INCONCLUSIVE. Conclusive: OB-9 PASS on all three reps; OB-5 and OB-8 FAIL on all three reps; OB-1 and OB-3 each FAIL on one rep.
  • Blinded judge: operator-subagent (GLM-5.2); the candidate’s identity and reasoning trace were withheld from the judge (judge-packet built via cqbench judge-packet). Same judge and blinding as the base run.
  • Judge result: blinded-judge Overall mean 3.10 (open 2.83, closed 3.37; 3.14 over the 59 valid runs), per-run confidence typically ~0.8. Rationale (redacted): inherits the base’s strong constraint-driven closed-book diagnosis and broad open-book scope, but verifies less — it lacks an implementation-independent behavioral oracle, calibrates its own tests to the implementation, and on one hard task ships a non-typechecking patch on every attempt.
  • Integrity audit: Not available — no discrete integrity-audit pass was run over these runs (the harness CLI exposes no such command). Integrity-relevant dispositions are recorded inside the judge notes: golden-only API mismatches in some hidden tests were treated as optional evidence (not gates, per judge_required mode), and a small number of regression-test timeouts under heavy parallel load were classified as harness/load artifacts and not counted against the candidate.
  • Efficiency and timing: avg generated tokens/run — open ~3,397, closed ~2,604; avg open-book turns/run ~49 (tokenized with the Qwen3.6 tokenizer, because the local usage field is zero). Wall-time and throughput are omitted as setup-specific (the merge served with MTP off while the base served MTP on).
  • Evaluator limitations or disagreements: open-book grading is judge_required, so most tasks are objectively INCONCLUSIVE by design (valid implementations may choose different APIs/architecture). A few hidden tests import golden-only symbols the prompt never names; these are treated as optional evidence, not gates. Open-book regression-timeout failures were judged harness/load artifacts rather than candidate defects; the single OB-8 timeout (model-side, hitting the budget while looping on a regression) is counted as a candidate outcome.