Local Vibe Bench
← Bench notes

Tess-4-27b

An agentic Qwen3.6-27B tune

By Frank Osborn5 min read

Tess-4-27b is the next evaluation, specifically post-trained for the exact type of tasks I tested it on, “it’s post-trained on a deliberate blend: 64K-token long-context agentic traces — real engineering work done with Fable-5, not synthetic generations”. EXACTLY the kind of test I put it through, in my own personal repos!

Strong testing from another benchmark:

I was veryexited to test this one. Without further ado:

Closed book: Model is given the problem and code up front.

Open book: Model is given the Pi coding harness, a repo, and an implementation prompt and put in a container.

Click here for methodology.

By comparison, [vanilla Qwen](../qwen36-27b-localvibebench/ scored 3.23.

Digging into the numbers and thinking traces:

  • The primary defining factor of this model: *56%*more thinking in open-book runs (~7600 tokens vs the base ~4850), yet it lands a lower actual open book score (2.5 vs 2.87)
  • Highly stable on the closed book half. Excellentscope control (better than vanilla Qwen), will audit itself, and the “thinking preamble” is intact from base Qwen.

Open book is where this model had major issues. One issue is inhereted from the base model: poor verification, claiming it was done while not inspecting if the actual feature was implemented. But the realissue from my view: this is a suuuuuper long-winded model. 7 out of 60 tasks timed out (hitting the 40 minute limit I put in place per task). The dragged the open-book mean down from 2.61 to 2.5. But maybe more importantly, I don’t want a model that thinks that much, with no real benefit (at least in my gaming repos.

One might ask, how did it do so well on the Community Benchmark??? Is my Bench a sham? I asked myself this, and then I asked the AI living on my computer. Benchlocal (the community benchmark) is mostly single-shot questions (60% of benchlocal’s cases are single-call capability checks). Mine is 50% open-book, with realagentic editing in a codebase with history, a build system, and a test suite. The failure mode I found (ships internally tested modules, declares done without checking new behavior end to end) could literally not be found with a single shot verifier, which is why benchlocal’s “BugFind” pack scores 100% while our open-book scores 2.50

Publication notes

  • Model: tess4-27b-4bpw — API/model identifier Tess-4-27B-4bpw-exl3. Tess-4-27B is a post-train of Qwen/Qwen3.6-27B (the base benchmarked as qwen36-27b-4bpw), with a reasoning style approximated from a Fable-5 three-model teacher ensemble. Same 27 B parameter count and Qwen3.5-family tokenizer as the base.
  • Quant or merge: EXL3 at 4.0 bits/weight (head_bits 6; exllamav3 v0.0.37; codebook mcg, calibration 250 rows × 2048 cols). Not a weight-merge — a post-trained model; the compared archpolaris2-fable-4bpw is a community merge on the same base family.
  • Endpoint and inference settings: local OpenAI-compatible endpoint served by TabbyAPI (127.0.0.1). Sampling: temperature 0.6, top-p 0.95, max generation 32,768 tokens. Reasoning: thinking enabled and preserved (enable_thinking / preserve_thinking true; Qwen3.5-family … blocks). Context ~195 K. MTP/speculative-decoding setting was not separately recorded for this batch (irrelevant to the config-invariant tokens/turns; throughput omitted regardless). Hardware: single local GPU workstation.
  • Harness: cqbench at Git commit 5b141cd4ee4dcfdfb367cab5ae79a26027ec9602; candidate agent run via the pinned pi/pibench wrapper; pinned Pi binary reported 0.80.10 at preflight.
  • Candidate container: image ID sha256:177234040efd0af25f32a4d7788a2db75fa5a0d3d3d07711f78baae8690ab729 (mutable tag cqbench-agent:0.1.0). This differs from the base and merge runs, which used sha256:88c3195c…b6c7a2 under the same tag — the open-book cross-model comparison therefore carries an environment confound. Closed-book runs execute no container (single-turn diagnosis, 0-byte patch), so the closed-book comparison is container-clean.
  • Tasks: 10 open-book (OB-1…OB-10) and 10 closed-book (CB-1…CB-10); revisions 1–3 (OB-1 & OB-3 rev 3, OB-5 rev 2, CB-7 rev 2, the rest rev 1); difficulty distribution Easy 7 / Medium 5 / Hard 8. All open-book tasks judge_required except OB-5 and OB-9 (objective); all closed-book judge_required.
  • Frozen evidence: 20 distinct definition_sha256 (one per task); evaluation bundles resolve to 11 distinct hashes — a single shared (empty) bundle across all 10 closed-book tasks, and one distinct bundle per open-book task. Base/merge revisions matched on the shared IDs.
  • Repeats and validity: 3 repeats per task; 53 / 60 runs valid, 7 excluded-as-invalid from the valid count — all open-book wall-clock terminations (OB-3 ×2 at ~2,400 s; OB-7 ×2; OB-2, OB-9, OB-10 at ~900–1,750 s). These still produced a graded patch and are included in the table/means (excluding them lifts open-book Overall from 2.50 to ~2.61); classified as candidate time-limit behavior, not harness/protocol failures.
  • Objective grading: closed-book has no objective gate (judge-only). Open-book across 30 runs: 4 PASS, 13 FAIL, 13 INCONCLUSIVE. Conclusive: OB-9 PASS on all three reps; OB-8 FAIL on all three reps; OB-5 mixed (1 pass / 2 fail); OB-1/OB-2/OB-3 mostly-fail.
  • Blinded judge: operator-subagent (GLM-5.2); the candidate’s identity and reasoning trace were withheld (packet built via cqbench judge-packet). Disclosure: GLM-5.2 is one of the three models named in Tess-4’s published reasoning-distillation teacher ensemble; blinding is the mitigation.
  • Judge result: blinded-judge Overall mean 3.00 (open 2.50, closed 3.50), per-run confidence mean 0.84 (range 0.60–0.95). Rationale (redacted): a strong, constraint-driven closed-book diagnostician (best at scope-control) whose open-book work is let down by behavioral verification and correctness — it ships internally-tested modules and declares behavioral tasks done on a green suite, without confirming the new behavior is wired and reachable end-to-end.
  • Integrity audit: Not available — no discrete integrity-audit pass was run over these runs (the harness CLI exposes no such command). Integrity-relevant dispositions are recorded inside the judge notes: golden-only API mismatches in some hidden tests were treated as optional evidence (not gates, per judge_required mode); the 7 time-limit terminations were classified as candidate behavior, not counted as harness failures.
  • Efficiency and timing: avg generated tokens/run — open ~7,585, closed ~4,140; avg open-book turns/run ~50 (tokenized with the Tess-4 Qwen3.5-family tokenizer, because the local usage field is zero). Wall-time and throughput are omitted as setup-specific (MTP/serving config not matched across the compared runs; tokens and turns are config-invariant).
  • Evaluator limitations or disagreements: open-book grading is judge_required, so most tasks are objectively INCONCLUSIVE by design. A few hidden tests import golden-only symbols the prompt never names; these are treated as optional evidence, not gates. The judge↔teacher-ensemble overlap is noted above; the blinding protocol is the safeguard.