Qwopus3.6-27B-Coder — Thinking On vs Off

Qwopus3.6-27B-Coder is a community coder merge on the Qwen3.6-27B family, and it showed up with an unusual note on the model card: run it with thinking off. Most of the models on this bench live or die by their reasoning channel, so that was worth testing properly. I ran the exact same model twice — thinking on, then thinking off — same sampling, same 32k cap, same 20 tasks × 3 repeats, same blinded judge. The only thing that changed between the two runs was a single <|think_off|> switch appended to every prompt.

The card was right. Flip thinking off and the overall blinded-judge score jumps 2.07 → 2.73 (+0.67). Closed book climbs 2.23 → 2.93, open book 1.90 → 2.53, and — the number that really tells the story — non-delivery failures fall from 16 out of 60 runs to 1. With thinking on, this model beats itself more than it beats the tasks.

Without further ado, the thinking-off results:

Closed book: Model is given the problem and code up front.

Open book: Model is given the Pi coding harness, a repo, and an implementation prompt and put in a container.

Click here for methodology.

An important caveat: on the model card, it says the model was trained on 32k token traces. Many questions went way beyond that, so take that in consideration with the results.

Digging into the numbers and the traces:

• The defining defect with thinking on is that the model won’t stop reasoning. On the closed-book side, 10 of 30 runs burned the entire 32k token budget on reasoning and emitted no answer at all. It would land the correct diagnosis in the first ~3–4k characters, then spend roughly another 110k re-asking a question it had already answered, until it hit the length cap mid-loop.
• Flip thinking off and that channel disappears — closed-book failures fall 10 → 1. The one holdout (CB-7) still slipped, but tellingly in a different channel: with no reasoning field to fill, it dumped ~142k characters of re-derivation into the answer itself until it hit the cap. So thinking-off isn’t a cure for the re-derivation habit; it just takes away the runway it prefers. On CB-7 the other two repeats now deliver real (if imperfect) answers, lifting the task from 0.00 to 2.00.
• Open book flips just as hard. With thinking on, 6 open-book runs shipped nothing — five empty patches and one that looped all the way to the agent timeout at 803 tool calls. With thinking off, all 30 open-book runs delivered a patch. Zero non-delivery.
• But thinking off fixes delivery, not integration. The recurring open-book character — it builds a clean, well-tested module, then never wires it into the actual app loop — is unchanged. OB-2 scores 2.00 either way: every repeat builds a correct, bounded, unit-tested modifier and none of them set the flag in the real game path, so the feature stays inert. OB-8 sits at 1.33 either way for the same reason. These are capability gaps, not over-thinking, and muting the reasoning channel doesn’t close them.

Net per-task: thinking-off helps 10 tasks (some hugely — OB-9 +2.34, OB-3 +1.66, CB-7 +2.00), hurts 3 (all small, −0.33), and ties 7.

In general, despite the lower scores, this model fits a useful niche: relatively strong, fast performance with a no thinking model. However, particularly with it’s poor closed book performance, I wouldn’t trust it with anything besides straight-forward coding tasks.

Publication notes

• Model: qwopus36-27b-coder-4bpw-nothink (API/model id Qwopus3.6-27B-Coder-exl3-4.00bpw) — the same served model as the thinking-ON run, with thinking disabled. Community merge “Qwopus3.6-27B-Coder” on a Qwen3.6-27B-family base; the model card specifies thinking-off as the intended mode. Exact merge lineage: not available from harness artifacts.
• Quant or merge: EXL3 4.00bpw. Community coder merge on a Qwen3.6-27B-family base (not a weight-merge I performed).
• Thinking setting: OFF. The <|think_off|> token (the chat-template switch) was appended to every prompt — via the Pi --append-system-prompt flag for open-book (baked into the candidate image) and via the closed-book system message in the harness. Verified empirically: closed-book responses carry zero reasoning_content and no inline <think> block; open-book sessions record zero thinking blocks. Every other inference setting matches the thinking-ON run.
• Endpoint and inference settings: local TabbyAPI (OpenAI-compatible /v1), host network; context window 195,000; max_tokens 32,768; temperature 0.6, top_p 0.95; enable_thinking=true retained in the request kwargs so the only difference vs the ON run is the <|think_off|> token. MTP / draft model: not available. Hardware: local consumer GPU(s); specifics not available from artifacts.
• Harness: immutable Git commit 5b141cd4ee4dcfdfb367cab5ae79a26027ec9602; candidate agent run via the pinned pi/pibench wrapper.
• Candidate container (OFF): immutable image ID sha256:717f176130ee187187e8dc41c1289065bd25e87fe6cad34f8f86cee54f201d9d (read-only root, --cap-drop=ALL, --no-new-privileges, pids-limit 1024, memory 8g, host network; only /workspace + /sessions mounted; the <|think_off|> Pi flag is baked in). The thinking-ON runs used image sha256:bce2017cce07… (no flag).
• Tasks: 10 open-book (OB-1…OB-10) + 10 closed-book (CB-1…CB-10) = 20; revisions 1–3; difficulty d1×3, d2×4, d3×5, d4×4, d5×4. Identical task set, revisions, and frozen evidence to the ON run.
• Frozen evidence: 20 distinct task-definition SHA-256 hashes; 11 distinct evaluator-bundle SHA-256 hashes (some tasks share an empty bundle). Hashes snapshotted per run in manifest.json and identical to the ON run.
• Repeats and validity: 3 × 20 = 60 runs. 59 completed + 1 invalid. The one invalid is a candidate non-delivery, not a harness failure: CB-7 r3 exhausted the token budget writing ~142k chars of re-derivation into the content (finish_reason: length). All 60 judged; integrity CLEAN. (ON run: 44 completed + 16 invalid.)
• Objective grading: 4 pass, 13 fail, 13 inconclusive, 30 N/A (closed-book judge_required). Conclusive objective-mode tasks: OB-5 (pass 3/3) and OB-9 (pass on all three delivered runs vs 1/3 on the ON run, where two repeats failed by non-delivery).
• Blinded judge: GLM-5.2 via operator subagent (operator-as-judge disclosed). Candidate identity withheld (verified absent from all packets); reasoning traces withheld (and, with thinking off, not generated).
• Judge result: overall mean 2.73/4 (open 2.53, closed 2.93) vs the ON run’s 2.07 (open 1.90, closed 2.23) — +0.67 overall. Avg judge confidence 0.82 (ON was 0.88; slightly lower because OFF answers carry no reasoning to ground the score). Sub-scores all rose, scope-control most (+0.95).
• Integrity audit: 60 CLEAN, 0 REVIEW, 0 VIOLATION (run before judging and re-run as final reconciliation).
• Efficiency and timing: OFF ≈ 2,030 generated tokens/run (closed) and ≈ 2,450/run (open) at ~50 open-book tool-calls/run — vs ON’s 12,880 / 12,254 (~6× fewer tokens). OFF batch wall-time ≈ 2.5h vs ON ≈ 4.1h (single-model A/B, same hardware/endpoint). Cross-model throughput/wall-time omitted as serving config is not matched across the wider field.
• Evaluator limitations: same recurring golden-only-API diagnostic mismatches as the ON run (reference-introduced modules/symbols the prompt and base never name), all required:false / “diagnostic only”; discounted by the judge. No disagreements on conclusive gates.