By ByteBot
An open-source Go binary called Memdoor just ran five mid-range Qwen3.8-27B instances as a coordinated team and outscored a single call to Claude Fable 5 on a hard coding benchmark: 9.8 versus 8.0 out of 10. The result has caveats — ten problems, possible training contamination, 40-minute runtimes — but the architecture lesson it demonstrates is immediate and cheap to replicate: for verifiable coding tasks, orchestration strategy now matters more than model size.
How the team-solve works
Memdoor implements what it calls “ledger discipline.” Five Qwen3.8-27B instances each start from a fresh context and pass information only through files on disk — no shared chat history, no message queues. The roles are fixed:
- Planner — generates the strategic approach
- Ideator A and Ideator B (run in parallel) — each proposes a different algorithm
- Solver — picks the strongest approach and implements it
- Manager — reviews test output
- Repair Worker — patches failures the manager flags
This stateless, file-based design sidesteps the coordination overhead that plagues most multi-agent systems. Agents do not need to agree on shared state or tolerate message loss — they just read and write files. It is a mundane implementation choice that turns out to matter a lot in practice.
What the benchmark actually proves
The headline number — 9.8 versus 8.0 on LiveCodeBench Hard — deserves scrutiny. The sample is ten problems. The benchmark problems date from early 2025, and the Qwen model may have seen them in training. Fable 5 was tested with a single call, not a comparable multi-pass workflow. The methodology has not been fully published.
What the result does credibly show is what a solid independent analysis called “five tries with a verifier beat one try.” Coverage — the probability of getting at least one correct solution — scales predictably with sample count on verifiable tasks. ICLR 2026 research confirmed this: DeepSeek-Coder improved from 15.9% to 56% on SWE-bench Lite when samples increased from one to 250. Memdoor runs that same logic with five specialized agents and self-correction loops instead of brute sampling.
This is a test-time compute story. It is not proof that Qwen3.8-27B is a better model than Fable 5.
The cost math
Memdoor’s 50-run benchmark cost $35.79 total — a median of $0.39 per hard problem. That is competitive with or cheaper than a single frontier-model call for a complex task, depending on your provider. Running locally via Ollama drops marginal cost to near zero; you need roughly 40 GB of VRAM to run five 27B instances concurrently.
The tradeoff is latency. A full team-solve takes 15 to 40 minutes. That rules out any real-time use case — IDE autocomplete, interactive debugging. The sweet spot is async workflows: PR reviews, automated test generation, security scans, release note drafts. Anything where waiting 20 minutes for a verified, corrected answer is better than waiting two seconds for a guess.
How to use it today
Memdoor is a single static binary with no Docker, no Python, and no sudo required:
curl -fsSL https://memdoor.ai/install.sh | bash
memdoor # first run walks through provider setupBuilt-in workflows cover the most common async CI/CD tasks:
- pr-review — reviews a pull request, verifies findings by running the code, outputs P0/P1/P2 severity findings
- security-review — scans for secrets, injection, path traversal, and missing authorization
- fix-tests — finds the failing test command, fixes code until it passes, leaves a report
- release-notes — turns commits since the previous tag into Added/Changed/Fixed entries
GitHub Actions integration is available via guregodevo/memdoor-action. Supported providers include OpenRouter, Anthropic, OpenAI, Gemini, DeepSeek, Groq, and any OpenAI-compatible endpoint, plus Ollama and llama.cpp for local runs.
The transferable lesson
The Memdoor result is a data point, not a mandate to swap out your frontier model subscription. But the pattern it demonstrates transfers to any team already running coding agents in CI/CD: add a verifier loop. The gains from multi-sample orchestration on verifiable tasks are large enough that the choice of model matters less than whether your workflow catches and retries failures automatically.
What this does not change: engineering judgment. Architecture decisions, security context, and domain expertise stay with humans. The Faros.ai 2026 developer survey found that experienced engineers using AI agents were 19% slower on complex open-ended tasks in their own codebases. Memdoor handles the mechanical parts of a code review before the engineer opens the diff — it does not replace the engineer opening the diff.
The repo is live, Apache 2.0, and seeing daily commits as of this week. If your team runs PR reviews or automated test fixes in CI/CD, it is worth a trial run.
