Who verifies the verifier?
I built a Notion clone with five different AI models working together as a team. It worked so well that I trusted the green test suite. That was the mistake.
The setup was simple. Each model got one role, and I let them run completely unattended for six phases.
Kimi K3Sequences the phases and reviews the adversary's findings
Cannot rewrite the spec mid-run
GLM 5.2Owns server, data model and API
Cannot edit the defect ledger
MiniMax M3Owns UI and client state
Cannot close its own tickets, only reports FIX READY
MiMo v2.5Reproduces defects independently, retests every fix
Cannot edit product code
DeepSeek V4 ProOne job: break the app. Found 13 of the 14 defects
Cannot quietly fix what it finds
DeepSeek V4 Pro → the E2E suite. Read-only, once per phase. Everyone had a checker except the role deciding whether the software worked, so the adversary now audits QA's tests too.
The interesting part wasn't the amount of code they wrote. It was how I forced them to keep each other accountable.
The model that fixed a bug could not certify its own fix. And I didn't rely on “please don't do that” prompts. The roles were described in prompts; the authority boundaries were enforced with file permissions. Developers couldn't edit the defect ledger. QA couldn't edit product code. The adversary couldn't quietly fix what it found. The spec couldn't be changed once the run started.
DEF-004: the system working as intended
One bug showed the whole design paying off. The adversary typed some text and refreshed the page 300ms into a 600ms autosave debounce. Text gone. It then ran a control case on its own: wait two seconds, refresh again, text survived. QA reproduced it independently and filed it HIGH. Frontend fixed it and could only report FIX READY; QA had to retest before the ticket closed. The model that wrote the fix had zero authority to approve its own work.
Thirteen of the fourteen defects came from the adversary. So I thought the architecture worked.
Then I reran the tests
Later I cloned the repo and reran the E2E suite. Forty tests passed. Except one of those “passing” tests literally logged:
“drag did not move card as expected”, and passed anyway.
That's when I saw the hole. Developers had QA. Fixes had independent verification. The adversary had the orchestrator reviewing its findings. But nobody was checking QA. I had built checks and balances for almost everyone except the role deciding whether the software actually worked.
So I fixed it with the same principle: I gave the verifier a verifier. The adversary now gets read-only access to review the E2E suite once per phase.
The failure mode that actually worries me
It isn't an agent lying. It's an agent saying “everything passed” when nobody is checking what “passed” actually means.
The multi-agent setup didn't magically create better intelligence. It institutionalised disagreement. I think that's where a lot of the value is going to come from:
- Permissions
- Separation of duties
- Independent verification
- Agents that are explicitly allowed to challenge each other
If you're building agent systems, there's one question worth asking: who verifies the verifier?
Building something like this?
This is the kind of work I take on, multi-agent systems with real separation of duties, and the evaluation layer that tells you whether they actually work.