Reviewer roles and failure modes¶
The artefact audit is run independently by more than one reviewer, because a single pass normalises away its own blind spots. This page documents who runs the passes, what each brings, the failure modes each tends to exhibit, and the false-positive discipline that keeps the consolidation honest.
The passes¶
Each artefact is reviewed by several reviewers with different training distributions, each producing a separate report; the author then consolidates.
| Pass | Reviewer | Strengths brought to the table |
|---|---|---|
| 1 | Claude (working agent on the repo) | Familiar with the repo state, can cross-reference durable memory, fast iteration on edits |
| 2 | DeepSeek (independent, via the panel/brainstorm channel) | Independent reading, different distribution, strong at code-level scrutiny |
| 3 | MiMo (independent, third opinion) | Different prompt style, surfaces things the first two normalised away |
| escalation | Deep pass (the same Claude model at maximum/xhigh effort, used selectively) | Closes the hard, non-obvious findings a cheap pass cannot resolve — not a re-run of the audit |
The first three are the routine passes; the deep pass is invoked only on findings explicitly flagged as hard/doubtful, via a dedicated escalation file (see the verification README).
Failure modes to guard against¶
These are real patterns observed across review rounds. The audit prompt is shaped to block each one; reviewers should self-check against this list.
- Verifying for internal coherence. Marking a value "verified" because it is arithmetically consistent with another sentence, without opening the underlying data. → the iron rule + check 1.
- Ratifying a wrong fix because it reads well. Accepting a change because it is consistent with the surrounding prose, instead of recomputing the quantity from the real source. Reviewers have agreed with each other on an incorrect value this way. → check 2.
- Quoting stale rendered output. Reading a value off a PDF/figure that predates a fix. → check 4 + "read the source, not the rendering".
- Inventing values or citations. Filling a per-seed number or a reference from memory rather than from the artefact. → anti-patterns + the "possible false positives of mine" section.
- Green-checking the unchecked. Writing "pass" on something not actually opened, instead of "not verified". → the explicit "Not verified (and why)" section.
False-positive discipline¶
Independent reviewers also produce false positives — confident findings that are wrong. The consolidation step exists precisely to filter them, but the reviewer can help:
- End every report with "possible false positives of mine". Flag items you are unsure about so the consolidator weights them correctly.
- Keep a false-positive ledger. Findings that recur across rounds and have already been investigated and dismissed (a citation that is in fact correct, a "typo" that is intentional, a name that does exist) should be pre-listed so later rounds do not re-raise them. A recurring false positive costs as much to re-dismiss as a real finding costs to fix.
- "No new findings" is success, not failure. A reviewer who reports a clean pass with a full verification log has done the job. Manufacturing findings to look productive is itself a failure mode.
Consolidation¶
After all routine passes for an artefact are in, the author consolidates into one ledger: items all reviewers agreed on, items some agreed on, and items they disagreed on. Disputed and doubtful items either get resolved against the source or escalated. The surviving fixes are applied, and the next round opens from zero — see audit to convergence.