Skip to content

Artefact audit prompt

The evidence-first review of one deliverable against the code and the data. One reviewer, one artefact, one round; produces a single report in which every check cites its source. This is the generic spine; a per-deliverable verification playbook may add artefact-specific checks on top.

Fill <artifact> (e.g. paper_00, mini_course), <reviewer> (claude | ds | mimo), <round>.

AUDIT <artifact> — reviewer: <reviewer>, round: <round>

Source under review: docs/publications/<artifact>/
Read the source files (.qmd / .md), NOT the rendered PDF / .tex / _site —
those can be stale.
Write your report to: tmp/<reviewer>_<artifact>_round<round>_check.md

IRON RULE — evidence is mandatory:
Every "pass"/"fail" on a number, a formula, or a figure must cite a concrete
source: a data-file path + the quoted value, OR code at file:line, OR the
figure's mtime. It is forbidden to declare something "verified" by checking
only that a number is consistent with another sentence in the document. If
you cannot open the source, write "NOT VERIFIED (reason)", never "pass".

CHECKS (for each: open the source, quote the value, compare it to the text):

1. NUMBERS vs DATA. For every number/table in the source, open the canonical
   data file and compare. Distinguish canonical/frozen data from working
   data: if the text cites a value that is not in the canonical snapshot (or
   the snapshot predates a bugfix the text mentions), it is STALE — flag it.
   If a data directory is missing or git-ignored, say so.
2. DERIVED QUANTITIES (%, effect sizes, peaks, asymmetries, table sums):
   RECOMPUTE them from the source numbers/parameters, using the REAL defaults
   in the code (not the values baked into a standalone figure). Do not accept
   consistency with the prose as proof.
3. DEFINITIONS / FORMULAS vs CODE. Diff every formula and algorithm (metrics,
   agents, dynamics, perturbations) against the implementation — cite
   file:line.
4. FIGURES & STALENESS. Does each figure exist? Compare mtime(figure) against
   mtime(data/script that generates it): if the figure is older than the data
   or than a fix the text cites, it is stale — flag it. Spot-check: a value
   stated in a caption must exist in the data.
5. CROSS-REFS & TABLES. Every reference resolves to a definition; every table
   is referenced. List the broken ones (with file:line).
6. CITATIONS. Every citation resolves in the bibliography. For citations that
   support a CLAIM, verify the primary source exists and that the source text
   actually supports the claim — no claim without its source. Check
   year/version. Report entries cited but absent, and present but never cited.
7. ARGUMENT CHAINS & CLAIMS. Are the headline findings supported? Hunt for
   OVER-CLAIM: the abstract/intro asserting more than the body supports.
   Compare abstract vs body vs conclusions on the numbers.
8. NON-DETERMINISTIC (LLM) DATA. LLM pilots are one realisation: verify the
   aggregate on disk matches the text; claims must be directional, not point
   estimates; flag anything presented as robust that is noise at small N.

ROUND >= 2: first VERIFY that every fix from the previous round was actually
applied (quote file:line evidence; do not trust the prior report), then look
ONLY for new or residual findings. Actively hunt your OWN mistakes and the
over-cautious flags from the previous round.

ANTI-PATTERNS (do not):
- do not validate a number against another sentence — re-derive it from the
  source;
- do not read the rendered PDF/.tex for values (it can be stale): use the
  source + data;
- do not invent per-seed values or citations; if you do not have them, say so;
- distinguish "the document is internally consistent" from "the document
  matches the code and data".

OUTPUT:
- One severity per finding: blocker (factual error) / important / minor —
  each with file:line, quoted evidence, and a proposed fix.
- "Verification log": a table of what you checked against which source (with
  the values).
- "Not verified (and why)": everything you could not open or recompute.
- "Possible false positives of mine": items you are unsure about.
- "No new findings" is a VALID outcome — do not invent problems to fill the
  report.

Why these checks (observed failure modes)

The check list and the iron rule exist because reviewers — human and model — repeatedly failed in specific, predictable ways. See reviewer roles for the catalogue. The short version: reviewers mark things "verified" for internal coherence without opening the data, ratify a wrong fix because it reads consistently, quote stale rendered output, or invent values. Each check above closes one of those holes.