Skip to content

Verification plan — BDPD deliverables

This directory holds independent-reviewer playbooks for every BDPD deliverable. Each playbook is a self-contained brief: a fresh reviewer (human or LLM) can pick one up, read only that file plus the pointed-to artefacts, and produce a usable report without needing the rest of the project context.

The plans are written so that:

  • We can re-run a verification ourselves in three months and remember exactly what to check.
  • We can subcontract a piece (e.g. "please verify paper_03") by handing one file to an external reviewer; the file lists every input path, every claim, every acceptance criterion.
  • We can build on someone else's verification — each playbook closes by describing what report format the reviewer should hand back.

What lives here

File Artefact under review
verify_software.md Platform code, regression tests, reproducibility (scripts/, platform/, agents/, engine/, experiments/, tests/)
verify_paper_00.md BDPD⁰ — The foundational cliff
verify_paper_01.md BDPD¹ — Governance: cheap talk and sanctioning
verify_paper_02.md BDPD² — Polycentric governance, four vignettes
verify_paper_03.md BDPD³ — Seneca: leading vs lagging signals
verify_mini_course.md The ten-lecture course The Limits of Good Will
verify_blog_book.md BDPD Research Notes — six collected blog dispatches

How each playbook is structured

Every file follows the same five-section pattern, so a reviewer who has read one can pick up another in two minutes:

  1. Purpose — one paragraph stating what claim the verification is testing and what a "pass" means.
  2. Where to look — exhaustive list of file paths, grouped by category: sources, compiled output, supporting material (library PDFs / pilot data / code).
  3. Procedure — five-to-eight numbered steps, executable without further clarification.
  4. Acceptance criteria — checklist for pass / fail / partial.
  5. Common pitfalls — known gotchas the reviewer should not waste time on.

Each playbook closes with a copy-pasteable prompt that wraps the whole brief into a single message suitable for handing to a fresh LLM or external reviewer.

Conventions used throughout

Convention Meaning
migration/docs/library/<bibkey>.pdf Primary source PDFs. Always check the PDF exists before citing.
migration/docs/library/txts/<bibkey>.txt Extracted plain text — useful for grep/search but PDF is the source of truth.
migration/data/pilot/<scenario>/ Pilot run outputs (json, logs, plots) — the empirical record the papers draw from.
migration/docs/publications/<artifact>/_site/ Rendered output (HTML site + PDF) — what the reader actually receives.
tmp/verification/<artifact>_report.md Where the reviewer's report is expected to land.
Path roots All paths in these playbooks are relative to the repo working tree (bdpd/migration/). External reviewers should cd there before starting.

Suggested order for a full review pass

If the goal is to verify the whole project in a single sweep, work in this order — the dependencies flow downstream:

  1. verify_software.md first. If the code does not reproduce numerically, every paper claim downstream is suspect.
  2. verify_paper_00.md, then paper_01, paper_02, paper_03. The papers build on each other: BDPD¹ assumes BDPD⁰ holds, etc.
  3. verify_mini_course.md. Most lecture claims are drawn from the four papers; if those passed, the lecture's empirical claims are inherited and only the pedagogical structure + external-citation backing need fresh checking.
  4. verify_blog_book.md. Same logic — by the time the blog notes are checked, the empirical record is settled; the verification is about alignment and overclaim detection.

Each playbook is also valid stand-alone — the dependency order is a convenience, not a hard requirement.

Modes of operation

There are two modes in which these playbooks can be applied. Pick the mode at the top of every reviewer report.

Mode A — Working repository (the project author + collaborators)

This is the mode used for the initial three-pass verification on this very repo.

In Mode A the reviewer is working inside the live project tree, with all dependencies already installed and all baselines already cached. The reviewer should therefore:

  • Verify by reading, not by executing. Confirm that scripts exist, that they document the expected behaviour, and that the baselines and pilot data they would produce are on disk. Do not re-run setup.sh, do not re-execute long regression sweeps, do not re-render every PDF from scratch.
  • Spot-execute selectively — at most one figure reproduce per paper playbook, at most one smoke test in the software playbook — to confirm the recipe is current.
  • Trust the pre-render output. The _site/ directories already contain the HTML + PDF artefacts the playbooks ask about. Render only if the playbook step is checking render hygiene and the existing output is stale.
  • Skip the "clean clone" step in verify_software.md entirely. It belongs to Mode B.

Mode B — External reviewer on a clean clone

This is the mode for an external reviewer or LLM who has never seen the project before, working from a fresh git clone at the tag under review (currently v1.1.2; round 2 also targets the working tree on the round-2 working branch).

In Mode B the playbooks are followed as written: install, smoke tests, regression sweeps, figure reproduce, the full pass. This is the mode the reviewer prompts at the bottom of each playbook target.

Which mode you are in is determined by who you are, not by which playbook you run. Mode A and Mode B produce different reports — Mode A is concentrated on content (claims, citations, narrative drift); Mode B adds machinery (install, regression, render-from-scratch).

Round 2 — Complete re-verification with five audit dimensions

Round 2 opened 2026-06-16 on branch papers-rework-v1.2-dev (from d0a26ff on main). Round 1 closed 2026-06-11 at commit 5fa0eaf.

Round 2 is a complete re-verification from zero, not a delta of round 1. Round-1 reports (tmp/verification/<reviewer>_<artefact>_report.md) are inputs only and have no privileged status; round-2 reports land at tmp/verification_round2/<reviewer>_<artefact>_report.md (do not overwrite).

Each playbook augments the core procedure with audit dimensions that round 1 did not require:

  • Paper playbooks (paper_00–03): five dimensions — Numeric / Narrative-flow / Balance / Harmonization / Epistemic-honesty.
  • Software playbook: two dimensions — Numeric (every doc-quoted value traced) / Documentation health (absorbs narrative + balance + harmonization + epistemic-honesty applied to the docs surface).

Each paper playbook ends with a Quick reference table listing the load-bearing numbers with their canonical anchors. Reviewers must consult this table before re-deriving anything from pilot data — round 1's DS pitch_01 audit produced 6/7 false positives by skipping this step (searching only raw pilot dirs while the canonical anchor lived in the paper qmd).

The full scope of round 2 is 5 artefacts (software + paper_00 + paper_01 + paper_02 + paper_03) × 3 reviewers = 15 reports + 5 consolidated + 1 escalation file. mini_course, blog_book, pitch_01, overview_00 are not in round-2 scope; their round-1 verifications stand.

Three-pass internal verification (Mode A)

The plan for the v1.1 verification round on this repo uses three independent passes, each by a different reviewer, each producing a separate report:

Pass Reviewer Strengths brought to the table
1 Claude (working agent on this repo) Familiar with the repo state, can cross-reference memory, fast iteration on edits
2 DeepSeek (via OpenCode + the brainstorm channel) Independent reading, different training distribution, good at code-level scrutiny
3 MiMo (via OpenCode + the brainstorm channel) Third opinion, different prompt style, surfaces things the first two normalised away

Each pass produces its own report at tmp/verification/<reviewer>_<artefact>_report.md (round 1) or tmp/verification_round2/<reviewer>_<artefact>_report.md (round 2). Examples: claude_paper_03_report.md, ds_paper_03_report.md, mimo_paper_03_report.md.

After all three passes for a given artefact are complete, the project author consolidates them into a single <artefact>_consolidated.md (in the same tmp/verification[_round2]/ directory) with: items where all three agreed, items where two of three agreed, items where reviewers disagreed.

Escalation: a deep pass

Any finding that the reviewer flags as hard / doubtful / non-obvious — something the reviewer is unsure about and that a fast pass cannot resolve — should be added to a dedicated escalation file:

tmp/verification/escalation.md

This file is consumed by a deep pass: the same Claude model run at maximum (xhigh) effort — deeper-reading, more tokens, used selectively. (On this project the deep pass was Opus 4.8 at xhigh effort.) The deep pass's job is not to redo the verification: it is to handle the items the cheaper passes could not close. Each escalation entry should be short and self-contained:

### [reviewer · artefact · finding-id] Short title

**Context:** one paragraph stating what the reviewer was checking.
**Observation:** what was found / could not be confirmed.
**Why it is hard:** one or two sentences on the reason a cheap pass cannot close it (e.g. needs to read a 40-page paper end-to-end; needs to recompute a non-trivial statistic from raw json; needs cross-paper consistency across all four papers at once).
**Where to look:** explicit file paths.
**What we need from the deep pass:** Pass / Fail / a specific narrow question.

Keep the escalation file lean: each item should clearly justify why it merits the more expensive pass. Items that the cheap reviewer was able to close, even with uncertainty, do not belong here — they belong in the reviewer's main report under partial.

How the playbooks were built

These verification briefs were drafted by the project author against the state of the repository on 2026-06-11 (round 1) and extended on 2026-06-16 (round 2 — five audit dimensions, Quick reference tables, cliff vs collapse discipline, false-positive prophylaxis from the round-1 DS pitch_01 audit). They reflect the v1.1 / v1.1.1 substrate. When the substrate changes (a new engine, a new sanctioning ladder, a major refactor of the World layer) the playbooks should be re-read first — the Where to look sections decay fastest.