Verify the software¶

Purpose¶

Confirm that the BDPD codebase is runnable, reproducible, and numerically stable on a clean clone. A pass means: a fresh checkout installs cleanly, the regression sweeps reproduce the recorded baselines bit-for-bit, the documented reproduce procedures yield the artefacts they claim to yield, and the public API surface matches what the documentation says it does.

This playbook does not verify whether the findings are correct — that is the job of the per-paper playbooks. It verifies that the machinery underneath the findings is sound.

Round-2 scope (2026-06-16+). Round 2 is a complete re-verification from zero, not a delta of round 1. Every item is re-checked; round-1 verdicts have no privileged status. Two new audit dimensions are added at the bottom: Numeric audit (every quoted number in the documentation traced to a regression baseline or a script) and Documentation health audit (narrative flow, balance, harmonization, epistemic honesty applied to the docs surface). Round-1 reports live at tmp/verification/<reviewer>_software_report.md; round-2 reports land at tmp/verification_round2/<reviewer>_software_report.md (do not overwrite). A reviewer may consult round-1 reports as input but must not inherit their conclusions.

Where to look¶

Sources — code under verification¶

main.js — Express entry point (Arena server)
platform/*.js — Arena, World, registry, scheduler, observability, perturbation, pacts, messages, reputation, sandbox, streamer, reporter
agents/*.{js,py} — built-in heuristics, API agent, governance meta-agents, LLM middleman, card-game runner, bdpd_sdk/
engine/*.js — base-engine, engine-registry, logistic, seneca, commons, metrics
experiments/*.{js,py} + experiments/definitions/*.json — sweep framework and 17 sweep definitions
shared/*.py — version, manifest, logging, env check
tools/*.{sh,py,mjs} — regression, manifest, plugin checks, card generator, panel

Compiled output / evidence to consult¶

Regression baselines: tests/regression/baseline/{adaptive_effectiveness,aggressive_fraction,mule_strategy_override,observability_noise,perturbation_regen_shock}.json — these are the immutable numerical fingerprints the code must reproduce.
Smoke tests: tests/smoke_codeagent_memory.mjs, tests/smoke_scheduler_determinism.mjs
Existing pilot data (for spot-checking, not regression): data/pilot/{c1,d1,d2,d3,…,v10_*,v11_*}/

Supporting documentation¶

docs/getting-started/installation.md — install instructions (Linux native)
docs/getting-started/installation-docker.md — Docker route
docs/experiments/reproduce.md — the master reproduce document. Every figure/finding in the papers must be reachable from a recipe listed here.
docs/dev/setup.md, docs/dev/shared.md, docs/dev/plugin-api-contract.md — internal contracts the code must respect.
docs/dev/index.md — architecture overview.
docs/platform/*.md — public API documentation (Arena, World, agents, perturbations, governance, pacts, cheap-talk, reproducibility).

Library — not used here¶

The software verification does not consult the bibliography library. It is purely about runtime behaviour and code correctness.

Procedure¶

The numbered steps below are written for Mode B (clean external clone, full execution). For Mode A (working repository — the three-pass internal verification), the execute verbs become verify by reading; see the Mode-A adaptation block under each step.

Clean clone. Start from a fresh git clone (not your working tree). Check out the tag corresponding to the version under review (v1.1.2 for this playbook iteration).
Mode A: skip entirely. The working tree at migration/ is the substitute.
Install. Run ./scripts/setup.sh. Expected: Python venv created at venv_py_bdpd/, Node modules installed, no manual prerequisites missing. Note any prompt the script asks the human for — those should be documented in installation.md.
Mode A: read scripts/setup.sh end-to-end. Confirm each step is documented in docs/getting-started/installation.md. Confirm the venv path it produces matches what the docs claim (canonical: venv_py_bdpd/ at the migration root, per setup.sh VENV_PATH and installation.md; the ../venv_py_bdpd/ form is the author's local layout, not the clean-clone path — see hint below). Do not re-run.
Activate environment. source venv_py_bdpd/bin/activate. Confirm python -c "import bdpd_check_env; bdpd_check_env.require_venv()" exits 0.
Mode A: read shared/bdpd_check_env.py. Confirm the guard is reachable from every Python entry point. Do not re-execute.
Smoke checks.
node tests/smoke_scheduler_determinism.mjs — must exit 0.
node tests/smoke_codeagent_memory.mjs — must exit 0.
node main.js — Arena server should bind to port 3000; curl http://localhost:3000/api/health should respond.
Mode A: read each smoke test file. Confirm the assertions it makes still match the current code behaviour by tracing the relevant functions. Spot-execute one of the two smokes to confirm the recipe is current. Do not start the server.
Regression sweeps. Run ./tools/check_regression.sh. The script executes all five canonical sweeps (aggressive_fraction, adaptive_effectiveness, observability_noise, perturbation_regen_shock, mule_strategy_override) and diffs the JSON output against tests/regression/baseline/<sweep>.json with non-deterministic fields stripped. Any single decimal divergence is a failure — the regression contract is bit-identity, not "close enough".
Mode A: read tools/check_regression.sh and the five baseline JSONs in tests/regression/baseline/. Confirm the baselines exist, the sweep names match experiments/definitions/, and the strip-list in the script matches the non-deterministic fields actually present in current output. Do not run the sweeps.
Reproduce one figure end-to-end. Pick one figure from docs/experiments/reproduce.md, follow the recipe exactly, and compare the produced artefact against the same figure in the corresponding paper's _site/. The reproduce doc lists recipes for every figure in BDPD⁰..³.
Mode A: read docs/experiments/reproduce.md end-to-end. For each documented recipe, confirm: (a) the script it references exists; (b) the input data path exists; (c) the output figure exists in the corresponding paper's _site/ or under experiments/figures/. Pick one figure and re-execute the recipe to confirm the pipeline still runs.
Public API surface. For each docs/platform/<topic>.md, confirm that every method / endpoint / parameter documented exists in the code at the documented path. Mismatches in either direction (documented but missing in code, present in code but undocumented) are findings.
Mode A: same as Mode B (no execution involved here). Reads only.
Plugin contract. If docs/dev/plugin-api-contract.md lists hooks or extension points, run node tools/plugin-check.mjs (if present) or manually grep the engine registry to confirm each contract method is callable.
Mode A: read-only contract check is sufficient; skip plugin-check.mjs execution.
Round-2 git-log waypoint. git log d0a26ff..HEAD -- {engine,platform,agents,scripts,experiments,tests}/ (the round-1 closing commit was 5fa0eaf on 2026-06-11; round-2 waypoint is d0a26ff on 2026-06-16). This list is a reading-priority hint, not a scope limit — every check above is still performed in full.

Round-2 audit dimensions¶

The two new dimensions below augment the core procedure. They are mandatory for round 2; round 1 did not require them.

Numeric audit (round 2)¶

Purpose. Round 1 verified that regression sweeps match baselines and that smoke tests pass. Round 2 additionally re-traces every literal number quoted in the documentation (README, installation, reproduce.md, dev docs, platform API docs) to a regression baseline or a script, on the same logic the paper playbooks apply to their prose.

Procedure.

Numeric ledger. Grep docs/, README.md, CITATION.cff for literal numbers (regex [0-9]+\.?[0-9]*\s*(%|×|x)?\b minus stoplist of port numbers, years, file paths). Build a ledger: number ↔ doc:line ↔ candidate source.
Source-of-truth precedence: regression baseline > pilot aggregate > experiment definition > paper qmd. A number in README that comes from a regression baseline must match it to the documented precision.
Engine constants — confirm any quoted parameter default matches the code:
r = 0.12 (canonical regen) → engine/logistic.js:23
r ≤ 0.32 (slow-regen band) → experiments/definitions/regen_rate_extended.json
[1, 3, 10] (canonical graduated ladder) → scripts/pilot_d2.mjs:9
r ≈ 0.95 (rescue point) → paper_00/A-B-supplement.qmd:112-115 with bootstrap CI [0.95, 1.025]

Acceptance.

Check	Pass	Partial	Fail
Numeric ledger coverage	every literal in docs traced	1–2 untraced	systemic untraced
Engine constants	every quoted default matches code	1 stale value in docs	systemic
Regression baseline ↔ doc claims	match	1 rounding drift	magnitude mismatch

Documentation health audit (round 2)¶

Purpose. Single dimension that absorbs the paper-playbook dimensions Narrative-flow / Balance / Harmonization / Epistemic-honesty applied to the documentation surface (README, getting-started, dev, platform, experiments). The codebase itself has its own quality controls (regression, smoke); the docs do not, and round 2 closes that gap.

Procedure.

Narrative read-through. Start at README.md, follow "Quick Start" to installation.md, follow installation to a working npm test, then to reproduce.md to a recreated figure. A new user must be able to walk this path without backtracking. Note every place where the path forks ambiguously or the next step is not where the previous step pointed.
Surface-area balance. Compare doc-page lengths vs the code-domain weight they cover. docs/platform/world.md should not be 5× longer than docs/platform/arena.md if Arena is the load-bearing primitive. A top-level concept with no doc page is a hole; a niche helper with three pages is bloat.
API ↔ doc harmonization. Every name in docs/platform/*.md matches the symbol in engine/, platform/, agents/. Naming drift (harvest in docs vs extract in code) is a finding. Same for log formats, env vars, CLI flags.
Epistemic-honesty sweep. Grep README, top-level docs, and CITATION.cff for: \b(only|first|never|every|all|none|unique|novel|breakthrough)\b. Each unqualified hit is a finding. The canonical transparency statement ("source-grounded by prompt, validated by extensive triaging with different models, integrated under author supervision") must appear verbatim in the README / CITATION.cff / getting-started one-liners — no "every output reviewed" / "all decisions verified" / "extensively validated" stale variants. (The publications carry an extended transparency footnote — paper_NN/00-abstract.qmd and the index.qmd of blog_book / mini_course / overview_00 — whose prose reads "…validated by extensive triaging with different models and finally integrated under author supervision". The substance is identical; the "and finally" connective is intentional sequencing prose, not a stale variant — do not flag it.) Confirm zero non-bibliographic name-drops of any individual (social capital) in README, CITATION.cff, getting-started.

Acceptance.

Check	Pass	Partial	Fail
Narrative read-through	reads continuous from README to reproduce	1 fork unclear	path broken
Surface-area balance	doc pages proportional to code-domain weight	1 outlier	systemic bloat or holes
API ↔ doc harmonization	full match	1–2 name drifts	systemic
Epistemic honesty	every superlative qualified, canonical transparency statement	1 stale form	systemic bragging or stale forms

Hints / known traps.

The transparency statement is canonical and must appear verbatim. Stale variants to grep-and-replace: "every output was reviewed", "all decisions verified", "extensively validated", "thoroughly vetted".
The card game is never "validates" the platform; only "partially reproduces / cross-checks / illustrates on a tangible substrate."
The repo uses AGPL-3.0 for code and CC BY 4.0 for docs/game. Any README claim of "MIT" or "GPL" is stale (we hit this in the 2026-06-07 audit; DS gave a wrong-direction fix that contradicted migration/LICENSE).
migration/ is the working tree, not the repo root on GitLab — bdpd/ is. Doc references like "migration/docs/..." are correct in this playbook (cwd-relative) but a reviewer following them on a clone must cd migration first.

Acceptance criteria (composite)¶

Check	Pass	Partial	Fail
Install via `setup.sh`	exits 0, all deps present	exits 0 but warns	non-zero exit or missing dep
Smoke tests	both exit 0	one warns	any non-zero exit
Regression sweeps	5/5 match baseline bit-identically	4/5 with divergence in known field	any divergence on unknown field
Figure reproduce (pick one)	output equals paper figure	minor stylistic delta (font, dpi)	numerical difference
API doc ↔ code	full bidirectional match	minor labels off	a documented method does not exist
Numeric audit	every doc-quoted number traced	1–2 untraced	systemic
Documentation health	narrative + balance + harmonization + epistemic-honesty all pass	one dimension partial	one dimension fails

A reviewer should annotate each row as Pass / Partial / Fail with one sentence of evidence each.

Common pitfalls¶

venv_py_bdpd lives at the migration root. ./scripts/setup.sh creates it at migration/venv_py_bdpd/; docs/getting-started/installation.md documents that path. Do not look for a .venv/ inside the working tree. The path ../venv_py_bdpd/ (one level above migration) may appear in some legacy docs and in the project author's local setup, but the canonical location for a clean-clone reviewer is inside migration/.
data/ is untracked and runtime-state — committing it would pollute the regression baselines. Confirm it stays out of git status.
Quarto rendering is not part of software verification. Render checks belong to the paper playbooks.
bdpd_check_env.require_venv() will hard-fail if the wrong Python is active. This is a feature, not a bug.
tools/check_regression.sh requires jq. Confirm jq --version works before running.
Smoke tests use built-in agents. They do not touch LLM providers; no API key is needed.
DS audits can issue wrong-direction fixes for license/version/dependency claims (see [[verify-ds-claims]]). For any "the repo declares X" issue raised by DS, read the canonical file (LICENSE, package.json, _quarto.yml, CITATION.cff) directly before applying anything.
Round-1 vs round-2 reports: round-1 lives at tmp/verification/<reviewer>_software_report.md; round-2 at tmp/verification_round2/<reviewer>_software_report.md. Do not overwrite, do not inherit verdicts.

Output¶

The reviewer is expected to produce a single report at tmp/verification_round2/<reviewer>_software_report.md with:

Environment — host OS, Node version, Python version, commit hash under review (round-2 waypoint: d0a26ff on papers-rework-v1.2-dev).
Per-check table — the composite acceptance-criteria table above filled in with Pass / Partial / Fail + one-sentence evidence per row.
Findings list — each finding numbered, with: severity (block / major / minor), file:line, what the reviewer expected vs what was observed, suggested fix or unknown.
Numeric ledger — table of every literal number found in docs/ and README.md, with its anchor.
Documentation health table — narrative / balance / harmonization / epistemic-honesty rows with pass-partial-fail + one finding each.
Reproducibility statement — a one-paragraph verdict on whether a third party could replicate the published results from a clean clone, listing any unmet prerequisite the documentation should disclose.

Reviewer prompt (copy-paste)¶

You are an independent software-engineering reviewer asked to verify that the BDPD codebase reproduces what its documentation claims, completely and from zero — this is round 2, no round-1 verdict is inherited. You have a clean clone of the repository at tag v1.1.2 (or working tree on the round-2 working branch). Follow the playbook at migration/docs/verification/verify_software.md from top to bottom: core Procedure (steps 1–9) plus the two Round-2 audit dimensions (Numeric audit and Documentation health audit). Produce a written report at tmp/verification_round2/<reviewer>_software_report.md (replace <reviewer> with your reviewer slug: claude, ds, mimo). Do not consult the per-paper verification playbooks for this task — the software pass is upstream of the scientific findings. Limit yourself to the machinery and its documentation surface: installability, regression bit-identity, smoke tests, figure reproducibility, bidirectional API ↔ doc consistency, numeric traceability of every doc-quoted value, and the four documentation-health checks (narrative read-through, surface-area balance, API harmonization, epistemic honesty). Flag any acceptance criterion you cannot evaluate (skip with reason; do not fabricate a result). Read the Common-pitfalls section before you start: it lists the wrong-direction DS fixes and the canonical transparency statement to match against. Time budget: roughly six hours of focused work for the standard round-2 pass, ten hours if you also reproduce two figures end-to-end.