Skip to content

Verify the software AND the experiments

Status / relationship to verify_software.md. This playbook supersedes verify_software.md for the current pass. It keeps that document's machinery checks (install, smoke, regression bit-identity, API ↔ doc) as Part A (condensed — the runnable detail still lives in verify_software.md) and adds a new Part B (xhigh): Experiment-definition & usage audit — the dimension the round-2 plan did not cover. Part B is the centre of gravity of this pass. The companion documentation-completeness pass is a separate (high-effort) playbook; this one is software + experiments.

Purpose

Two questions, both answered from zero:

  1. Is the machinery sound? A clean clone installs, the regression sweeps reproduce their baselines bit-for-bit, the smokes pass, the public API matches the docs. (Part A — same contract as verify_software.md.)
  2. Are the experiments defined and used the way the papers say they are? Every experiment that backs a published number must trace, with no silent divergence, through the chain paper Model/Methods prose -> experiment config (definition JSON or inline in a scripts/*.mjs) -> engine/agent semantics in code -> output data file -> paper table/figure -> reproduce.md recipe. (Part B.)

This pass does not re-judge whether the findings are correct (the per-paper playbooks did that, three rounds). It verifies that the experiment apparatus underneath them is correctly defined, correctly wired, and faithfully described.

Why Part B is xhigh and new. Every substantive bug found in the three paper review rounds lived here — in the gap between what a script actually does and what the paper said it does. Concretely, the failure class to hunt:

  • B1 (paper_03): prose said "Euler step h=0.10 held fixed" but the engine uses microDt = dt/10 (10 fixed micro-steps that scale with dt). The script was right; the prose was wrong — only caught by reading engine/commons.js.
  • B3 (paper_03): prose said RCPAgent extends AdaptiveAgent with a subtractive schedule; the code is extends BaseAgent with a multiplicative pollutionPressure. Caught only by opening agents/built-in.js.
  • baseFraction (paper_03): prose imported the S0 single-arena calibration 0.035; the S3 run actually uses baseFraction = 0.005, N=6. Caught only by reading scripts/pilot_v11_s3_pollution_governance.mjs + aggregate.json.
  • N2 / P-threshold sweep (paper_03): the published sweep tested only P > {0.05, 0.10, 0.15}; the "every P > eps" claim was unproven below the floor until the canonical script's P_SWEEP was extended to {0.02, 0.03, ...} and re-run.

The discipline that catches all four: re-derive the experiment config from the code and data, never from the paper's own prose. Part B operationalises that.

The experiment landscape (orient here first)

There are two experiment families, audited differently, plus a third, illustrative apparatus — the card game (Family 1b, below).

Family 1 — definition-driven (paper_00)

experiments/definitions/*.json (17 files) consumed by experiments/runner.js (Experiment class) and the Python drivers experiments/sweep.py, experiments/sweep_supplement.py, experiments/sweep_card_tournaments.py.

Definition schema (see observability_noise.json for the canonical shape): name, description, seed (default 42), baseConfig (engine params: commonsInitial, commonsCapacity, regenRate, maxTurns, threshold, scheduler), baseAgents[], sweep (axis + points[] each with configOverride and _obs), runsPerPoint, metrics[]. The runner seeds deterministically (base seed + point index + run index).

Maps to: P1-P11 + B-series sweeps in paper_00, plus card tournaments (sweep_card_tournaments.py). Recipes in docs/experiments/reproduce.md "Reproducing Platform Sweeps" and "Reproducing Card Tournaments".

Only 5 of 17 definitions have a regression baseline (tests/regression/baseline/{aggressive_fraction, adaptive_effectiveness, observability_noise, perturbation_regen_shock, mule_strategy_override}.json). The other 12 are result/supplement sweeps with no bit-identity contract — they must instead be checked definition <-> paper config and definition <-> result JSON.

Family 2 — bespoke inline scripts (paper_01 / 02 / 03)

scripts/pilot_*.mjs, scripts/robust_*.mjs, scripts/probe_*.mjs, scripts/sweep_v10_seed.mjs, scripts/calibration_v11_s0_seneca.mjs. These do not read a definition JSON — they hard-code the entire config inline (seeds, dt, k1, baseFraction, linkRate, thresholds, ladders, shockFactor, nPlayers, maxTurns, sweep arrays). Output lands in data/pilot/*, data/calibration/*, data/robust/* (all git-untracked runtime state).

Rough mapping (confirm during the audit, do not assume):

Paper Scripts Data dirs
BDPD¹ (paper_01) pilot_c1, pilot_d1_n5, pilot_d1_cell_c, pilot_d2_llm_n5, pilot_d2_ladder_sweep data/pilot/{c1,d1_n5,d1_cell_c,d2_llm_n5,d2_ladder_sweep}
BDPD² (paper_02) pilot_v10_d{1,2,3,4,5}_*, pilot_v10_d{3,4,5}_llm_n5, sweep_v10_seed, reaggregate_v10_d3_llm_n5 data/pilot/v10_*
BDPD³ (paper_03) calibration_v11_s0_seneca, pilot_v11_s{1,2,3}_*, robust_v11_s3_ofat, probe_v11_s3_dt_peaks data/pilot/v11_s*, data/calibration/*, data/robust/v11_s3

These are the high-risk family: inline config means no schema guard, and the only witness to "what was actually run" is the script + its output JSON.

Family 1b — card game (illustrative; paper_00 Humbaba appendix, paper_02 A3)

The "Forest of Humbaba" card game is a separate Python apparatus — not an engine/*.js consumer and not a definition-JSON sweep. Its pieces:

  • Card definitions — cards/cards_v03.json (the single source of truth for archetype stats, deck composition, Forest Die faces, event cards).
  • Game engine + LLM/demo play — agents/cards_ai_play.py (GameEngine, ForestDie, Player, resolve_effects; collapse only at true-zero forest).
  • Tournaments — experiments/sweep_card_tournaments.py (CT sweeps) driven by experiments/run_cards_ge_sweeps.sh and run_cards_llm_sweeps.sh.
  • Print-and-play generator — tools/card_generator/{generate_cards.py, svg_builder.py} (physical SVG cards; tooling, not a results path).
  • Data / figures — experiments/results_paper/cards/, results/<timestamp>_<sweep>/, experiments/figures/cards/.
  • Prose under test — docs/cards/*.md, paper_00/A-02-humbaba.qmd, paper_02/A3-card-game.qmd; recipes in reproduce.md §"Reproducing Card Tournaments".

Audit it at the same rigour as the commons experiments, but mind two differences: (a) the card game only illustrates / partially reproduces — it never "validates" the platform (documentation-health pitfall); (b) LLM-mode tournaments are non-deterministic (treat under B3's LLM-cell rules: N games, seeds, sigma as diagnostic spread), whereas --demo mode is seeded-random and should re-run identically. The card-specific checks are folded into B1/B2/B4.

Part A — Machinery (condensed; full detail in verify_software.md)

Effort tiering (the pass is deliberately uneven — do not run it uniformly). Part A = high (the static/regression/API checks are mechanical: the tool finds, you triage — xhigh buys nothing here). Part B = xhigh, non-negotiable: it is the centre of gravity and the only part that requires re-deriving config from code+data against the paper's prose — every real bug across three review rounds lived here. Part C = high. If forced to pick one level for the whole pass, pick xhigh (Part B dominates the value), but that wastes deep reasoning on mechanical checks — prefer xhigh on B, high on A/C.

Run, in this order, against the working tree (Mode A) or a clean clone at the current release (Mode B). The current HEAD is 3ddf706 on main; latest tag v1.1.2 (papers live on the papers-rework-v1.2-dev branch; code is on main). Update the waypoint to the actual reviewed commit.

  1. Install — ./scripts/setup.sh; venv at migration/venv_py_bdpd/ (canonical, not ../venv_py_bdpd/, not .venv/).
  2. Env guard — bdpd_check_env.require_venv() exits 0.
  3. Static consistency sweep (cheap pre-flight; run before the smokes — if the code doesn't even resolve, the rest is moot):
  4. node --check on every *.js/*.mjs under engine/ platform/ agents/ experiments/ scripts/ tools/ tests/ and main.js — zero parse errors.
  5. python -m pyflakes (or ruff check --select F) over agents/ experiments/ shared/ tools/ — undefined names, unresolved imports, bad call signatures. Treat F-class (undefined name / unresolved import / redefinition) as a finding; pure style noise (line length, etc.) is out of scope.
  6. Spot-grep the engine/plugin registry: every method named in docs/dev/plugin-api-contract.md and every extends/register* target resolves to a real symbol. This catches broken references, dead imports and renamed-but-not-updated callers that a number-only regression would miss. It is structural only — it does not assert semantic correctness (that stays in B2).
  7. Smokes — node tests/smoke_scheduler_determinism.mjs, node tests/smoke_codeagent_memory.mjs both exit 0.
  8. Regression bit-identity — ./tools/check_regression.sh (needs jq): 5/5 sweeps match tests/regression/baseline/* to the decimal. Any single decimal divergence on a non-stripped field is a FAIL.
  9. API ↔ doc — every method/endpoint/param in docs/platform/*.md exists in engine/,platform/,agents/ and vice-versa.

Part A acceptance is the table in verify_software.md. Carry its verdicts in; do not re-litigate them here unless something changed since d0a26ff.

Part B (xhigh) — Experiment-definition & usage audit

Effort: xhigh. This is the only part where the reasoning level matters — spend the bulk of the budget here (see "Effort tiering" under Part A).

For every experiment that backs a published result. Work paper-by-paper. The cardinal rule: re-derive config from code + data, never trust the paper's own prose (that prose is the thing under test).

B1. Config fidelity — script/definition vs paper Model/Methods

For each experiment, open the script (or definition JSON) and the paper's Model/Methods section side by side. Build a parameter ledger and confirm every knob matches verbatim:

  • Engine params: dt, k1/k2/l1/l2/l3 (Seneca); regenRate (=r), commonsCapacity (=K), threshold (logistic).
  • Agent params: strategy, baseFraction, intensity, reductionFactor, nPlayers/maxPlayers/minPlayers, privateResource.
  • World/coupling: linkRate, link type/direction, meta-agent mode, triggerOn, threshold, capitalThreshold, throttledK1, shockFactor.
  • Sweep axes: the exact swept values (e.g. P_SWEEP, C_SWEEP, ladder, baseFraction levels) must equal the paper's table rows — count and values. (This is where the paper_03 ladder t6/t9 and the baseFraction {.003..007} enumeration bugs lived.)
  • Run economy: seed(s), runsPerPoint/N, maxTurns.
  • Card game (Family 1b): the archetype stats, deck composition and Forest Die faces in cards/cards_v03.json must match docs/cards/archetypes.md and the paper appendix (paper_00/A-02-humbaba.qmd, paper_02/A3-card-game.qmd) verbatim; the tournament sweep values in sweep_card_tournaments.py (matchups, rounds 9/12/15, --forest-noise, Stranger-King defection turn, games-per-matchup) must equal the paper / reproduce.md table — count and values.

Flag any value that differs from the paper, any default the paper claims is overridden but isn't (and vice-versa), and any "default" cited in prose that disagrees with the engine default in code (e.g. SENECA_DEFAULTS, GAME_DEFAULTS).

B2. Semantic fidelity — does the code mean what the paper says it means

Open the engine/agent code the experiment exercises and confirm the paper's description of the mechanism matches the implementation:

  • Integration: engine/commons.js senecaEulerStep / microDt = dt/steps (the B1 trap). Harvest-before-ODE ordering as documented.
  • Agent rules: agents/built-in.js — RCPAgent extends BaseAgent, pollutionPressure = min(1, 5P), baseFraction*(1-p), halving at R/K<=0.5 (the B3 trap); Conservative tiers 0.60/0.35/0.10; Adaptive 0.40 start with -0.20f/-0.30f; Aggressive intensity=0.5.
  • Governance: pollution_regulator lever semantics (cap=regen_set, levy=capital_shock, fine=wealth_shock); perturbation ladder ladder[min(ordinal-1, len-1)] (last sticks); pact term types.
  • Metrics: the 5 governance metrics in engine/metrics.js, welfare (1-G)*meanWealth, capacity 1 + 0.05*w.
  • Card game (Family 1b): agents/cards_ai_play.py is Python and independent of engine/*.js — read it directly. Confirm the paper's described mechanics match GameEngine/ForestDie/resolve_effects: collapse only at true zero forest (no fixed Gate threshold), noisy observability as a Gaussian blur on the Forest Deck, the Stranger-King defection deadline scaling with max_rounds, and the per-archetype behavioural nudges in the system prompt.

B3. Output fidelity — script/definition vs its own data file

Re-run (Mode B) or read the committed output (Mode A) and confirm the headline numbers the paper quotes are present in the output JSON at the cited precision:

  • Deterministic experiments (Seneca ODE, built-in sweeps): re-running must reproduce the data bit-identically. Spot-check 2-3 by re-run + checksum; for paper_03 note trajectories.json must stay byte-identical when only a sweep array is extended (figures depend on it).
  • Re-derive any computed quantity yourself from the raw fields (recoverB, Gini, %-of-peak, ratios, Cohen's d) — do not copy the paper's arithmetic.
  • LLM (non-deterministic) cells: confirm N, seeds, and that sigma is reported as diagnostic spread, not a confidence bound; confirm cost/usage accounting (trace-level calls vs aggregate decision calls — the paper_02 cost-table case).

B4. Traceability matrix — completeness, orphans, stale

Build the full matrix and check it is total and clean:

experiment id <-> script/definition <-> data dir <-> paper table/figure <-> reproduce.md recipe

  • No orphan scripts: every scripts/pilot_*/robust_*/probe_* and every definitions/*.json either backs a published result or is explicitly labelled exploratory/superseded. Resolve look-alikes (e.g. pilot_d1.mjs vs pilot_d1_n5.mjs, pilot_d2_llm.mjs vs pilot_d2_llm_n5.mjs — which is the paper's? is the other dead?).
  • No orphan claims: every headline number in every paper has a script/def + data + reproduce recipe behind it.
  • reproduce.md is total and current: every recipe's script path, input data path, and output path exist; no recipe points at a renamed/removed file.
  • Card game (Family 1b): sweep_card_tournaments.py, run_cards_ge_sweeps.sh, run_cards_llm_sweeps.sh and cards_ai_play.py each back a CT result in paper_00 (Humbaba) / paper_02 A3, or are explicitly labelled exploratory. CT-count is inconsistent across sources — verified reality (2026-06-23): paper_00 §3 defines CT1–CT6; the script implements CT1–CT5 (sweep_ct1..5, --sweep choices ct1..ct5) yet its docstring still says "CT1–CT4" (stale); results_paper/cards/ holds CT1–CT5 data; reproduce.md header claims "CT1–CT6" but lists recipes only for CT1–CT5. CT6 ("LLM awareness-without-restraint") is a qualitative LLM finding with no sweep, no data JSON and no recipe — legitimate, but label it qualitative-LLM in the matrix, not a missing sweep. Fixes to land: (a) script docstring CT1–CT4 -> CT1–CT5; (b) reproduce.md header CT1–CT6 -> CT1–CT5 + a note that CT6 is qualitative (from cards_ai_play.py LLM logs). tools/card_generator/* is print-and-play tooling, not a results path: confirm the cards it emits agree with cards/cards_v03.json rather than treating it as backing a number.

B5. Determinism & seed discipline

  • Claimed-deterministic experiments are deterministic: Seneca seed is inert (seed 17 ≡ seed 42), RCP agents draw no RNG, built-in sweeps re-run identical. Verify by reading the RNG usage, not by trusting the comment.
  • N=5 LLM sweeps use the documented seed set; aggregate.json reflects those seeds; per-seed values aren't silently cherry-picked into an "N=5" cell (the paper_02 V3 final-S case).

Part B acceptance

Check Pass Partial Fail
B1 config fidelity every knob matches paper for every experiment 1-2 minor prose/label drifts a swept value / default / count disagrees
B2 semantic fidelity every mechanism description matches code 1 wording imprecision a described rule contradicts the implementation
B3 output fidelity every quoted number traced to output JSON; deterministic re-runs bit-identical 1 rounding drift a number absent from data / re-run diverges
B4 traceability matrix total, 0 orphans, reproduce.md current 1 unlabelled exploratory script a published claim with no runnable backing
B5 determinism claims verified against RNG usage; seeds documented 1 undocumented seed non-determinism where determinism is claimed

Part C — Numeric & documentation-health (reference)

Effort: high (tracing + grep, rigorous but not deep reconstruction).

Run the two round-2 dimensions from verify_software.md unchanged: Numeric audit (every doc-quoted literal traced to a baseline/script; engine-constant table) and Documentation health (narrative read-through, surface-area balance, API↔doc harmonization, epistemic-honesty sweep + canonical transparency statement). Their acceptance tables carry over verbatim.

Common pitfalls (read before starting)

  • Re-derive from code+data, not from paper prose. The prose is under test; this is the lesson of B1/B3/baseFraction. Validating a number against another sentence in the paper is not validation.
  • Inline config is the high-risk surface. Family-2 scripts have no schema guard; the script body + its output JSON are the only ground truth.
  • data/ is untracked runtime state. Re-running regenerates it; that is fine and has no commit impact, but never git add it, and confirm figures' source JSON (e.g. trajectories.json) stays byte-identical when you only meant to extend a sweep.
  • Only 5/17 definitions have a regression baseline. The other 12 are checked by definition↔paper↔result, not by bit-identity — do not report them as "no baseline => fail".
  • Recurring reviewer false positives to pre-empt (verified spurious across 3 paper rounds): (a) DS "hyperref typo" — \usepackage{hyperref} is the correct package name in every _quarto.yml; (b) "GLM-5.1 vs GLM-5" — the model used is GLM-5.1, which on Z.ai points to the GLM-5 report, so text "5.1"
  • ref glm2026 is intentional; (c) MiMo "gao2023s3 not in local bib" — paper_02 uses the shared ../paper_00/bdpd_full.bib, which contains it.
  • DS issues wrong-direction fixes on license/version/dependency/config claims (see [[verify-ds-claims]]). For any "the repo declares X" finding, open the canonical file (LICENSE, package.json, _quarto.yml, a definition JSON, the script) before applying anything.
  • migration/ is the working tree, not the repo root (bdpd/ is). Quarto rendering is not part of this pass.

Output

One report per reviewer at tmp/verification_sw_experiments/<reviewer>_report.md (<reviewer> ∈ claude,ds,mimo; do not overwrite the round-2 verification_round2/ reports), containing:

  1. Environment — OS, Node/Python versions, reviewed commit hash.
  2. Part A table — machinery acceptance (install/static-consistency/smoke/regression/API↔doc), Pass/Partial/Fail + one-sentence evidence.
  3. Part B — experiment audit, the centrepiece:
  4. the traceability matrix (every experiment: script/def ↔ data ↔ paper table/figure ↔ reproduce recipe);
  5. the parameter ledger per experiment (B1) with any divergence flagged;
  6. the B2/B3/B5 findings;
  7. the B-acceptance table filled in.
  8. Numeric + documentation-health tables (Part C).
  9. Findings list — numbered, severity (block/major/minor), file:line, expected vs observed, suggested fix or unknown.
  10. Reproducibility verdict — could a third party reproduce every published result from a clean clone following only reproduce.md? List any undocumented prerequisite.

Reviewer prompt (copy-paste)

You are an independent reviewer verifying both the BDPD software machinery and, primarily, its experiment apparatus — from zero. Follow migration/docs/verification/verify_software_and_experiments.md top to bottom. Part A (machinery) is condensed from verify_software.md; run it but spend the bulk of your time on Part B: the experiment-definition & usage audit, at maximum rigour. For every experiment that backs a published number, re-derive its configuration from the script/definition and the output data — never from the paper's own prose — and confirm the full chain paper Model -> config -> engine/agent code -> output JSON -> paper table -> reproduce.md recipe holds with no silent divergence. The four bug classes to hunt are named at the top of the playbook (B1 integration prose, B3 agent superclass, baseFraction import, sweep-coverage). Build the traceability matrix and the per-experiment parameter ledger explicitly. There are two experiment families: definition-driven (experiments/definitions/*.json, paper_00) and bespoke inline scripts (scripts/pilot_*/robust_*/probe_*, papers 01-03) — the inline family is higher risk. Read the Common-pitfalls section first (it lists three recurring false positives to NOT re-raise, and the wrong-direction DS fixes). Write your report to tmp/verification_sw_experiments/<reviewer>_report.md. Flag anything you cannot evaluate (skip with reason; never fabricate). Time budget: ~8-10h for the standard pass, more if you re-run deterministic sweeps for bit-identity.