Experiments — Protocol & Metrics¶
BDPD experiments span two instruments — the Platform (continuous, deterministic largely) and the Card Game (discrete, stochastic) — with unified metrics and statistical procedures.
New here? Start with the Experiment Map: the four tiers from simple deterministic sweeps to full LLM arenas with cheap-talk, pacts and sanctions, plus how to make your own variant. This page covers the metrics and statistical protocol behind them.
Run Counts¶
| Instrument | Runs per Cell | Reason |
|---|---|---|
| Platform sweeps (P1–P11, B1–B5) | 20–30 | Outcomes are near-binary far from phase boundaries; 20 runs suffice to establish threshold regions |
| Card tournaments (CT1–CT5) | 200 | Higher stochastic variance from Forest Die, card shuffle, and LLM non-determinism |
| LLM case studies (CT6) | 12–20, fixed seed | Qualitative, preliminary — not statistical |
Metrics¶
Three primary metrics are tracked in every experiment:
Gate Pass Rate (Platform) / Collapse Rate (Cards)¶
Binary outcome per run: did the commons survive?
- Platform: Gate passes if \(S_\text{final} > \text{threshold}\) at turn limit
- Cards: Forest survives if Forest Die never triggers collapse by round 8
Welfare Score¶
Sen-type aggregate balancing collective efficiency and equality:
where \(G\) is the Gini coefficient and \(\bar{w}\) is mean final wealth. A welfare of 100 means perfect equality at full carrying capacity; 0 means total collapse with perfect inequality.
Gini Coefficient¶
Standard measure of wealth inequality across agents at the final turn. 0 = perfect equality; 1 = one agent holds all wealth.
Statistical Approach¶
| Instrument | Method | Reporting |
|---|---|---|
| Platform deterministic sweeps | Bootstrap at between-cell level (resampling cells, 10,000 iterations) | 95% bootstrap CIs |
| Platform stochastic sweeps | Mean ± 1σ per cell | Shaded σ bands |
| Card tournaments | Bootstrap (10,000 resamples) | 95% bootstrap CIs on all metrics |
| Cross-sweep inference (B5) | Logistic regression + permutation tests | 95% CIs excluding zero = robust |
For deterministic sweeps (all except observability_noise and
scheduler_comparison), intra-cell variance is zero. Inference is
performed at the between-cell level via bootstrap resampling of cells.
Reproducibility¶
All experiments are deterministic given a seed. Seeds, configuration
files (JSON sweep definitions), and raw output are included in the
repository under experiments/definitions/. The BDPD platform and card
simulator are open-source (Node.js and Python respectively).
See Reproducing Results for exact commands.
Experiment Catalogue¶
| Experiment | Instrument | Type | Key Variable |
|---|---|---|---|
| P1 | Platform | Aggressive fraction sweep | Number of aggressive agents |
| P2 | Platform | Regen rate sweep | \(r \in [0.04, 0.32]\) |
| P3 | Platform | Pool size scaling | \(N \in [2, 12]\) |
| P4 | Platform | Pool size fine | \(N \in [8, 20]\) |
| P5 | Platform | 2D parametric map | \(r \times \text{aggressive fraction}\) |
| P6 | Platform | Asymmetry sweep | Aggressor initial endowment |
| P7 | Platform | Scheduler comparison | Simultaneous / sequential / wealth-weighted |
| P8 | Platform | Adaptive effectiveness | Adaptive vs. conservative pools |
| P9 | Platform | Observability noise | \(\sigma/\mu \in [0.0, 0.5]\) |
| P10 | Platform | Regen shock | Shock factor \(f\) at turn 20 |
| P11 | Platform | Mule strategy override | Defection turn \(T \in [5, 45]\) |
| B1 | Platform | Intensity cliff-edge | \(i \in [0.05, 1.0]\) + fine \([0.04, 0.11]\) |
| B2 | Platform | Adaptive reductionFactor | \(f \in [0, 1.5]\) |
| B3 | Platform | Extended regen rescue | \(r \in [0.30, 1.50]\) + fine \([0.80, 1.05]\) |
| B4 | Platform | Pure Seneca shock | Shock factor (no aggressor) |
| B5 | Platform | Cross-sweep inference | Logistic regression, bootstrap CIs |
| CT1 | Cards | Collapse-rate matrix | 4×4 archetype matchups |
| CT2 | Cards | Game-length sweep | 9, 12, 15 rounds |
| CT3 | Cards | Observability noise | \(\sigma/\mu \in [0.0, 0.5]\) + hidden |
| CT4 | Cards | Defection timing | Forced defection turn 3–11 |
| CT5 | Cards | Stranger-King deep dive | Patience, nudge, hand composition |
| CT6 | Cards | LLM qualitative | Single-model case studies |
The LLM governance pilots (cheap-talk, pacts, graduated sanctions,
governance meta-agents, Seneca cascade) are standalone scripts in
scripts/ rather than sweep definitions — catalogued in the
Experiment Map.
Figures¶
All experiment figures are stored in docs/publications/pictures/.
They are rendered with plot_theme.py with both dark and
_light_paper variants. The paper uses light variants; the docs
website (dark theme) can reference either.