Reproducibility¶

This page collects the operational details a third party needs to re-run the LLM-driven pilots reported in the BDPD papers. The intended audience is anyone (including future maintainers) who wants to regenerate data/pilot/<name>/aggregate.json from scratch and confirm the published numbers within seed-level noise.

It complements rather than replaces the papers: each paper carries the scientific claims and limitations; this page carries the mechanical recipe to reproduce them.

Prerequisites¶

Requirement	Tested with
Node.js	`>= 20.x` (the pilots use `node:*` core modules and ESM)
Python	`>= 3.12` for the LLM agent process
DeepSeek API key	Exported as `DEEPSEEK_API_KEY=sk-...` before running an LLM pilot
Python deps	Install from the project root: `pip install -r agents/requirements.txt` (or activate the project venv at `venv_py_bdpd/`)
Node deps	`npm install` at the project root

All LLM-driven runs hit the public DeepSeek API (https://api.deepseek.com/v1, OpenAI-compatible). A run without DEEPSEEK_API_KEY set will print SKIP: no DEEPSEEK_API_KEY in environment. and exit without spending tokens.

What "reproducible" means here¶

LLM-driven pilots are not bit-reproducible, and re-running one will not reproduce a published aggregate.json to the digit:

Temperature 0.4 makes sampling stochastic, so runs vary from seed to seed.
Even at temperature 0 a hosted endpoint is not bit-deterministic: expert routing in a mixture-of-experts model depends on batch composition, and floating-point reductions are order-sensitive on the server's hardware.
The checkpoint drifts. deepseek-v4-flash is a cloud-served model with no user-pinnable version; the provider may update or deprecate it between a paper's run date and any replication attempt (for example, the deepseek-chat/deepseek-reasoner aliases are deprecated on 2026-07-24).

Re-running therefore confirms the published numbers within cross-seed noise and within the run-date model, not exactly. Bit-identical reproduction requires a locally hosted, version-pinned model with deterministic inference, or trace-based replay of the recorded LLM responses rather than fresh API calls. The papers' quantitative claims are framed accordingly — as directional effects and cross-seed bands, not point estimates.

Running the pilots¶

Each pilot script is a self-contained orchestrator that spawns one Python subprocess per LLM-driven player, runs the configured number of seeds, and writes data/pilot/<name>/aggregate.json alongside per-seed artefacts.

BDPD¹ — governance pilots (single-arena)¶

# C1 — mechanism check: do sanctions fire 1:1 against detected violations?
DEEPSEEK_API_KEY=sk-... node scripts/pilot_c1.mjs

# D1 — architecture/talk decomposition: Cell A (built-in) + Cell B (LLM, talk on)
DEEPSEEK_API_KEY=sk-... node scripts/pilot_d1_n5.mjs

# D1 Cell C — LLM agents with the cheap-talk tool surface disabled.
DEEPSEEK_API_KEY=sk-... node scripts/pilot_d1_cell_c.mjs

# D2 — graduated vs flat sanctioning.
DEEPSEEK_API_KEY=sk-... node scripts/pilot_d2_llm_n5.mjs

# D3 — ladder geometry sweep (soft, hard, flat-5 control).
DEEPSEEK_API_KEY=sk-... node scripts/pilot_d2_ladder_sweep.mjs

Plots (no further LLM calls):

python scripts/plot_d1_n5.py
python scripts/plot_d2_llm_n5.py
python scripts/plot_d2_ladder_sweep.py

BDPD² — nested-arena vignettes (multi-arena `World`)¶

# V1 — cliff isolation across a 10-arena polycentric World (built-in).
node scripts/pilot_v10_d1_cliff_polycentric.mjs

# V2 — sanctioner ladder, built-in baseline + LLM cell.
node scripts/pilot_v10_d3_yoon_voluntary.mjs                    # built-in
DEEPSEEK_API_KEY=sk-... node scripts/pilot_v10_d3_llm_n5.mjs    # LLM

# V3 — world-level treaty enforcer, built-in baseline + LLM cell.
node scripts/pilot_v10_d4_treaty_enforcer.mjs                   # built-in
DEEPSEEK_API_KEY=sk-... node scripts/pilot_v10_d4_llm_n5.mjs    # LLM

# V4 — coercive exclusion to a junk arena, built-in + LLM.
node scripts/pilot_v10_d5_exclusion_sweep.mjs                   # built-in
DEEPSEEK_API_KEY=sk-... node scripts/pilot_v10_d5_llm_n5.mjs    # LLM

Plots:

python scripts/plot_v2_inversion.py
python scripts/plot_signal_adaptive.py

The vignette ↔ pilot mapping is also tabulated in BDPD² Appendix A1 (@sec-v-d-mapping); the V/D naming reflects the development order (D-numbered pilots predate the final V-numbered vignettes).

Notes on output paths¶

The figure outputs land in experiments/figures/pilot/ for BDPD¹ and docs/publications/pictures/ for BDPD², each with up to three theme variants per headline (__light_paper.png, __solarized_light.png, __solarized_dark.png). The papers reference the *_light_paper.png variant.

data/pilot/ is intentionally git-ignored: it is a runtime cache, not versioned content. Re-running a pilot script overwrites the matching subdirectory under data/pilot/<name>/.

Reaggregating from per-seed artefacts¶

The per-vignette aggregate files in data/pilot/<name>/aggregate.json are produced as a side-effect of the seeded pilot loop in pilot_*.mjs: the loop keeps an in-memory allResults array and serialises it at the end. If a pilot is interrupted and resumed only for the missing seeds (a recovery pattern common before the SDK retry layer was added in agents/bdpd_sdk/client.py), the resumed process sees only its own subset of seeds in memory, even though the per-seed comparison.json files from the prior partial run are all present on disk — and writes an aggregate with a truncated $N$.

For BDPD² V2 LLM (v10_d3_llm_n5), a standalone reaggregator — scripts/reaggregate_v10_d3_llm_n5.mjs — rebuilds aggregate.json from the on-disk per-seed files. It takes no API key and no LLM cost. Run it whenever an aggregate's $N$ disagrees with the number of seed_* subdirectories actually present:

node scripts/reaggregate_v10_d3_llm_n5.mjs

The same recovery pattern generalises trivially to other pilots if the situation recurs: the loader walks seed_*/comparison.json and delegates to the standard stats() reducer.

Agent prompt template¶

LLM agents receive a layered system prompt built at process start by agents/bdpd_agent.py:get_system_prompt. The layers are:

BASE_GAME_RULES — the game contract (goal, commons dynamics, observation fields). Identical in every pilot.
A response-format block — either JSON_RESPONSE_INSTRUCTION (when the announcement channel is exposed) or JSON_RESPONSE_INSTRUCTION_NO_TALK (Cell C of D1, where --no-talk disables the announced field and all communication tools).
An archetype nudge — one of ARCHETYPE_NUDGES[archetype]. The governance pilots use only conservative and aggressive.
(optional) A noisy-observation note when the arena is configured with hidden or noisy commons stock. None of the governance pilots enable this.

The exact text of each block lives in agents/bdpd_agent.py:98–216. For reference, the two archetype nudges used in the pilots are reproduced verbatim:

conservative:

You are a conservative steward. You prioritise the long-term health of the commons
over short-term gain. You will reduce your harvest drastically if the stock appears
low, and you are willing to accept a smaller final wealth to avoid collapse.
If the stock is below 40% of maximum, you harvest very little (≤20% of capacity).
If the stock is healthy (>60%), you may harvest up to 40% of capacity.
You never harvest more than 50% of your capacity, no matter how tempting.
You watch others: if they are over-harvesting, you may reduce further to compensate.

aggressive:

You are an aggressive extractor. Your goal is to maximise your own wealth quickly,
even if it risks depleting the commons. You believe the resource is vast and
regeneration will keep pace. When in doubt, harvest near your full capacity.

The nudges include numerical heuristics on purpose: they encode the behavioural priors the papers attribute to each archetype, and they are the most likely component to drive the architectural lift reported in BDPD¹ Cell C. A prompt-free control would be needed to isolate the model's intrinsic prosocial tendency from the prompt effect; this is flagged as a limitation rather than removed.

The dynamic part of each turn — observation, inbox, pact view — is appended to the user message by obs_to_text so that the cache-stable prefix (system prompt + tool schema + baked history) sits at the front of the request, eligible for provider-side prompt caching where supported.

LLM configuration¶

Knob	Value	Where set
Model	`deepseek-v4-flash` (thinking disabled)	`DEFAULT_MODEL` in `agents/bdpd_agent.py`; overridable via `--model`
Temperature	`0.4`	`CFG.get('temperature', 0.4)` in `agents/bdpd_agent.py`
Max tokens	`256`	`CFG.get('max_tokens', 256)`
Max history turns	`32` (rolling window over prior turns)	`CFG.get('max_history', 32)`
Parallel tool calls	disabled (one tool per loop iteration)	SDK `tools.py`
Tool loop cap	`4` iterations per turn before forcing a final harvest	SDK `tools.py`

At temperature = 0.4 two runs on the same seed are not bit-identical: the published variance is cross-seed only, not intra-seed. A campaign-scale extension that needs intra-seed replicability should drop to temperature = 0 and re-run on the same provider checkpoint.

deepseek-v4-flash is a cloud-served checkpoint with no user-pinnable version. Provider updates between the run date of a paper and a replication attempt may shift the underlying model; a follow-up campaign should either record the API checkpoint identifier from the response headers or switch to a locally hosted model for a frozen substrate.

Output: `aggregate.json` structure¶

Each pilot writes one aggregate.json at data/pilot/<name>/aggregate.json summarising the cross-seed distribution of every recorded metric. The structure is consistent across pilots:

{
  "cell_A": {                       // one block per experimental cell
    "turnsSurvived": {
      "n": 5,                       // sample size (number of seeds)
      "mean": 30.0,
      "sd":   0.0,
      "min":  30.0,
      "max":  30.0,
      "values": [30, 30, 30, 30, 30] // per-seed raw values, in seed order
    },
    "finalStock":    { "n": 5, "mean": 26.0, "sd": 8.1, ... },
    "muleWealth":    { "n": 5, "mean": 54.7, "sd": 43.1, ... },
    ...
  },
  "cell_B": { ... },
  "config":  { ... }                // arena config recorded verbatim
}

Every numeric summary follows the { n, mean, sd, min, max, values } shape. The values array preserves per-seed inputs in the order the seeds were swept, so a reader can recompute any statistic independently. The papers sometimes display rounded means and standard deviations; the aggregate.json artefacts always carry the unrounded values for exact cross-check.

Cost envelope¶

Approximate DeepSeek spend per pilot at the run dates of the BDPD¹ mini-pilots, with prompt caching enabled provider-side:

Pilot	Players × seeds × turns	Approx. spend
C1	3 × 5 × ≤ 7	$\sim$ $0.05
D1 (Cell B + Cell C)	6 × 5 × 30, twice	$\sim$ $0.40
D2 (LLM, both cells)	5 × 5 × 30, twice	$\sim$ $0.30
D3 (3 ladders)	5 × 5 × 30, thrice	$\sim$ $0.45

These figures are indicative and depend on the cache-hit ratio realised at run time. A clean replication of the full BDPD¹ pilot suite is comfortably under $2 of DeepSeek credit, modulo provider pricing drift.

Determinism notes¶

The simulation engine is fully deterministic per seed: built-in agents, perturbation triggers, and the logistic/Seneca engines all consume the same seeded PRNG. The non-deterministic component is the LLM call. Two operational consequences:

Built-in (heuristic) cells — for example D1 Cell A — produce identical per-seed trajectories on every run. The recorded $\sigma$ across seeds in these cells is genuine cross-seed variance, not noise.
LLM cells carry an intra-seed noise component on top of the cross-seed variance, both folded into the reported $\sigma$. At $N = 5$ seeds, the published standard deviations should therefore be read as diagnostic of cross-seed spread, not as confidence bounds.

Standard-deviation convention¶

Two standard-deviation conventions coexist in the codebase, by historical accident. Every $\sigma$ reported in the papers comes from the pilot scripts and uses the sample (Bessel) convention.

Convention	Divisor	Where	Feeds
Sample (Bessel)	$n-1$	`scripts/pilot_d1_cell_c.mjs`, `pilot_d1_n5.mjs`, `pilot_d2_ladder_sweep.mjs`, `pilot_d2_llm_n5.mjs` (and the C1/D pilots)	All published $\mu \pm \sigma$ in BDPD¹ (paper_01) and the BDPD² main vignettes V1–V5 (paper_02)
Population	$n$	`experiments/aggregator.js`, `scripts/sweep_v10_seed.mjs`	`experiments/runner.js` sweep outputs (not cited in any paper) and the paper_02 V6 seed-robustness appendix

Impact on published results:

BDPD¹ (paper_01): unaffected — every figure/number comes from the data/pilot/ artefacts produced by the sample-std pilots.
BDPD³ (paper_03): unaffected — the OFAT robustness claims are reported as counts (X/5) with no $\sigma$, and the Seneca+RCP runs are deterministic.
BDPD² (paper_02): main vignettes V1–V5 unaffected (pilots); only the V6 seed-robustness appendix uses the population convention. The discrepancy is $\sqrt{n/(n-1)}$ — about $2.6\%$ at $n = 20$, $\sim 12\%$ at $n = 5$ — and that appendix already carries an aggregator caveat.

This inconsistency is documented rather than unified on purpose: the load-bearing numbers are already on the sample convention, and rewriting the two population-std sites would force a regeneration of the regression baselines (tests/regression/baseline/, whose sweeps run runsPerPoint 10–30) plus a re-render of the paper_02 V6 appendix — for no change to any cited result. If ever unified, standardise on the sample convention through one shared helper (this is review finding SC3, the duplicated meanSd), regenerate the five baselines, and verify the diff touches only std fields, each by the expected $\sqrt{n/(n-1)}$ factor.

Pilot	Players × seeds × turns	Approx. spend
C1	3 × 5 × ≤ 7	\(\sim\) $0.05
D1 (Cell B + Cell C)	6 × 5 × 30, twice	\(\sim\) $0.40
D2 (LLM, both cells)	5 × 5 × 30, twice	\(\sim\) $0.30
D3 (3 ladders)	5 × 5 × 30, thrice	\(\sim\) $0.45

Convention	Divisor	Where	Feeds
Sample (Bessel)	\(n-1\)	`scripts/pilot_d1_cell_c.mjs`, `pilot_d1_n5.mjs`, `pilot_d2_ladder_sweep.mjs`, `pilot_d2_llm_n5.mjs` (and the C1/D pilots)	All published \(\mu \pm \sigma\) in BDPD¹ (paper_01) and the BDPD² main vignettes V1–V5 (paper_02)
Population	\(n\)	`experiments/aggregator.js`, `scripts/sweep_v10_seed.mjs`	`experiments/runner.js` sweep outputs (not cited in any paper) and the paper_02 V6 seed-robustness appendix