Experiment Runner Scripts¶

BDPD provides four shell scripts under experiments/ that orchestrate the execution of experiments, figure generation, and statistical analysis with fine-grained control over which steps to execute.

All scripts run from the experiments/ directory. Ensure the BDPD Python virtual environment is active before invocation.

`run_all.sh` — Unified Experiment & Figure Runner¶

Runs all experiments (P1–P11, B1–B5, Seneca ODE) and generates figures in a single pipeline.

Quick Start¶

cd experiments
./run_all.sh                           # full pipeline (experiments + figures)
./run_all.sh --plots-only              # regenerate figures from existing JSON
./run_all.sh --experiments-only        # run experiments, skip figure generation
./run_all.sh --fast                    # halve runs per cell (smoke test)
./run_all.sh --dry-run                 # print commands without executing
./run_all.sh --list                    # list all experiments and exit

Selective Execution¶

# Specific experiments
./run_all.sh --only P1 P2 P7          # platform experiments
./run_all.sh --only B1 B3 B3f         # supplement experiments
./run_all.sh --only S1                # Seneca ODE step

# By group
./run_all.sh --group platform         # all P-sweeps
./run_all.sh --group supplement       # all B-sweeps
./run_all.sh --group seneca           # all Seneca ODE figures

Experiment Keys¶

Key	Label	Runs/pt	Group
P1	Aggressive fraction sweep	20	platform
P2	Regen rate sweep	20	platform
P3	Pool size scaling	20	platform
P4	Pool size fine	20	platform
P5	2D heatmap regen × aggr	15	platform
P6	Asymmetry sweep	20	platform
P7	Scheduler comparison	30	platform
P8	Adaptive vs conservative	20	platform
P9	Observability noise	20	platform
P10	Perturbation regen shock	20	platform
P11	Mule strategy override	20	platform
B1	Aggressor intensity (coarse)	30	supplement
B1f	Aggressor intensity (fine)	5	supplement
B2	Adaptive reduction sweep	20	supplement
B3	Regen rate extended (coarse)	20	supplement
B3f	Regen rate extended (fine)	5	supplement
B4	Pure Seneca shock	20	supplement
S1	Bardi canonical	—	seneca
S2	k2 sweep	—	seneca
S3	Regime map	—	seneca
S4	Logistic limit	—	seneca

Pipeline Steps¶

Experiment execution — node experiments/experiment.js run <name> for each selected sweep. Results stored as JSON in experiments/results/.
Figure generation — python3 experiments/sweep.py and sweep_supplement.py read result JSONs, produce PNGs in experiments/figures/platform/.
Seneca ODE figures — python3 experiments/seneca_validate.py for S1–S4. Output to experiments/figures/seneca/.
Statistical inference (B5) — python3 experiments/statistical_inference.py --results-dir experiments/results_paper/platform --seed 42 runs logistic regression, bootstrap CIs (B1/B3 prefer fine sweeps), and permutation tests. Writes experiments/results/B5_statistical_report.md.

# Skip statistical inference (faster, for iteration)
./run_all.sh --no-stats

Output¶

Path	Contents
`experiments/results/`	Raw JSON results, one file per sweep
`experiments/results/B5_statistical_report.md`	Statistical inference report
`experiments/figures/platform/`	P-sweep and B-sweep PNGs (3 themes × 300 DPI)
`experiments/figures/seneca/`	Seneca ODE figures

`run_experiments_supplement.sh` — Supplement Sweep Runner¶

Runs only the four supplement experiments (B1–B4) and their figures. This is a focused variant of run_all.sh for the paper revision.

Quick Start¶

cd experiments
./run_experiments_supplement.sh                      # all four (B1–B4)
./run_experiments_supplement.sh --only B1 B2         # selective
./run_experiments_supplement.sh --only B4
./run_experiments_supplement.sh --fast               # halve runs (smoke test)
./run_experiments_supplement.sh --dry-run             # print, don't execute
./run_experiments_supplement.sh --list               # show experiments
./run_experiments_supplement.sh --plots-only         # re-plot existing results

Pre-Flight Patch¶

Before running, the script auto-applies a patch that forwards strategyParams (intensity, reductionFactor) to built-in agents. Required for B1 and B2; harmless for B3 and B4.

# Skip the auto-patch if already applied
./run_experiments_supplement.sh --skip-patch

Experiments¶

Key	Definition	Runs/pt
B1	`aggressive_intensity_sweep`	30
B2	`adaptive_reduction_sweep`	20
B3	`regen_rate_extended`	20
B4	`pure_seneca_shock`	20

Note: the fine-grained sweeps (aggressive_intensity_sweep_fine, regen_rate_extended_fine) are not included in this script. Run them manually:

node experiments/experiment.js run aggressive_intensity_sweep_fine --runs 5 --out experiments/results/
node experiments/experiment.js run regen_rate_extended_fine --runs 5 --out experiments/results/

`run_cards_ge_sweeps.sh` — Card Game Heuristic Sweep Runner¶

Runs the five card tournament sweeps (CT1–CT5) using the heuristic engine (no LLM). All games are deterministic given a seed.

Quick Start¶

cd experiments
./run_cards_ge_sweeps.sh               # run CT1–CT5 (200 games/pt)
./run_cards_ge_sweeps.sh --only-images # regenerate figures from existing results

Sweeps Executed¶

Sweep	Games/pt	Rounds	Notes
CT1	200	9	4×4 archetype collapse matrix
CT2	200	15	Game-length sweep
CT3	200	12	Observability noise (0%–hidden)
CT4	200	12	Defection timing T=3–11
CT5	200	12	Stranger-King deep dive (CT5a/b/c)

Configuration¶

Hardcoded defaults (edit the script to change): - Forest Deck: 20 cedars - Box Reserve: 10 cedars - Collapse threshold: 3 (Forest Die)

Output¶

Path	Contents
`experiments/cards_sweep_demo_results/`	Result JSONs + PNG figures (3 themes)
`experiments/cards_sweep_demo_logs/`	Sweep logs

`run_cards_llm_sweeps.sh` — Card Game LLM Sweep Orchestrator¶

Runs card tournament sweeps with LLM agents via an OpenAI-compatible API or a local llama-server. Supports phased execution with verification before production.

Quick Start¶

cd experiments

# Phase 1: Verify prompt quality (1 game/pt, --nudge active)
./run_cards_llm_sweeps.sh verify

# Phase 2: Production sweeps (20 games/pt)
./run_cards_llm_sweeps.sh all

# Individual sweeps
./run_cards_llm_sweeps.sh ct1     # CT1 collapse matrix
./run_cards_llm_sweeps.sh ct3     # CT3 noise sweep
./run_cards_llm_sweeps.sh ct4     # CT4 defection timing
./run_cards_llm_sweeps.sh ct5     # CT5 complete (CT5a+CT5b+CT5c)

# DeepSeek-specific variants (require DEEPSEEK_API_KEY)
./run_cards_llm_sweeps.sh deepseek_flash
./run_cards_llm_sweeps.sh deepseek_pro

# Regenerate figures from existing JSON
./run_cards_llm_sweeps.sh ct4 --only-images
./run_cards_llm_sweeps.sh --only-images all

Configuration¶

Edit the top of the script to configure:

Variable	Default	Description
`USE_API`	`true`	Use external API instead of local llama-server
`API_MODEL`	`deepseek-v4-flash`	Model identifier
`API_BASE_URL`	`https://api.deepseek.com`	API endpoint
`API_KEY_ENV`	`BDPD_API_KEY`	Environment variable holding the key
`ENABLE_THINKING`	`false`	Enable extended thinking (DeepSeek Pro)
`REASONING_EFFORT`	`""`	Reasoning effort: `low`, `medium`, `high`
`INTER_CALL_DELAY_MS`	`100`	Pause between API calls (rate limiting)
`PROMPT_LOG`	`""`	Path to JSONL prompt/response log file

API Key¶

Set the key before running:

export BDPD_API_KEY="sk-your-key-here"
./run_cards_llm_sweeps.sh all

Local Model Mode¶

To use a local llama-server instead of an API:

# Edit the script: set USE_API=false
# Start two llama-server instances
llama-server --model /path/to/model.gguf --port 8081 --ctx-size 4096 &
llama-server --model /path/to/model.gguf --port 8082 --ctx-size 4096 &

./run_cards_llm_sweeps.sh verify

Output¶

Path	Contents
`experiments/llm_sweep_results/<sweep>/`	Result JSONs + PNG figures
`experiments/llm_sweep_logs/<sweep>_<ts>.log`	Full sweep logs
`experiments/llm_sweep_logs/prompts.jsonl`	Prompt/response traces (if `PROMPT_LOG` set)

Requirements¶

All scripts require:

Tool	Version	Check
Node.js	≥ 20	`node --version`
Python 3	≥ 3.10	`python3 --version`
matplotlib	any recent	`python3 -c "import matplotlib"`
numpy	any recent	`python3 -c "import numpy"`

Additional for statistical inference and LLM sweeps: - statsmodels, pandas (B5 statistical inference) - openai (LLM API sweeps) - llama-server or API key (LLM sweeps)

Experiment Runner Scripts¶

run_all.sh — Unified Experiment & Figure Runner¶

Quick Start¶

Selective Execution¶

Experiment Keys¶

Pipeline Steps¶

Output¶

run_experiments_supplement.sh — Supplement Sweep Runner¶

Quick Start¶

Pre-Flight Patch¶

Experiments¶

run_cards_ge_sweeps.sh — Card Game Heuristic Sweep Runner¶

Quick Start¶

Sweeps Executed¶

Configuration¶

Output¶

run_cards_llm_sweeps.sh — Card Game LLM Sweep Orchestrator¶

Quick Start¶

Configuration¶

API Key¶

Local Model Mode¶

Output¶

Requirements¶

`run_all.sh` — Unified Experiment & Figure Runner¶

`run_experiments_supplement.sh` — Supplement Sweep Runner¶

`run_cards_ge_sweeps.sh` — Card Game Heuristic Sweep Runner¶

`run_cards_llm_sweeps.sh` — Card Game LLM Sweep Orchestrator¶