Skip to content

Reliable LLM-assisted research production

BDPD was written, analysed, and checked with large language models in the loop at every stage. That is only defensible if the method by which LLMs are used is itself disciplined and reproducible. This page documents that method: the working pipeline, the golden rules that constrain it, the multi-model audit-to-convergence loop, and the two small tools (panel and mimo-mem) that support it.

It is the front door to this section. The per-deliverable verification playbooks are the operational checklists; the prompt library holds the reusable templates that drive each pass.

The pipeline

A BDPD artefact (a paper, a lecture, a blog dispatch) moves through four stages. LLMs participate in all four, but with sharply different roles, and the output of every stage is a checked-in artefact, never a transient chat.

  1. Ideate. Open questions and framing are explored in a multi-model panel: several models discuss a question via the filesystem and hand back a shared deliverable. The convener (the author) keeps editorial control.
  2. Draft. The author drafts against the data and the codebase. Models assist with prose and structure, but every empirical statement must trace to a number that already exists in the experimental record.
  3. Audit. The draft is handed to independent reviewers — different models, different training distributions — each running the same audit prompt and producing a separate, evidence-cited report. See audit to convergence below.
  4. Render & verify. The artefact is rendered to its published form (HTML + PDF) and the rendered output is spot-checked against the source on the points the audit touched.

The discipline is the same at every stage: the source of truth is the code and the data, not the prose — and not the model.

Golden rules

These are general, transferable rules for using a language model as a research collaborator. They are deliberately stated at the level of principle, not project mechanics.

  1. No claim without its primary source. Do not assert — or let a model assert — a referenced fact unless the source itself is in hand and the text actually supports the claim. A bibliography entry is not evidence; the source is. (In BDPD this is operationalised as "no citation without the PDF in the library".)
  2. Re-derive, don't paraphrase. Numbers and derived quantities come from recomputing them off the data or the code, never from lifting them out of the prose under review. The text is the thing being tested; it cannot also be the evidence.
  3. Coherent is not correct. Internal consistency — a number that agrees with another sentence — is not verification. Check against ground truth, not against the document's own claims.
  4. Read the source, not the rendering. Verify against inputs (source files, raw data) rather than downstream artefacts (rendered PDFs, exported tables) that can silently go stale.
  5. "Not verified" beats a false "verified". If you cannot open or recompute the source, say so explicitly. An honest gap is recoverable; a green check on an unchecked item poisons the whole review.
  6. A clean pass is a valid result. Do not manufacture findings to look productive, and actively hunt your own false positives. "Nothing new" is an acceptable — and common — outcome of a good audit.

Audit to convergence

A single review pass, however careful, normalises away its own blind spots. BDPD therefore audits in rounds, each run from zero, by independent reviewers, until a round surfaces no new substantive findings.

  • Independent reviewers. Each pass is run by a different model so that different training distributions catch different errors. The roles and the failure modes each tends to exhibit are documented in reviewer roles.
  • From zero, not as a delta. A later round does not trust the previous round's report. It first re-verifies that each prior fix was actually applied (quoting file:line evidence), then looks only for new or residual findings — and actively hunts the previous round's own mistakes and over-cautious flags.
  • Consolidate, then converge. After all reviewers finish an artefact, the author consolidates the reports into one ledger (agreed by all / agreed by some / disputed), applies the surviving fixes, and opens the next round. Convergence is reached when a fresh round finds nothing load-bearing.
  • Escalate the hard cases. Findings a cheap pass cannot close are pushed to a deeper pass (the same model at maximum effort) on a dedicated escalation file, rather than being left as a false "verified". See the verification README.

The full mechanics — Mode A vs Mode B, the three-pass table, the escalation format, the per-round scope — live in the verification README.

The tools

Two dependency-light tools support the method. Both treat the filesystem as the substrate, which keeps them inspectable and crash-safe.

  • panel — a chamber-style workspace where several models discuss one question by appending to a shared transcript on disk, then distil a single deliverable. Used in the ideate stage and for multi-model bug hunts. No chat server: a session is a directory, the WebUI is a read-only viewer.
  • mimo-mem — a read-only browser over the markdown memory stores the assisting models accumulate. It exists to review the durable memory: surfacing, for example, a name promoted into a notes file next to the real identifier in the codebase it was conflated with. Honest memory is part of reliable production.

Why this is in the public docs

Publishing the method — the rules, the prompts, the tools — is part of the claim. A reader who wants to trust an LLM-assisted result needs to see how the assistance was constrained. The prompts in the library are the literal templates used on this project; they are offered for reuse and for scrutiny.