Goal. In counterparty due diligence a person signs the output, so “looks plausible” is not good enough: a confident answer that cites a source it does not support, or that quietly misses a sanctions hit, is worse than no answer at all. dd‑evals is a small, transparent harness that grades an LLM on exactly those failure modes, grounding, honesty about what is absent, and never being wrong on a high‑stakes yes/no, behind a judge that must earn trust before its scores count.
A frozen, entirely fictional 16-document corpus: 4 companies × 4 document types. Every file is stamped “fictional entity, for testing only,” which is what makes the planted honesty traps possible.
The system under test retrieves the relevant documents per question, then must answer only from the cited sources.
Each field is scored by one of five gates with a stakes-tiered pass bar. One gate failing fails the whole run.
| Gate | What it checks | Pass | Rate | Verdict |
|---|---|---|---|---|
| G1 | Deterministic field accuracy | 11/11 | pass | |
| G2 | Faithfulness to cited source | 6/7 | fail | |
| G3 | Honesty about absent fields | 2/2 | pass | |
| G4 | High-stakes yes/no (sanctions) | 3/3 | pass | |
| G5 | Answer relevancy | 2/2 | pass |
Overall verdict: fail, and that is the point. One faithful-looking fabrication is exactly the error that ends a candidacy when a person signs the file.
Field q13 · Northwind Restoration Ltd · faithfulness gate (G2) · source: northwind_restoration/press_release.md
Q: What certification does Northwind claim in its press release, and what body awarded it?
The faithfulness judge decomposed the answer into 4 atomic claims, found 2 unsupported, and scored it 0.50 against a 0.98 bar, failing G2 and the overall run. No human had to notice; the harness did.
| Judge vs. human agreement | 10 / 10 = 1.00 |
| Calibration gate (≥ 0.85) | passed |
| Grader self-test (seeded mutations caught) | 3 / 3 |
| Confusion vs. ground truth (false accepts / rejects) | 0 / 0 |
| Replay reproducibility (no API key) | byte-identical |
The judge is pinned and calibrated against independent human labels before its scores count; a mutation check proves the scorer flips the right gate when a known-bad answer is injected.
Every committed artifact has a committed command that regenerates it, listed in results/RUN_PROVENANCE.md.
pip install -e ".[dev]"
python run_eval.py --mode replay --out-dir results
python make_report_plots.py results
python src/mutation_check.py
python -m pytest -q
Synthetic data only. Not a production KYC product. Calibration for this run used a Sonnet-class stand-in judge under the pinned rubrics (no API key required).