# Writer bench — artifact bundle

Everything needed to check the results published at
https://labs.pollar.news/bench, as promised by its reporting rules.

## Runs

Each run ships as `<tag>.json` (raw writer outputs per fixture; per-judge
verdicts where persisted) and `<tag>.md` (the full generated report with
metric columns and adjudication lines).

| Tag | What it is |
|---|---|
| bench-v04-first-run | Shakedown: rubric v1, draft trap scoring. Not the published numbers. |
| bench-v05-prod / -fable / -sol | Published run 1 (three writers, closed production vocabulary, rubric v2). |
| bench-2x2-prod-open | 2x2 completion: production model under the open chart vocabulary. |
| bench-kimi-k3 | Run 2 (July 23): a third frontier writer, Kimi K3 (Moonshot), under the open chart vocabulary, all 13 fixtures. |

## Scoring inputs

- `*.gold.json` — gold sets per fixture: dataset inventories (extraction
  recall), key-fact probes (coverage), trap checks (requireAny / forbid with
  assertion-surface scoping).
- `trap-*.json` — the four synthetic trap fixtures in full, articles
  included: their sources and events are invented, so publishing them
  redistributes nothing. Never treat their content as real news.
- `fixtures-provenance.md` — the six real-event fixtures: event ids and the
  exact articles (source, title, URL, date) each writer received. Article
  text is third-party copyrighted material and is not redistributed.
- `rubric-formfit-v2.txt` — the verbatim judge rubric (hardened version used
  in the published runs; v04 used a lenient v1, since superseded).
- `cost-ledger-bench.jsonl` — one line per LLM call in these runs: model,
  tokens, cost, latency, provider.
- `adjudication-log.md` — every human adjudication decision with reasoning,
  including the two scoring-draft errors that were overturned and the
  external-audit erratum.

## Known gaps, stated rather than hidden

- Per-chart, per-judge scores for the three bench-v05 runs were not
  persisted to JSON (the harness gained persistence the same day); their
  judge outcomes survive as the form-fit column and per-judge comments in
  the `.md` reports. bench-2x2-prod-open carries full per-judge verdicts.
- Gold sets exist for 10 of 13 fixtures; the three production-snapshot
  fixtures (iran-ceasefire-collapse, france-morocco-worldcup,
  volkswagen-model-cuts) have none yet, so recall/coverage columns are blank
  for them.
- The adjudicator is a single reviewer identified as Pollar Research.