the-daily-epub/docs/plans/curation-v2-briefs/step6.md

---

YOUR STEP: 6 — Paper telemetry, stats, lock (plan §21 step 6)

Steps 1–5 have landed: the full v2 ranking pipeline (hygiene → embeddings → signals → triage → admission → deep assessment → utility → cluster-capped shortlist → Claude editor → editorial), candidate_runs telemetry, explain, features, ratings. Read git log -10, then src/pipeline.rs, src/report.rs, src/main.rs, src/curate/telemetry.rs, src/epub/chapters.rs and src/epub/templates/, src/types.rs (Colophon), src/db.rs, README, config.example.toml.

Scope (plan §5 lock, §15.1 Behind the paper + In-this-issue, §15.3 stats, §15.4 report/log block, §16 lock coverage, §19 startup logging, README/config docs):

1. Behind the paper (§15.1): a new short chapter after the World Briefing and before the colophon (src/epub/templates/behind.xhtml, render_behind_the_paper) with exactly the content shape in the plan: the considered/eligible/triaged/read-closely/shortlisted/selected line; the Admitted via: line; the Learned signals: line (rated-with-embeddings count and knn percentage, feed affinity state); Near misses — the 10 highest-utility non-selected articles as <title> — <feed> · quality X · fit Y · <stage/reason>; the Models: line with triage/assessment, editor/summaries and embeddings models; cost and generation time. Both editions (X4 gets the same text, no links). The data comes from the run's RunReport/StageCounts and the run's candidate_runs rows (add a BehindThePaper struct to types.rs filled by the pipeline; keep templates pure). Chapter ids stable (behind).
2. In this issue page: verify each entry shows the why line under the summary (step 2 was asked to do this; complete it if missing). The colophon keeps its fields and shows per-provider cost lines and models (verify; complete if missing).
3. stats (§15.3): daily-epub stats [--days 14] printing issues, articles published, explicit ratings by label, ratings per issue, up/down ratio per admitting retriever (admitted_by[0] of rated picks, from candidate_runs joined to rating_events via the latest explicit event), exploration yield (exploration picks selected / admitted, and how many were rated positively), mean issue size, cost per day per provider (from runs.provider_costs_json), mean generation time. Plain text, one fact per line, no tables wider than 80 columns.
4. Run report and log block (§15.4): make sure StageCounts has every field listed — eligible, embedded, triaged, admitted, admitted_by (map), assessed, shortlisted, clusters, exploration_admitted, exploration_selected, verdicts_in_prompt, rated_with_embeddings, per-provider usage — and that stage timings include embed, signals, triage, admit, assess, rank, editor, summaries, brief. Emit the four-line info block from §15.4 once per run (curation:, admission:, preference:, providers:). print_report in main.rs prints the same lines.
5. src/lock.rs (§5): flock(LOCK_EX | LOCK_NB) on <database_path>.lock, taken in main for generate, profile rebuild, features backfill, and backfill-social; a second invocation exits with generate is already running (name the command that holds it if cheap, else the generic message). serve, explain, stats, ratings, db migrate, features prune do not take it. Use libc/rustix only if already in the dependency tree (check Cargo.lock); otherwise std::fs::File + the fs2-free approach via libc::flock is acceptable to add as a tiny dep — say which in the report. Twenty lines, no table, no TTL.
6. Docs: README rewritten where it describes the pipeline, feedback links, costs, prerequisites (Anthropic and Voyage keys, dashboard spend limits as the real backstop, the server-side fallback note), the CLI (ratings, explain, stats, features, --rescore, --skip-embeddings, --max-articles as ceiling), and data/profile.md. config.example.toml carries every key from plan §19 with comments. Startup logs resolved models and per-provider enabled state (verify).
7. systemd/daily-epub-generate.service: no change needed unless an env var name changed; note the two new env vars in the README's systemd section.

Tests (§20 "Lock", "Pipeline" behind-the-paper, stats): two processes/one wins (spawn the binary or use two File handles from separate threads with try_lock semantics — no sleeping longer than a second), a killed holder frees the lock, serve does not take it (parse-level or by checking no lock file is created); the behind-the-paper chapter renders with the counts and near misses on the mocked pipeline run and is parseable XHTML in both editions (extend tests/m4_epub.rs); stats runs on a temp DB with a couple of runs, issues and rating events and prints every line; the info block appears in the log (capture with tracing test subscriber or assert on the formatted strings).