Curation v2: progress handoff and per-step agent briefs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe

docs/plans/2026-09-02-curation-v2-progress.md

@@ -0,0 +1,90 @@
+# Personalized Curation v2 — implementation progress and handoff
+
+**Updated:** 2026-09-02 (end of session 1)
+**Plan:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§21 is the step sequence)
+**Branch:** `curation-v2` (branched from `main` at `a599745`; not merged, not pushed)
+**Briefs:** `docs/plans/curation-v2-briefs/` — `00-preamble.md` + one `stepN.md` per step. Each
+brief was handed to an implementation agent as `cat 00-preamble.md stepN.md`.
+
+## Where things stand
+
+| Step (§21) | Status | Commit |
+|---|---|---|
+| 1. Feedback and profile | **done**, reviewed (codex review: no actionable issues) | `3a9f4b9` |
+| 2. Claude editor and editorial | **done** | `57efbb4` |
+| 3. Embeddings, signals, telemetry | **done** (branch `curation-v2-step3`, merged) | `ea3b141` + merge commit |
+| 4. Triage replaces the gate | not started — brief ready: `step4.md` | |
+| 5. Deep assessment, utility, diversity | not started — brief ready: `step5.md` | |
+| 6. Paper telemetry, stats, lock | not started — brief ready: `step6.md` | |
+| 7. Cleanup + implementation notes | not started — brief ready: `step7.md` | |
+
+`cargo fmt --check`, `cargo clippy --all-targets` (only the three pre-existing `src/world.rs`
+`needless_borrow` warnings) and `cargo test` (294 lib tests + 6 integration suites) are green at HEAD.
+
+Nothing has been run against the production database or the real providers. No local
+`config.toml`, database, or API keys exist on the dev box, so verification so far is tests only.
+
+## Decisions and deviations made while implementing (read before step 4)
+
+- **`scores` table not yet dropped.** Migration `0002_curation_v2.sql` created every new table, copied
+  `ratings` → `rating_events`, and dropped `ratings` and `feed_priors`, but kept `scores` because the
+  old Stage A scoring (`src/curate/score.rs`, `db::upsert_score`, `recently_low_scored_ids`) still
+  uses it. Step 4 adds `migrations/0003_drop_scores.sql` and moves the churn rule to
+  `article_assessments` (already in `step4.md`).
+- **Old prefilter still gates.** Hygiene → embeddings → signals → preliminary blend now run for every
+  article, and `candidate_runs` rows are written with the stage vocabulary mapped onto the old flow
+  (`admitted_by` is `["prefilter"]`/`["auto"]` for now), but `prefilter::run` + `prefilter_keep`
+  still decide the deep set until step 4.
+- **Legacy `up` links.** `Vote::parse("up")` → `Loved`, and `auth::verify_token` also accepts tokens
+  signed over the literal `up` segment so already-published issues keep working.
+- **Same-date regeneration** no longer excludes its own picks (`published_before` uses issue dates
+  strictly before the run date), per plan §8.1.
+- **`VoyageConfig.api_key`** exists as a field (figment maps `DAILY_EPUB_VOYAGE__API_KEY` into it;
+  `deny_unknown_fields` would otherwise reject the env var). Never document it in TOML.
+- **`anthropic.max_concurrent_requests`** is validated but not consumed yet; summary concurrency is
+  the constant `SUMMARY_CONCURRENCY = 4` in `editorial.rs`.
+- `UsageMeter::new(&DeepseekConfig, ..)` survives as a compat constructor over `with_prices`.
+- `tests/fixtures/deepseek_front_page.json` was replaced by `tests/fixtures/claude_brief.json`.
+- `RATINGS_LOOKBACK_DAYS` in `profile/mod.rs` is effectively unbounded (36,500) for the prompt
+  verdict block and the weekly rebuild; the knn/feed preference state uses
+  `curation.ranking.rating_lookback_days` (180) as the plan says.
+- Footer CSS: `.rating` has no `white-space: nowrap` (it would clip on narrow e-ink screens).
+
+## Operator to-dos before the first real run
+
+1. Set `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` in the systemd env file.
+2. Set hard spend limits in the DeepSeek, Anthropic and Voyage dashboards (the meters are runaway
+   guards, not accounting).
+3. Copy `data/profile.md` to wherever `profile_path` points on the server (default is relative to
+   `WorkingDirectory=/var/lib/daily-epub`, like `data/scour-interests.opml`).
+4. Run `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`; back up the DB first).
+5. `daily-epub features backfill --rated-only` then `--days 30` to warm the embedding cache.
+6. A `generate --dry-run` and read the paper; `explain --near-misses` once step 4 lands.
+
+## How the work was run (so the next session can repeat it)
+
+- Orchestrator: Claude Code (this repo), one implementation agent per step, review + commit by the
+  orchestrator after independent `cargo fmt/clippy/test`.
+- Codex: `codex exec -C <repo> --sandbox workspace-write --add-dir ~/.cargo -c
+  sandbox_workspace_write.network_access=true -c model_reasoning_effort=high -o <last.md> - < <brief>`,
+  detached with `setsid nohup`, exit code written to a file and watched with a monitor. The
+  `openai-codex` Claude Code plugin's `task` runs fail on this host (bubblewrap cannot create user
+  namespaces: `kernel.apparmor_restrict_unprivileged_userns = 1`); the CLI works if the brief tells
+  the agent to edit files via shell commands instead of the `apply_patch` tool (see the preamble).
+  The plugin's read-only `review --background --scope working-tree` does work and was used on step 1.
+  Fix for the sandbox: `sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0`.
+- Codex ran out of ChatGPT usage after ~560k tokens (step 1 complete, steps 2 and 3 cut off
+  mid-way); Claude `general-purpose` sub-agents finished steps 2 and 3 from the partial trees and
+  resolved the step 2/3 merge. Budget roughly one large step per Codex usage window.
+- Steps 2 and 3 were run in parallel (step 3 in a git worktree); the merge cost ~19 conflict hunks
+  in config/pipeline/report/main/README/config.example. Steps 4 → 5 → 6 → 7 are sequential.
+
+## Next session: exact starting instructions
+
+1. `git checkout curation-v2 && cargo test` (expect green).
+2. Read this file, then `docs/plans/curation-v2-briefs/00-preamble.md` and `step4.md`.
+3. Launch the step 4 agent with `cat 00-preamble.md step4.md` as the prompt (Codex CLI as above, or a
+   Claude general-purpose agent — tell it to ignore the "host quirk" paragraph in that case).
+4. Review the diff against plan §10–§11, run the checks, commit as "Curation v2 step 4: …".
+5. Repeat for steps 5, 6, 7. After step 7: `git merge --no-ff curation-v2` into `main`, deploy, and
+   do the operator to-dos above.

docs/plans/curation-v2-briefs/00-preamble.md

@@ -0,0 +1,52 @@
+# Implementation agent brief — The Daily EPUB, "Personalized Curation v2"
+
+You are implementing ONE step of a multi-step plan in the Rust repository at
+`/home/thallada/workspace/the-daily-epub` (branch `curation-v2`, already checked out).
+
+The orchestrator hands you `docs/plans/curation-v2-briefs/00-preamble.md` (this file) followed by one
+`stepN.md` from the same directory; `docs/plans/2026-09-02-curation-v2-progress.md` records what has
+landed so far.
+
+The plan is `docs/plans/2026-09-02-personalized-curation-v2.md`. Read it in full before
+writing code: §0 records settled decisions (do not reopen them), §1 lists the files to
+read first, §21 is the implementation sequence. Also read
+`docs/plans/2026-08-15-implementation-notes.md` §"Cross-cutting implementation decisions"
+(note: item 5 is stale — the code uses a hand-rolled `reqwest` client in `src/curate/llm.rs`,
+keep that).
+
+## Hard rules
+
+- Implement ONLY the step assigned below. Later steps land separately; do not start them.
+  Where this step needs a type or table that a later step fills in, create it now exactly as
+  the plan specifies and leave it empty/unused with a short `// filled in by step N` comment.
+- Conventions: sqlx *runtime* queries (`sqlx::query(...).bind(...)`, never the `query!` macros),
+  `jiff` for time, RFC3339 UTC strings in SQLite, `thiserror`/`anyhow` error style, no `unwrap()`
+  outside tests, tracing spans per stage, rustfmt defaults, askama templates.
+- Tests never touch the network. Use mock backends following `MockBackend` in `src/curate/llm.rs`.
+- Follow the plan's names for modules, functions, config keys, table and column names, enum
+  strings and prompt constants exactly. If the plan is internally contradictory or impossible
+  for this step, pick the closest behaviour, keep the paper publishable, and describe the
+  deviation in your final report. Do not invent scope beyond the plan.
+- API keys only from env vars; never in config files, logs, tests or the database.
+- Prefer existing dependencies in `Cargo.toml`. Add a new crate only if there is no reasonable
+  way without it, and say so in the report.
+- Do NOT commit and do NOT create branches. Leave all changes in the working tree; the
+  orchestrator reviews and commits. Do not edit anything under `docs/`.
+- Keep `config.example.toml` and README in sync with any config keys you add or remove
+  (the test `shipped_example_config_parses` must pass).
+- Before finishing, run and make pass: `cargo fmt`, `cargo clippy --all-targets` (no new
+  warnings), `cargo test`. Fix what you broke; do not delete or `#[ignore]` tests to get green
+  unless the plan removes the feature they cover (then move/replace the tests as §18 says).
+
+## Final report
+
+End with a concise report (this is what the orchestrator reads): what you implemented, every
+deviation from the plan and why, anything from the step you could not finish, and the exact
+`cargo test` summary line(s). Keep it under 60 lines.
+
+## Host quirk (important)
+
+On this machine the built-in `apply_patch` tool's filesystem helper fails with
+`bwrap: ... Operation not permitted`. Do not keep retrying it. Edit files through shell commands
+instead (the `apply_patch` CLI invoked from bash works, as do `python3 - <<'EOF'` rewrite scripts,
+`sed -i`, and heredocs). Shell commands, `cargo build`, `cargo test` and `cargo clippy` all work.

docs/plans/curation-v2-briefs/step1.md

@@ -0,0 +1,73 @@
+
+---
+
+# YOUR STEP: 1 — Feedback and profile (plan §21 step 1)
+
+Scope (plan §6, §7, §8; tests in §20 under Vote, Rating events, Migration, and the prompt):
+
+1. **Migration `migrations/0002_curation_v2.sql`** (§7). Create `rating_events`,
+   `article_embeddings`, `interest_embeddings`, `article_assessments`, `candidate_runs` with
+   the exact columns/indexes in §7; add `runs.config_json`, `runs.provider_costs_json`,
+   `issue_articles.why`. Copy `ratings` rows into `rating_events` per §6.2 (`vote=1 → 'loved',1.0`;
+   `vote=-1 → 'not_for_me',-1.0`; `source='migration'`; `kind='explicit'`; `event_at=rated_at`),
+   then `DROP TABLE ratings; DROP TABLE feed_priors;`.
+   **Orchestrator decision:** do NOT drop `scores` in this step. Stage A scoring
+   (`db::upsert_score`, `recently_low_scored_ids`) still writes/reads it until step 4 replaces
+   it with `article_assessments`; step 4 will add a `0003` migration dropping it. Do not edit
+   `0001_init.sql`. Migration is plain SQL, no Rust bootstrap.
+2. **Three-way `Vote`** (§6.1) in `src/types.rs`: `Loved | Good | NotForMe`, `as_str` →
+   `"loved" | "good" | "down"`, `parse` also accepts legacy `"up"` → `Loved`, `value(&FeedbackConfig)`
+   → configured 1.0 / 0.35 / −1.0. Add `[curation.feedback]` config
+   (`loved_value`, `good_value`, `not_for_me_value`, `verdicts_in_prompt = 60`). Remove `Vote::as_i64`,
+   `Rating`, `FeedPrior`, and everything that depended on `ratings`/`feed_priors`
+   (`db::upsert_rating`, `ratings_with_feed`, `upsert_feed_prior`, `feed_priors`,
+   `profile::rebuild_feed_priors`, the feed-prior term in `prefilter.rs` and `ScoredArticle.feed_prior`
+   — set the prefilter's feed-prior contribution to nothing; the field can go). HMAC links
+   (`src/auth.rs`) must verify for all three votes; message stays `{issue_date}/{article_id}/{vote}`.
+3. **Footer + confirmation page** (§6.1): `RatingLinks` in `src/epub/chapters.rs` and
+   `src/epub/templates/chapter.xhtml` render three links on one line sized for e-ink:
+   `Was this a good pick?   [ Loved it ]   [ Good ]   [ Not for me ]      Read online ↗`.
+   X4 edition still renders no links. `server::handle_rating` appends one `rating_events` row
+   (`kind='explicit'`, `source='epub'`, label `loved|good|not_for_me`, value from config) and
+   returns "Recorded: Loved it — thanks." (etc.) plus the other two links so a mis-tap can be
+   corrected. No feed-prior rebuild, no provider call.
+4. **`rating_events` access** (§6.2) in `src/db.rs`: `append_rating_event(...)`,
+   `db::current_ratings(lookback_days) -> Vec<RatedArticle>` implementing the latest-explicit-event
+   rule (`ORDER BY event_at DESC, id DESC`, `cleared` removes the article from the learned set),
+   joining `articles`, the best entry's feed title, and the most recent `issue_articles.summary`.
+   Move `RatedArticle` to `src/types.rs` with `summary: Option<String>`, `facets: Option<Facets>`
+   (define `Facets` per §12.1 now, all `Option`s, unused until step 5), `note: Option<String>`,
+   plus `RatingEvent`. Also define `RatedArticle.value: f64` and `label`.
+5. **Ratings CLI** (§6.3) in `src/main.rs`: `ratings list [--days 90] [--label loved|good|down|cleared]`,
+   `ratings set --article ID|--url URL --label loved|good|down [--note "..."]`,
+   `ratings clear --article ID|--url URL`. `set`/`clear` append rows with `source='cli'` and
+   `issue_date` from the latest `issue_articles` row for the article if any; `--url` canonicalizes
+   with `dedupe`'s canonical URL function then `db::article_id_for_url`. Print something useful.
+6. **`data/profile.md`** (§8.2): create the file with the initial content shown in §8.2 verbatim;
+   add `profile_path` config (default `data/profile.md`); loader that parses any `## Interests`
+   section (one per line, leading `- ` stripped) and unions it case-insensitively with the OPML
+   interests, passing everything else through verbatim. Remove `STATED_PREFERENCES` and the
+   reader/wants/does-not-want/how-to-judge parts of `PROFILE_PREAMBLE` from code (they now live
+   in the file); the editor-in-chief framing paragraph stays in code. If the file is missing,
+   warn and continue with the OPML interests only.
+7. **System prompt** (§8.4): rebuilt every run in this exact order: framing → profile.md verbatim
+   (minus Interests) → standing interests grouped by `profile::themes::group_into_themes` →
+   learned adjustments → **recent verdicts**: up to `verdicts_in_prompt` most recent explicit
+   ratings, newest first, `LOVED | title | feed | one-line summary` (labels `LOVED`/`GOOD`/`NOT FOR ME`),
+   cleared and duplicate articles removed. Byte-identical within a run. `profile_version` bumps only
+   on the weekly rebuild, as today. `kv` keeps `ingest_watermark`, `taste_profile`,
+   `taste_profile_learned`, `profile_version` (§7.8).
+8. **Weekly rebuild** (§8.3): still on the DeepSeek client this step (the Claude editor client
+   arrives in step 2). Each rated line carries
+   `LOVED | title | feed | summary | facets: format/depth/evidence/technicality/topic_group | note: …`
+   (omit empty parts), up to 200 most recent explicit ratings, cleared excluded; the softened
+   "strong prior, not a rule" instruction; the required "Diversity check:" bullet.
+9. Pipeline: remove the `rebuild_feed_priors` call and every `feed_priors` reference; the paper
+   must still build. `handle_rating` and `profile rebuild` must not touch dropped tables.
+
+Tests to add/adapt (§20): Vote parse/serialize incl. `up`→Loved; HMAC for all three; template
+renders three links standard / none X4; latest explicit event wins; `cleared` removes from the
+learned set; migration on a temp DB through 0001 then 0002 copies old ratings with the right
+labels/values and `ratings`/`feed_priors` are gone; CLI `set`/`clear` append `source='cli'` rows
+(test the db-level function); profile.md Interests parsing and union; system prompt section order
+and verdict block content. Update `tests/m7_server.rs` and `tests/m4_epub.rs` as needed.

docs/plans/curation-v2-briefs/step2.md

@@ -0,0 +1,86 @@
+
+---
+
+# YOUR STEP: 2 — Claude editor and editorial (plan §21 step 2)
+
+Step 1 has landed (three-way votes, `rating_events`, `data/profile.md`, the rebuilt system
+prompt). Read `git log -3` and the current `src/curate/llm.rs`, `select.rs`, `editorial.rs`,
+`pipeline.rs`, `config.rs`, `report.rs`, `db.rs`, `src/epub/chapters.rs` + templates before coding.
+
+Scope (plan §4.1, §4.2, §5, §7.6, §13, §14, §15.1 chapter/in-this-issue `why`, §18 types, §19 config):
+
+1. **`AnthropicBackend`** in `src/curate/llm.rs` implementing `ChatBackend` exactly per §4.2:
+   `POST {base_url}/v1/messages`; headers `x-api-key`, `anthropic-version: 2023-06-01`,
+   `content-type: application/json`, `anthropic-beta: server-side-fallback-2026-07-01`;
+   body `{model, max_tokens: 16000, system: [{type:"text", text, cache_control:{type:"ephemeral"}}],
+   messages:[{role:"user", content}], output_config:{effort}, fallbacks:"default"}`.
+   **Never** send `temperature`, `top_p`, `top_k`, `thinking`, or an assistant prefill. Ask for JSON
+   in the prompt text and parse tolerantly (reuse `strip_code_fence` + the existing tolerant parsing).
+   Response: concatenate `content[]` blocks with `type == "text"`. Usage: `input_tokens`,
+   `cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`; cost = input×price_input
+   + cache_creation×price_cache_write + cache_read×price_cache_read + output×price_output (per M).
+   `stop_reason == "refusal"` (HTTP 200) surfaces as a distinct `LlmError` variant that triggers the
+   fallback; 429/5xx/network are `Transient` (retried by the existing `RetryPolicy`); 400 is never
+   retried. Request timeout 300 s. API key only from `DAILY_EPUB_ANTHROPIC__API_KEY`.
+   `ChatRequest` gains `effort: Option<String>`; the Anthropic backend ignores `temperature`/`json`
+   and maps `system` to the cached system block. `TokenUsage` needs a cache-write counter (keep the
+   DeepSeek path's cost identical to today). Generalize `UsageMeter` to take a per-provider price
+   table instead of `&DeepseekConfig`.
+2. **`Llms { bulk: Option<LlmClient>, editor: Option<LlmClient> }`** with `editor_or_bulk()` (the
+   editor when configured and its meter is not tripped, else bulk). Both clients share the exact same
+   system prompt string (§8.4). `LlmError::MissingApiKey` and friends must name the provider.
+3. **Config** (§4.2, §19): `[anthropic]` with `enabled`, `base_url`, `model = "claude-opus-5"`,
+   `api_key` (env only), `effort = "high"`, `price_input_per_mtok = 5.0`,
+   `price_cache_write_per_mtok = 6.25`, `price_cache_read_per_mtok = 0.5`, `price_output_per_mtok = 25.0`,
+   `max_daily_usd = 3.0`, `max_concurrent_requests = 4`; `[deepseek].max_concurrent_requests = 4`;
+   `[curation].max_article_count = 28`; `[editorial]` with `summary_model = "editor"` (`editor|bulk`)
+   and `summary_input_tokens = 3000`. Validate `max_article_count >= target_article_count`, effort ∈
+   {low, medium, high, xhigh, max}, batch/concurrency ≥ 1. Startup logs the resolved models and whether
+   each provider is enabled (§19). Update `config.example.toml` and the README (prerequisites, cost
+   line, the note that server-side fallback is enabled, dashboard spend limits as the real backstop).
+4. **Budget and concurrency** (§5): one `UsageMeter` per provider, each with its own `max_daily_usd`.
+   The budget day is the **UTC date of the run's `started_at`**, preloaded by summing
+   `runs.provider_costs_json` for earlier runs that UTC day; this replaces `db::spend_for_date`
+   (remove it). Write `runs.provider_costs_json` (`{"deepseek": {...usage, cost_usd}, "anthropic": {...}}`)
+   and `runs.config_json` (the resolved `[curation]`, `[editorial]`, model names and prompt versions,
+   as JSON) in `finish_run`. Keep `runs.cost_usd` as the total across providers. Existing Stage A
+   scoring batches run through `futures::stream::iter(...).buffer_unordered(max_concurrent_requests)`
+   with the budget check before each request is spawned; a tripped meter skips remaining calls,
+   lets in-flight finish, records the number of unscored candidates in the report, and continues.
+5. **The editor** (§13): `EDITOR_INSTRUCTIONS` replaces `SELECT_INSTRUCTIONS` verbatim from the plan
+   with `{soft_target}`/`{hard_max}` substituted. Runs on `editor_or_bulk()`; on refusal/error fall
+   back to the same prompt on the bulk client; if that fails too, `select_without_llm`. Per-item
+   rendering follows §13 with what exists today (Stage A score/category/rationale instead of
+   quality/fit/facets — those arrive in step 5; render `flags: always-include | excerpt only`;
+   `opening:` first 60 words). Do not put the numeric blend in the prompt. `assemble()` keeps section
+   validation, unique lead, auto-include reinsertion, duplicate-id defence, malformed-response
+   fallback and the `hard_max` trim (by today's ranking key, utility arrives in step 5).
+   **Delete the "too few: top up" branch.** `--max-articles N` is a ceiling:
+   `hard_max = min(config.curation.max_article_count, N)`, `soft_target = min(target_article_count, hard_max)`.
+   `select_without_llm` respects `soft_target` as its size. Each pick's `why` (≤14 words) lands on
+   `Pick.why: Option<String>` and in `issue_articles.why` (`replace_issue_articles`).
+6. **Editorial** (§14): summaries run on the editor client per `editorial.summary_model` with the
+   input budget from config (3,000 tokens), concurrency 4 via `buffer_unordered`, fallback per article:
+   bulk client, then the excerpt. `BRIEF_INSTRUCTIONS` replaces `FRONT_PAGE_INSTRUCTIONS` verbatim
+   from §14.2 (JSON `{"brief": "..."}`); input is the lineup with sections, each pick's title, feed,
+   `why`, summary and the Stage A score. `Editorial.section_intros` is removed (or always empty) and
+   `section.xhtml` renders only the section name; `front_page.xhtml` renders the brief under the
+   masthead; fallback stays `fallback_front_page_html`. The weekly profile rebuild runs on
+   `editor_or_bulk()` (pipeline and the `profile rebuild` CLI).
+7. **Paper** (§15.1): `chapter.xhtml` gets a small italic `Why it's here: <why>` line under the meta
+   line; the In-this-issue page shows the `why` line under each summary. `Colophon` gains
+   `provider_costs` and `models`; the colophon template prints per-provider cost lines and the models
+   (editor and summaries model, bulk model). `StageCounts`/`RunReport` carry per-provider usage;
+   `print_report` in `main.rs` prints per-provider cost.
+8. Remove `Vote`-era leftovers you notice only if they are in your files; do not touch embeddings,
+   signals, telemetry or triage (steps 3–5).
+
+Tests to add/adapt (§20 "Anthropic backend", "Editor", parts of "Pipeline"): request body has the
+system block with `cache_control`, no `temperature`, `output_config.effort`, `fallbacks`, the beta
+header; usage fields parsed into cost with cache read/write prices; `stop_reason: refusal` surfaces as
+the fallback-triggering error; 429 retried, 400 not (use a mock backend or a local `axum`/`tokio`
+listener — never the real network); a nine-pick response is published as nine; `hard_max` trims;
+`--max-articles` is a ceiling; auto-includes reinserted; `why` lines land on picks and in
+`issue_articles.why`; refusal/error on the Anthropic mock falls back to the DeepSeek mock with the
+same prompt; the brief is parsed and rendered, section intros gone; `provider_costs_json` and
+`config_json` are written; budget day preload sums by UTC date.

docs/plans/curation-v2-briefs/step3.md

@@ -0,0 +1,103 @@
+
+---
+
+# YOUR STEP: 3 — Embeddings, signals, telemetry (plan §21 step 3)
+
+You are working in a git **worktree** on branch `curation-v2-step3`, which branches from the step 1
+commit. Another agent is implementing step 2 (the Claude editor) in parallel on the main checkout;
+you will not see its changes and must not need them. Read `git log -3` and the current
+`src/curate/`, `src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs` first.
+Do not run `git checkout`, `git stash`, `git worktree` or `git commit`.
+
+Scope (plan §4.3, §7.1, §7.2, §7.4, §7.5, §9, §12.2 normalization only, §12.4 preliminary blend,
+§15.2 `explain`, §15.4 log line parts, §16, §17 Voyage row, §19 `[voyage]` + `[curation.ranking]`).
+The old heuristic prefilter still gates in this step; signals are computed for every article that
+passes hygiene and persisted, nothing about *selection* changes yet.
+
+1. **`src/curate/embedding.rs`** (§4.3, §7.1, §7.2): `EmbeddingBackend` trait mirroring `ChatBackend`
+   (with a mock for tests), `VoyageBackend` (`POST {base_url}/embeddings`, `Authorization: Bearer`,
+   body `{input, model, input_type: "document"|"query", truncation: true, output_dimension, output_dtype: "float"}`,
+   response mapped by index and length-checked, usage `total_tokens` metered), batching by `batch_size`
+   with `buffer_unordered(max_concurrent_requests)`, retry 429/5xx via `RetryPolicy`, f32
+   little-endian BLOB encode/decode with length and finiteness checks, `dot(a, b)` with a dimension
+   check, and cache orchestration: `article_embeddings` keyed by `(model, dimension, input_hash = sha256
+   of the embedded text)`, overwritten on change. Embedded text = `"Title: {title}\n\n{plain body}"`
+   via `curate::prompt_text`, whitespace collapsed, cut at `max_input_chars` on a char boundary —
+   **no feed name, author or scores**. Interests go to `interest_embeddings` with `input_type = "query"`
+   and the bare interest string. A failed batch leaves those articles without embeddings; never fatal.
+   A Voyage `UsageMeter` with `price_per_mtok = 0.02` and `max_daily_usd` (0.50) as a runaway guard.
+   Key only from `DAILY_EPUB_VOYAGE__API_KEY`. Raw vectors never reach logs or reports.
+2. **Config** (§4.3, §19): `[voyage]` (`enabled`, `base_url = "https://api.voyageai.com/v1"`,
+   `model = "voyage-4-lite"`, `output_dimension = 512`, `batch_size = 32`, `max_concurrent_requests = 4`,
+   `max_input_chars = 60000`, `max_daily_usd = 0.50`) and the complete `[curation.ranking]` block from
+   §19 (`triage_max`, `deep_keep`, `shortlist_keep`, `assessment_reuse_days`, `rating_lookback_days`,
+   `rating_half_life_days`, `neighbour_k`, `negative_coefficient`, `knn_floor/full`, `feed_floor/full`,
+   `semantic_min_words`, `exploration_slots`, `embedding_retention_days`, `telemetry_retention_days`,
+   `[curation.ranking.quotas]`, `[curation.ranking.weights.preliminary]`, `[curation.ranking.weights.utility]`,
+   `[curation.ranking.diversity]`) with the plan's defaults and the validation rules in §19 (weights
+   non-negative; `deep_keep ≥ shortlist_keep ≥ target_article_count`; `*_full > *_floor ≥ 0`;
+   `0 ≤ cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; batch sizes ≥ 1; dimension ∈ {256,512,1024,2048}).
+   Steps 4–5 will *use* the ranking keys; you only add and validate them. Keep `prefilter_keep` for now.
+   Startup logs whether Voyage is enabled. Update `config.example.toml` and the README.
+3. **`src/curate/signals.rs`** (§9): a `Signals` type where every signal is `Option<f64>` (absent ≠ 0):
+   `interest` (§9.1 z-scored interest match, top-three interests recorded; raw top-1 cosine fallback
+   under 30 embedded articles, logged), `knn` (§9.2 preference state from `db::current_ratings(rating_lookback_days)`
+   joined to `article_embeddings`, decayed weights `value × 0.5^(age/half_life)`, top-k positives and
+   negatives, `knn = pos − negative_coefficient × neg`, top-three neighbours recorded, gate ramp
+   `clamp((n − knn_floor)/(knn_full − knn_floor), 0, 1)` multiplying the weight, absent at 0),
+   `feed` (§9.3 Beta-smoothed decayed rate credited to distinct direct `SourceKind::Feed` feeds, mean
+   over the article's rated feeds, gate `feed_floor/feed_full`), `social` (existing composite; absent
+   with no rows), `heuristic` (`longform_points(word_count)` − excerpt-only − roundup penalty, from
+   `prefilter.rs` **without** the social, Scour/HN, multi-source terms; expose those pieces as
+   functions). Log once per run:
+   `preference: N rated articles with embeddings → knn gate X; feed gate Y (n=…)`.
+   Also implement the **mid-rank percentile normalizer** of §12.2 and the **preliminary blend** of
+   §12.4 (present-and-active weights renormalized) so `signals_json.norm`/`weights` can be written now.
+4. **Pipeline**: after social enrichment, for every article that passes hygiene, compute embeddings
+   (cached), signals, and the preliminary blend, then run the old prefilter as today. New stage
+   timings `embed` and `signals`. `--skip-embeddings` uses cached embeddings only (zero Voyage calls);
+   `--skip-llm` unchanged. `StageCounts` gains `eligible`, `embedded`, `rated_with_embeddings`, and
+   the report/`print_report` show them. The paper must still build when Voyage is down or unconfigured
+   (§17): `interest`/`knn` absent, never a penalty.
+5. **`src/curate/telemetry.rs`** (§7.4, §7.5): the `candidate_runs` writer. One row per considered
+   article per run, upserted with `INSERT … ON CONFLICT(run_id, article_id) DO UPDATE` setting every
+   column. Hygiene-excluded articles get thin rows (`stage='excluded'`, `excluded_reason` ∈
+   `blocked | published_before | recently_rejected`, `signals_json='{}'`). In this step map the old
+   pipeline onto the stage vocabulary: passed hygiene → `eligible`; cut by the prefilter →
+   `eligible` + `excluded_reason='not_admitted'`; prefilter survivors → `admitted`; Stage-A scored →
+   `assessed`; sent to selection → `shortlisted`; picked → `selected` (+ `editor_why` if a `why`
+   exists on the pick, else NULL); unpicked shortlist → `excluded_reason='not_selected'`. `signals_json`
+   follows §7.5 exactly (`v: 1`, `raw`, `norm`, `present`, `weights` = effective preliminary weights,
+   `top_interests`, `neighbours`, `exploration: false`, `auto_include`, `notes`). `utility` and
+   `rank_utility` stay NULL until step 5.
+6. **CLI** (§15.2, §16) in `src/main.rs`:
+   `explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]` and
+   `explain --date YYYY-MM-DD --near-misses [N]` printing the persisted `candidate_runs` row for the
+   latest non-dry run of that date (or `--run-id`): stage and reason; every raw/normalized signal with
+   presence and effective weight; top interests with z; nearest rated neighbours; assessments from
+   `article_assessments` when present (empty until step 4); admitted_by; the editor's `why`. `--url`
+   canonicalizes (`dedupe`) and looks the article up; if absent from `articles`, print that it was never
+   ingested. `--near-misses` lists the top N by preliminary blend (utility from step 5 when present)
+   that were not selected. `features backfill [--days 30] [--rated-only] [--all] [--yes]` embeds rated
+   and published articles first, then interests, then other recent articles only under `--all`, prints
+   an estimate and asks for confirmation above 5M tokens unless `--yes`, and is idempotent (warm cache
+   ⇒ zero calls). `features prune` removes `article_embeddings` rows for articles neither rated nor
+   published older than `embedding_retention_days` and `candidate_runs` rows whose run started more
+   than `telemetry_retention_days` ago. `features backfill` takes no lock yet (step 6 adds `lock.rs`).
+7. Types (§18): `Signals`, `Signal` helpers, `RatedArticle` already exists — extend if needed. Do not
+   create `Candidate` yet (step 4/5).
+
+Tests (§20 "Embeddings", "Interest z-scores", "Preference", "Normalization", plus telemetry/explain):
+BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed
+text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the
+others; embedded text contains no feed name or author; a broad interest with uniformly high cosine does
+not dominate while a specific interest with one strong match does; raw fallback under 30 articles; one
+loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high
+(anti-centroid); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate
+0 below `knn_floor`, 1 at `knn_full`, linear between; feed credit sums to 1 across direct feeds; feed
+affinity uses the mean; constant signal → 0.5 for everyone; ties get equal percentiles (400 identical
+zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate
+missing a signal is scored on the rest; `candidate_runs` rows written for every considered article with
+the right stage/reason on a mocked run; `--skip-embeddings` makes zero Voyage calls; a Voyage failure
+still publishes; `explain` renders from persisted rows and reports "never ingested"; `features prune`
+respects rated/published.

docs/plans/curation-v2-briefs/step4.md

@@ -0,0 +1,89 @@
+
+---
+
+# YOUR STEP: 4 — Triage replaces the gate (plan §21 step 4)
+
+Steps 1–3 have landed: three-way votes and `rating_events`; the Claude editor (`Llms { bulk, editor }`,
+`AnthropicBackend`, per-provider `UsageMeter`s, `why` lines, the Brief); Voyage embeddings,
+`signals.rs` (interest/knn/feed/social/heuristic, percentile normalizer, preliminary blend),
+`telemetry.rs` (`candidate_runs` writer), `explain`, `features backfill|prune`, and the full
+`[curation.ranking]` config. Read `git log -6`, then the current `src/curate/` (all files),
+`src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs`, `src/types.rs`.
+
+Scope (plan §7.3 reuse and churn rules, §8.1 hygiene, §10 triage, §11 admission and exploration,
+§12.4 preliminary blend already exists, §15.4 log line parts, §17 DeepSeek row, §19 `deep_keep`):
+
+1. **Migration `migrations/0003_drop_scores.sql`**: `DROP TABLE scores;` (deferred from step 1).
+   Remove `db::upsert_score`, `recently_low_scored_ids` and every `scores` reader. The churn rule
+   now reads `article_assessments`: an article whose latest `triage` `score < recent_rejection_floor`
+   (3.0) or `deep` `score < recent_rejection_floor` within `recent_rejection_days` (7) is
+   `recently_rejected`, unless auto-include (§7.3, §8.1).
+2. **`src/curate/triage.rs`** (§10): `TRIAGE_INSTRUCTIONS` and `TRIAGE_PROMPT_VERSION = 1` verbatim
+   from the plan; per-article block exactly as §10 (id, title, feed (category), author, length ·
+   excerpt only, opening = first 200 words via `prompt_text` + `truncate_words`, `matches interests:`
+   from `top_interests` with z ≥ 1.5 labelled strong (z ≥ 2.5) / weak, `closest rated:` from
+   neighbours with cosine ≥ 0.55; omit lines with nothing). Batches of `deepseek.triage_batch_size`
+   (25) through `buffer_unordered(max_concurrent_requests)` on the **bulk** client, budget check
+   before each batch. Tolerant parsing generalized from `score.rs::parse_score_response`: a
+   malformed item never sinks the batch, ids not in the batch are dropped, unknown `kind` → `other`,
+   `interest` clamped 0–10. Persist every result to `article_assessments (stage='triage', model,
+   prompt_version, profile_version, score, kind, rationale=why, assessed_at)`.
+   **Cache** (§7.3): skip articles with a reusable `triage` or `deep` assessment (same `model` and
+   `prompt_version`, `assessed_at` within `assessment_reuse_days`); `generate --rescore` ignores the
+   cache. A failed batch leaves `triage` absent for its articles.
+   **Pool cap** (§10): if eligible > `triage_max` (800), triage the union of top `0.7 × triage_max`
+   by preliminary blend, top 100 by `interest`, top 100 by `knn` (if active), all auto-includes,
+   filled to `triage_max` by blend; the rest get `stage='eligible'`, `excluded_reason='not_admitted'`.
+3. **`src/curate/admit.rs`** (§8.1, §11): hygiene moves here (`blocked`, `published_before` = in
+   `issue_articles` for any issue date before this run's date, `recently_rejected` per item 1;
+   auto-includes never excluded) writing the thin `candidate_runs` rows, and the union admission
+   into `deep_keep` (120) slots in this order, each retriever taking its top-N by its own signal among
+   not-yet-admitted, not-excluded articles, recording every retriever that would have taken an
+   article in `admitted_by` (first = the one that admitted it):
+   `auto_include` (uncapped) → `triage` (quota 60, floor `interest ≥ 5`) → `interest` (20; floors
+   `word_count ≥ semantic_min_words`, not `looks_like_roundup`, triage `interest ≥ 3` if triaged)
+   → `knn` (20, gate > 0, same floors) → `exploration` (5, §11.1) → `blend` (remaining).
+   Inactive retrievers release their quota to `blend`. Not admitted → `stage='triaged'` (or
+   `'eligible'` if never triaged) + `excluded_reason='not_admitted'`.
+   **Exploration** (§11.1): five slots for articles ranked between `deep_keep` and `deep_keep × 2.5`
+   by the preliminary blend with `word_count ≥ 300`, not roundups, triage `interest ≥ 4`; ordered by
+   `sha256(run_date || article_id)`; flagged `exploration = true` through to the editor prompt.
+4. **Prefilter reduced** (§18): `prefilter.rs` keeps only hygiene helpers (`is_blocked`,
+   `is_auto_include`, `looks_like_roundup`) and the text heuristic pieces `signals.rs` uses. Delete
+   `prefilter::run`, `score_article`'s social/Scour/HN/multi-source terms, `PrefilterContext`, the
+   `prefilter_keep` config key (replace uses with `curation.ranking.deep_keep`; the config validation
+   `deep_keep ≥ shortlist_keep ≥ target_article_count` already exists), and `Curator::prefilter`.
+   A stale `prefilter_keep` key in a config file must fail loudly with a message naming
+   `curation.ranking.deep_keep` (follow the `bookorbit_dir` precedent in `config.rs`).
+5. **Pipeline**: hygiene → embeddings → signals → preliminary blend → triage → admission → the
+   *existing* Stage A scoring (`score.rs`, unchanged this step) over the admitted deep set → the
+   editor → editorial, as today. The deep set (`admitted`) is what Stage A scores and what the editor
+   sees; step 5 replaces Stage A with deep assessment and adds the shortlist. Telemetry stages now:
+   `excluded | eligible | triaged | admitted | assessed | shortlisted | selected` with reasons per §7.4;
+   `admitted_by` and `exploration` written; `signals_json.raw/norm/present` gain `triage` (÷10 for norm).
+   Stage timings `triage` and `admit`; `StageCounts` gains `triaged`, `admitted`, `admitted_by` (map),
+   `exploration_admitted`, `exploration_selected`. Log the admission line from §15.4:
+   `admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0`.
+   `--skip-llm` skips triage; DeepSeek down or tripped ⇒ no triage, admission by
+   `interest`/`knn`/`blend`, editor still runs (§17). The editor prompt (§13 rendering) shows the
+   triage score and `flags: exploration` where present. `explain` prints the triage assessment
+   (score, kind, why) and `admitted_by`.
+6. `types.rs` (§18): introduce `Assessment { triage: Option<Triage>, deep: Option<Deep> }` and
+   `Triage { interest, kind, why, model, prompt_version, assessed_at }` (keep `Deep`/`LlmScore`
+   compatible with the existing Stage A output for now), and `Candidate { article, auto_include,
+   exploration, signals, assessment, utility: Option<f64>, cluster: Option<...>, admitted_by: Vec<String>,
+   stage, excluded_reason }` replacing `ScoredArticle` where the pipeline flows through admission.
+   Keep the change mechanical where `score.rs`/`select.rs` still consume the old shape (step 5
+   retires them); an adapter is acceptable.
+7. `runs.config_json` gains `TRIAGE_PROMPT_VERSION`; `config.example.toml` and README updated
+   (`prefilter_keep` removed, `[deepseek].triage_batch_size`, `--rescore`).
+
+Tests (§20 "Triage and deep parsing" triage half, "Admission", churn/cache): realistic triage
+fixture parsed; malformed items do not sink a batch; unknown kind → other; every `kind` token in the
+prompt round-trips; cached triage reused within `assessment_reuse_days` and ignored with `--rescore`;
+churn rule excludes a recent low triage score and spares auto-includes; a strong-interest,
+weak-heuristic, no-social article reaches the deep set; a 60-word stub with high interest similarity
+is not admitted by `interest`/`knn`; quotas honoured; inactive retrievers release quota; exploration
+deterministic per date and rotating across dates; auto-includes always admitted; excluded articles get
+thin rows with the right reason; pool cap path yields `not_admitted` rows; the mocked full pipeline
+still publishes with DeepSeek failing; migration 0003 drops `scores`.

docs/plans/curation-v2-briefs/step5.md

@@ -0,0 +1,69 @@
+
+---
+
+# YOUR STEP: 5 — Deep assessment, utility, diversity (plan §21 step 5)
+
+Steps 1–4 have landed; triage and union admission now gate the deep set and the old prefilter is
+reduced to hygiene helpers. Read `git log -8`, then all of `src/curate/`, `src/pipeline.rs`,
+`src/types.rs`, `src/config.rs`, `src/report.rs`, `src/main.rs`, `src/db.rs`.
+
+Scope (plan §12 in full, §13 rendering with facets/neighbours, §7.3 deep rows, §18 module retirements):
+
+1. **`src/curate/assess.rs`** replaces `score.rs` (§12.1): `DEEP_INSTRUCTIONS` and
+   `DEEP_PROMPT_VERSION = 1` verbatim from the plan (the section palette is substituted from
+   `curation.sections`). Per article: title, feed, author, length, excerpt-only flag, the triage
+   `why`, the interest and neighbour hint lines from §10, and a **representative sample** of
+   ~2,000 tokens: body under ~1,500 words sent whole; otherwise first 600 words, 500 around the
+   midpoint, last 400, with visible `[BEGINNING]`/`[MIDDLE]`/`[END]` markers, split on word
+   boundaries. Batches of `deepseek.deep_batch_size` (8, replaces `score_batch_size`) through
+   `buffer_unordered`, bulk client, budget check per batch. Auto-includes are assessed too.
+   Tolerant parsing: `quality`/`fit` clamped 0–10, `category` validated against the palette (invalid →
+   `None`, the editor decides), `paywalled_guess`, and `facets` where every unknown enum token degrades
+   to `None` and `specific_topics` is capped at 3. **Every enum token in the prompt must round-trip
+   through the parser (test).** Persist to `article_assessments (stage='deep', score=quality, fit,
+   kind=facets.format, facets_json, rationale, category, paywalled_guess, ...)`; reuse per §7.3 with
+   `--rescore` bypass. Facets are shown to the editor, the profile rebuild and `explain`; **not** a
+   numeric signal.
+2. **`src/curate/rank.rs`** (§12.2–§12.5): normalization (LLM scores ÷ 10; everything else mid-rank
+   percentile over the deep set — reuse/move the step-3 normalizer), the **utility** blend over
+   present signals with `[curation.ranking.weights.utility]` renormalized and learned signals
+   multiplied by their gate ramp first, stored 0–100; the preliminary blend stays for the eligible set.
+   **Diversified shortlist** (§12.5): leader clustering by embedding cosine with `cluster_threshold`
+   (0.85) and `per_cluster_cap` (2): sort by utility desc then article id asc; assign each article to
+   the first existing cluster whose *leader* has cosine ≥ threshold, else make it a new leader;
+   articles without embeddings are singletons; admit in order while the cluster's admitted count is
+   below the cap until `shortlist_keep` (60); the top `utility_protected` (10) by utility and all
+   auto-includes are admitted regardless and still count toward their cluster; exploration picks that
+   reached the deep set get up to 3 reserved slots; if short, relax to cap 3, then uncapped. Persist
+   `cluster_id`, `cluster_rank`, `rank_utility`, `utility`, `excluded_reason ∈ cluster_suppressed |
+   shortlist_cap`, stage `shortlisted`.
+3. **`src/curate/editor.rs`** replaces `select.rs`: the §13 rendering in full —
+   `quality 8.5 · fit 7.0 · triage 8.0 — <deep rationale>`, `facets: format · depth · evidence ·
+   technicality · topic_group`, `matches:`, `closest rated:`, `flags: exploration | always-include |
+   excerpt only`, `opening:` first 60 words. The editor sees the 60-item shortlist. `assemble()`'s
+   `hard_max` trim and `select_without_llm` now order by **utility**, falling back to the preliminary
+   blend. Delete `ScoredArticle::combined_score()` and `ScoredArticle` itself; `Candidate` is the only
+   flow type. Move the tests from `score.rs`/`select.rs` into `assess.rs`/`editor.rs` and delete the
+   old files (update `curate/mod.rs`, `Curator`).
+4. **Editorial inputs**: the Brief and the summaries use quality/fit where the plan says so (§14.2
+   input lists quality/fit). The weekly profile rebuild's rated lines now get real facets.
+5. **Pipeline/report**: stage timings `assess` and `rank`; `StageCounts` gains `assessed`,
+   `shortlisted`, `clusters`; the §15.4 curation line
+   `curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected`.
+   `explain` prints the deep assessment (quality, fit, category, rationale, facets), utility and rank,
+   cluster id and what suppressed it; `--near-misses` orders by utility. `runs.config_json` gains
+   `DEEP_PROMPT_VERSION`. `--skip-llm`/DeepSeek down ⇒ no deep assessment, utility over present
+   signals (triage/interest dominate, §12.3), editor still runs on what it has (§17).
+6. `config.example.toml`/README: `deep_batch_size` replaces `score_batch_size`; a stale
+   `score_batch_size` key fails loudly naming `deep_batch_size`.
+
+Tests (§20 "Triage and deep parsing" deep half, "Normalization", "Clustering", "Editor" utility
+parts): realistic deep fixture; malformed items do not sink a batch; unknown facet tokens → `None`;
+every enum token round-trips; representative sample has the three markers and respects word
+boundaries, short bodies are sent whole; cached deep rows reused / bypassed with `--rescore`; constant
+signal → 0.5; ties equal; absent values do not shift others; effective weights sum to 1; a candidate
+missing a signal is scored on the rest; near-duplicates share a cluster and the third is suppressed;
+protected top-N survive and count; the bridge case (A~C, B~C, A≁B, utility A>B>C) yields two clusters;
+articles without embeddings are never suppressed; `hard_max` trims by utility; `select_without_llm`
+orders by utility; the mocked full pipeline publishes with DeepSeek down and writes `shortlisted`
+rows with cluster ids.

docs/plans/curation-v2-briefs/step6.md

@@ -0,0 +1,62 @@
+
+---
+
+# YOUR STEP: 6 — Paper telemetry, stats, lock (plan §21 step 6)
+
+Steps 1–5 have landed: the full v2 ranking pipeline (hygiene → embeddings → signals → triage →
+admission → deep assessment → utility → cluster-capped shortlist → Claude editor → editorial),
+`candidate_runs` telemetry, `explain`, `features`, `ratings`. Read `git log -10`, then
+`src/pipeline.rs`, `src/report.rs`, `src/main.rs`, `src/curate/telemetry.rs`, `src/epub/chapters.rs`
+and `src/epub/templates/`, `src/types.rs` (`Colophon`), `src/db.rs`, README, `config.example.toml`.
+
+Scope (plan §5 lock, §15.1 Behind the paper + In-this-issue, §15.3 `stats`, §15.4 report/log block,
+§16 lock coverage, §19 startup logging, README/config docs):
+
+1. **Behind the paper** (§15.1): a new short chapter after the World Briefing and before the colophon
+   (`src/epub/templates/behind.xhtml`, `render_behind_the_paper`) with exactly the content shape in
+   the plan: the considered/eligible/triaged/read-closely/shortlisted/selected line; the
+   `Admitted via:` line; the `Learned signals:` line (rated-with-embeddings count and knn percentage,
+   feed affinity state); `Near misses` — the 10 highest-utility non-selected articles as
+   `<title> — <feed> · quality X · fit Y · <stage/reason>`; the `Models:` line with triage/assessment,
+   editor/summaries and embeddings models; cost and generation time. Both editions (X4 gets the same
+   text, no links). The data comes from the run's `RunReport`/`StageCounts` and the run's
+   `candidate_runs` rows (add a `BehindThePaper` struct to `types.rs` filled by the pipeline; keep
+   templates pure). Chapter ids stable (`behind`).
+2. **In this issue** page: verify each entry shows the `why` line under the summary (step 2 was asked
+   to do this; complete it if missing). The colophon keeps its fields and shows per-provider cost
+   lines and models (verify; complete if missing).
+3. **`stats`** (§15.3): `daily-epub stats [--days 14]` printing issues, articles published, explicit
+   ratings by label, ratings per issue, up/down ratio per admitting retriever (`admitted_by[0]` of
+   rated picks, from `candidate_runs` joined to `rating_events` via the latest explicit event),
+   exploration yield (exploration picks selected / admitted, and how many were rated positively),
+   mean issue size, cost per day per provider (from `runs.provider_costs_json`), mean generation time.
+   Plain text, one fact per line, no tables wider than 80 columns.
+4. **Run report and log block** (§15.4): make sure `StageCounts` has every field listed —
+   `eligible`, `embedded`, `triaged`, `admitted`, `admitted_by` (map), `assessed`, `shortlisted`,
+   `clusters`, `exploration_admitted`, `exploration_selected`, `verdicts_in_prompt`,
+   `rated_with_embeddings`, per-provider usage — and that stage timings include `embed`, `signals`,
+   `triage`, `admit`, `assess`, `rank`, `editor`, `summaries`, `brief`. Emit the four-line info block
+   from §15.4 once per run (`curation:`, `admission:`, `preference:`, `providers:`). `print_report`
+   in `main.rs` prints the same lines.
+5. **`src/lock.rs`** (§5): `flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`, taken in `main`
+   for `generate`, `profile rebuild`, `features backfill`, and `backfill-social`; a second invocation
+   exits with `generate is already running` (name the command that holds it if cheap, else the
+   generic message). `serve`, `explain`, `stats`, `ratings`, `db migrate`, `features prune` do not
+   take it. Use `libc`/`rustix` only if already in the dependency tree (check `Cargo.lock`); otherwise
+   `std::fs::File` + the `fs2`-free approach via `libc::flock` is acceptable to add as a tiny dep —
+   say which in the report. Twenty lines, no table, no TTL.
+6. **Docs**: README rewritten where it describes the pipeline, feedback links, costs, prerequisites
+   (Anthropic and Voyage keys, dashboard spend limits as the real backstop, the server-side fallback
+   note), the CLI (`ratings`, `explain`, `stats`, `features`, `--rescore`, `--skip-embeddings`,
+   `--max-articles` as ceiling), and `data/profile.md`. `config.example.toml` carries every key from
+   plan §19 with comments. Startup logs resolved models and per-provider enabled state (verify).
+7. `systemd/daily-epub-generate.service`: no change needed unless an env var name changed; note the
+   two new env vars in the README's systemd section.
+
+Tests (§20 "Lock", "Pipeline" behind-the-paper, stats): two processes/one wins (spawn the binary or
+use two `File` handles from separate threads with `try_lock` semantics — no sleeping longer than a
+second), a killed holder frees the lock, `serve` does not take it (parse-level or by checking no lock
+file is created); the behind-the-paper chapter renders with the counts and near misses on the mocked
+pipeline run and is parseable XHTML in both editions (extend `tests/m4_epub.rs`); `stats` runs on a
+temp DB with a couple of runs, issues and rating events and prints every line; the info block appears
+in the log (capture with `tracing` test subscriber or assert on the formatted strings).

docs/plans/curation-v2-briefs/step7.md

@@ -0,0 +1,29 @@
+
+---
+
+# YOUR STEP: 7 — Cleanup (plan §21 step 7)
+
+Steps 1–6 have landed. This step removes what the plan retired and updates the implementation
+notes. Read `git log -12` and skim every file under `src/` for leftovers.
+
+1. **Dead code**: anything only the old prefilter/scores/select/feed-prior paths used — unused
+   functions, types (`LlmScore` if nothing reads it, `ScoredArticle`, `FeedPrior`, `Rating`),
+   `kv` keys no longer read, config aliases and `#[serde(alias)]`s introduced as transition aids,
+   `// filled in by step N` comments, stale doc comments that still describe 👍/👎, "From the Editor",
+   section intros, prefilter gating, or `async-openai`. `cargo clippy --all-targets -- -W dead_code`
+   must be clean; do not add `#[allow(dead_code)]`.
+2. **Prune paths** (§7.1, §7.4): confirm `features prune` covers `article_embeddings`
+   (`embedding_retention_days`) and `candidate_runs` (`telemetry_retention_days`); add
+   `article_assessments` older than `telemetry_retention_days` to it. Make `generate` call the prune
+   once per run after publishing (best effort, logged).
+3. **Docs**: update `docs/plans/2026-08-15-implementation-notes.md`: fix item 5 (hand-rolled
+   `reqwest` client, not `async-openai`), add the new providers (Anthropic Messages API facts from
+   plan §4.2, Voyage facts from §4.3, each with "verified 2026-09-02"), the new tables, the lock, the
+   budget-day rule, and a short "curation v2" section pointing at
+   `docs/plans/2026-09-02-personalized-curation-v2.md`. This is the one step allowed to edit `docs/`.
+   Do not edit the plan itself.
+4. **Consistency pass**: `config.example.toml` matches `Config::default()` key for key (write a test
+   that loads the example and compares every `[curation.*]`, `[anthropic]`, `[voyage]`, `[editorial]`
+   value to the defaults, since the defaults are the plan's numbers); README CLI section matches
+   `daily-epub --help` output for every subcommand (spot check by running the binary).
+5. Run `cargo fmt`, `cargo clippy --all-targets`, `cargo test`; report the summary lines.