Bisect provider-rejected batches in triage and deep assessment

DeepSeek's content filter rejects a whole request (400 "Content Exists
Risk") when one article trips it, which cost the other articles in the
batch their assessment and retried them every run. New curate/batch.rs
runs both stages through a bisecting runner: a rejected batch is split
until the offending article is isolated, that article is retried once on
the editor provider when it is a different one, and a still-rejected
article is recorded as a provider_rejected assessment row so it is not
retried for assessment_reuse_days. Cache reuse accepts rows from either
configured model. Each stage logs reused/requested/rejected counts, the
curation: line shows rejections when non-zero, explain prints them, and
the llm_assess span reports the deep-set size.

Implemented by a Claude agent from an orchestrator brief; verified
fmt/clippy(-W dead_code)/test green (354 lib tests).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe

Cargo.lock

@@ -815,7 +815,7 @@ dependencies = [
 
 [[package]]
 name = "daily-epub"
-version = "0.1.0"
+version = "0.2.0"
 dependencies = [
  "ammonia",
  "anyhow",
@@ -831,6 +831,7 @@ dependencies = [
  "hmac",
  "image",
  "jiff",
+ "libc",
  "rand 0.10.2",
  "reqwest",
  "resvg",

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "daily-epub"
-version = "0.1.0"
+version = "0.2.0"
 edition = "2024"
 
 [dependencies]
@@ -18,8 +18,17 @@ hex = "0.4.3"
 hmac = "0.13.0"
 image = "0.25.10"
 jiff = { version = "0.2.35", features = ["serde"] }
+libc = "0.2.189"
 rand = "0.10.2"
-reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "gzip", "json", "charset", "http2", "query", "system-proxy"] }
+reqwest = { version = "0.13.4", default-features = false, features = [
+    "rustls",
+    "gzip",
+    "json",
+    "charset",
+    "http2",
+    "query",
+    "system-proxy",
+] }
 resvg = "0.48.1"
 scraper = "0.27.0"
 serde = { version = "1.0.229", features = ["derive"] }

README.md

@@ -4,16 +4,29 @@ A personalized daily newspaper, delivered as an EPUB.
 
 Every morning a systemd timer wakes one Rust binary. It pulls the last ~26 hours
 from a self-hosted [Miniflux](https://miniflux.app), deduplicates and extracts
-the articles, enriches them with HackerNews/Lobsters/Reddit social proof, filters
+the articles, enriches them with HackerNews/Lobsters/Reddit social proof, has
-300–500 candidates down to ~120 with cheap heuristics, and asks DeepSeek to score,
+DeepSeek triage every eligible opening and closely assess a 120-article union,
-select and introduce 15–25 of them. It assembles two EPUB editions (a standard one
+then utility-ranks and diversity-caps a 60-item shortlist for Claude Opus 5 — the editor —
+which assembles the issue
+(no minimum size, a hard ceiling), writes a one-line *why* under every headline,
+the summaries and *The Brief*. It assembles two EPUB editions (a standard one
 and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and
 publishes the lot over its own OPDS catalog — which doubles as a
 [BookOrbit](https://github.com/thallada/bookorbit) watched folder if you run one.
-Each article chapter ends with 👍/👎 links that feed back into tomorrow's curation.
+Each article chapter ends with Loved it / Good / Not for me links that feed back
+into tomorrow's curation, and a short *Behind the paper* chapter before the
+colophon says what the run considered, how the deep set was admitted, whether
+the learned signals were active, the ten highest-utility near misses, and what
+it all cost.
 
-Steady-state cost is roughly **$0.05–0.30/day** in DeepSeek tokens, hard-capped by
+Steady-state cost is roughly **$1/day**: $0.05–0.30 in DeepSeek tokens plus
-`max_daily_usd`.
+~$0.50–0.80 for the Claude editor and a few cents of Voyage AI embeddings, each
+provider with its own per-UTC-day ceiling (`providers.<name>.max_daily_usd` and
+`voyage.max_daily_usd`). Those ceilings are runaway guards, not accounting — set
+hard spend limits in the providers' dashboards as the real backstop. The two
+LLM roles — *bulk* (triage, assessment, fallbacks) and *editor* — are assigned
+by name to entries of a provider registry, so swapping DeepSeek or Claude for
+Gemini (or anything OpenAI-compatible) is a config line plus an API key.
 
 - Full design: [`docs/plans/2026-08-15-the-daily-epub.md`](docs/plans/2026-08-15-the-daily-epub.md)
 - Implementation decisions: [`docs/plans/2026-08-15-implementation-notes.md`](docs/plans/2026-08-15-implementation-notes.md)
@@ -24,20 +37,48 @@ Steady-state cost is roughly **$0.05–0.30/day** in DeepSeek tokens, hard-cappe
 
 ```
 Miniflux ingest ─▶ dedupe ─▶ extraction ─▶ persist ─▶ social enrichment
-  ─▶ pre-filter ─▶ LLM scoring ─▶ selection ─▶ comments ─▶ editorial
+  ─▶ hygiene ─▶ embeddings (Voyage) + cheap signals ─▶ triage (bulk LLM)
+  ─▶ union admission ─▶ deep assessment (bulk LLM) ─▶ utility + diversity
+  ─▶ editor (editor LLM) ─▶ comments ─▶ editorial (editor LLM)
   ─▶ world briefing ─▶ EPUB (standard + X4) ─▶ XTC ─▶ publish ─▶ report
 ```
 
 Every stage writes to SQLite, so a run is idempotent per date: re-running
 `generate --date 2026-08-15` replaces that issue rather than duplicating it.
+Only one writer runs at a time: `generate`, `profile rebuild`, `features
+backfill` and `backfill-social` take an advisory `flock` on
+`<database_path>.lock`, and a second invocation exits with `generate is already
+running` (naming whichever command holds it). `serve`, `explain`, `stats`,
+`ratings`, `features prune` and `db migrate` never wait on it, and the kernel
+releases the lock when the holder exits, however it exits.
+
+Every run ends with a four-line summary in the log and on stdout:
+
+```
+curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected
+admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0
+preference: 14 rated w/ embeddings → knn 0.35 · feed off · 41 verdicts in prompt
+providers: anthropic $0.62 · deepseek $0.11 · voyage $0.02 · total $0.75 · 23m12s
+```
+
+The full report (per-stage counts and timings — `embed`, `signals`, `triage`,
+`admit`, `assess`, `rank`, `editor`, `summaries`, `brief` among them — and
+per-provider usage) is stored on the `runs` row and in `issues.report_json`.
 
 **Failure policy.** Miniflux ingest, SQLite writes, EPUB assembly and publishing
 are fatal — without them there is no issue, and the `runs` row records why.
 Social lookups, comment fetching, the world briefing, images and the XTC
 conversion are best-effort: they log, add a warning (run status `degraded`) and
-the run continues. Every DeepSeek stage *degrades*: a missing key, a dead API or
+the run continues. Every LLM stage *degrades*: an editor call that fails, is
-a tripped budget turns the run into the `--skip-llm` shape (prefilter order
+refused, or is over its provider's daily ceiling is retried with the same
-selects, feed excerpts stand in for summaries) instead of losing the day's issue.
+prompt on the bulk provider; if the bulk provider is missing, dead or over
+budget too, the run takes the `--skip-llm` shape (admission uses cheap signals
+and feed excerpts stand in for summaries) instead of losing the day's issue.
+Which provider plays which role is the `[llm]` table (`bulk = "deepseek"`,
+`editor = "anthropic"` by default); a role whose key is absent is simply
+unavailable, and `daily-epub config check` shows the resolved assignment before
+a run. Anthropic's server-side refusal fallback (`fallbacks = "default"`) is
+enabled on every request to an `anthropic`-kind provider.
 
 ---
 
@@ -47,14 +88,16 @@ selects, feed excerpts stand in for summaries) instead of losing the day's issue
 |---|---|---|
 | Rust (2024 edition toolchain) | building | `cargo build --release` |
 | **Miniflux** with an API key | the only content source | Settings → API Keys. The client is read-only and never mutates read state. |
-| **DeepSeek API key** | curation + editorial | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. |
+| A key for the **bulk** provider (DeepSeek by default) | triage and deep assessment, and the fallback for every editor call | <https://platform.deepseek.com>. `DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY`. Optional: `--skip-llm` runs the whole pipeline without it. |
-| A 32+ byte random secret | signs the 👍/👎 rating links | `openssl rand -hex 32` |
+| A key for the **editor** provider (Anthropic by default) | the editor: selection, summaries, The Brief, the weekly profile rebuild | <https://console.anthropic.com>. `DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY`. Optional: without it every editor call runs on the bulk provider. Set a dashboard spend limit; `providers.anthropic.max_daily_usd` is only a runaway guard. Any other `[providers.*]` entry (Gemini is shipped) can take either role — see *Switching providers*. |
+| **Voyage AI API key** | article and interest embeddings behind the learned ranking signals | <https://www.voyageai.com>. Optional: without it (or with `--skip-embeddings`) the run uses cached vectors only and the learned signals are absent, never a penalty. |
+| A 32+ byte random secret | signs the article rating links | `openssl rand -hex 32` |
 | **BookOrbit** library + watched folder | *optional* — a richer library UI on top of the same folder | Delivery does not need it: `daily-epub serve` has its own OPDS catalog over `publish.epub_dir`. If you do run it, create a dedicated "The Daily EPUB" library, enable *Watch folders*, and point `publish.epub_dir` at it. |
 | **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF — see below. It has **no global npm bin** — it is invoked as `node <repo>/cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. |
 | A reverse proxy for `daily.hallada.net` → `127.0.0.1:3499` | rating links must be reachable from e-readers on the internet | TLS via your existing setup. |
 
-`data/scour-interests.opml` (the ~220 Scour interests the taste profile is seeded
+`data/profile.md` is the hand-maintained reader profile; its optional interests are merged
-from) must be readable at the path in `interests_opml`.
+with `data/scour-interests.opml`. Both paths are configurable.
 
 ---
 
@@ -70,11 +113,20 @@ sudo install -m0755 target/release/daily-epub /usr/local/bin/
 ### Commands
 
 ```
-daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
+daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] [--skip-embeddings] [--rescore]
 daily-epub serve                # rating endpoints + OPDS catalog + downloads
-daily-epub profile rebuild      # regenerate the taste profile from ratings (weekly inside generate)
+daily-epub profile rebuild      # regenerate learned profile adjustments
-daily-epub backfill-social      # re-poll social scores for recent articles
+daily-epub ratings list [--days 90] [--label loved|good|down|cleared]
+daily-epub ratings set --article 42 --label loved --note "excellent"
+daily-epub ratings clear --url https://example.com/article
+daily-epub explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]
+daily-epub explain --date YYYY-MM-DD --near-misses [N]
+daily-epub stats [--days 14]    # the evaluation framework, one fact per line
+daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes]
+daily-epub features prune       # stale embeddings, old telemetry and assessments
+daily-epub backfill-social [--days 7]   # re-poll social scores for recent articles
 daily-epub db migrate           # run migrations (also automatic on every start)
+daily-epub config check         # validate the config, print the resolved roles, keys and paths
 ```
 
 `--dry-run` does everything except deliver: it still ingests, persists entries and
@@ -82,6 +134,49 @@ articles, curates and **builds both EPUBs into `--out`**, but it does not copy t
 BookOrbit, does not run the retention sweep, does not write the `issues` row and
 does not advance the ingest watermark. It prints the lineup and the cost report.
 
+`--skip-embeddings` reads the embedding cache but makes zero Voyage calls.
+`--rescore` ignores reusable triage/deep assessments for this run.
+`--max-articles N` is a ceiling, never a target: the hard maximum becomes
+`min(curation.max_article_count, N)` and the soft target is lowered to fit.
+
+`explain` answers "why was this (not) in the paper" from the `candidate_runs`
+row the run persisted for every considered article: the stage it reached and the
+reason it stopped, every raw and normalized signal with its presence and
+effective weight, the top interests, the nearest rated neighbours, the triage
+and deep assessments (quality, fit, category, rationale, facets), utility and
+rank, the cluster it landed in and what suppressed it, and the editor's reason
+for a pick. `--url` canonicalizes the address; an article that is not in the
+database at all is reported as never ingested (a feed problem, not a ranking
+one). `--near-misses` lists the highest-utility articles that were not selected
+(by preliminary blend for articles the ranker never reached).
+
+`stats` is the whole evaluation framework, on purpose: for the last `--days`
+(14) it prints the issues and articles published, the mean issue size, explicit
+ratings by label and per issue, the up/down ratio of rated picks per admitting
+retriever (`admitted_by[0]` — triage, interest, knn, exploration, blend,
+auto_include), the exploration yield (admitted, selected, rated positively),
+cost per day per provider from `runs.provider_costs_json`, and the mean
+generation time. Plain text, one fact per line. Tune from it and from reading
+the paper; anything more waits for more ratings.
+
+`features backfill` embeds the rated and published articles first (the learned
+set), then the standing interests, then — only with `--all` — every other
+article first seen in the window. It prints an estimate and asks before spending
+more than 5M tokens unless `--yes`; a warm cache makes zero calls. `features
+prune` drops embeddings of articles neither rated nor published that are older
+than `curation.ranking.embedding_retention_days`, and `candidate_runs` rows and
+`article_assessments` older than `curation.ranking.telemetry_retention_days`.
+`generate` runs the same sweep once after publishing, best effort.
+
+`config check` loads and validates the configuration exactly as `generate`
+would and prints one fact per line: the config path, the database, profile,
+interests and publish paths with `exists`/`MISSING`, each `[llm]` role with its
+provider name, kind, model, effort, ceiling and whether its key is present, the
+Voyage line likewise, and `editorial.summary_model`. Lines that need attention
+start with `!`. It exits non-zero only on a validation error — a missing key or
+file is a warning, since the run degrades rather than fails — and never opens
+the database or takes the lock, so it is safe to run next to a live `generate`.
+
 ---
 
 ## Configuration
@@ -93,12 +188,43 @@ Start from [`config.example.toml`](config.example.toml). Load order, later wins:
 3. `DAILY_EPUB_*` environment variables
 
 Nested keys use a **double underscore**: `[miniflux] api_key` becomes
-`DAILY_EPUB_MINIFLUX__API_KEY`. Top-level keys are just uppercased:
+`DAILY_EPUB_MINIFLUX__API_KEY`, and a provider's key is
-`DAILY_EPUB_LOOKBACK_HOURS=30`. As a convenience, plain **`DAILY_EPUB_SECRET`**
+`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY` with the `[providers.<name>]` table name
-is accepted as an alias for `server.hmac_secret` (the explicit key wins if both
+upper-cased — the shipped registry reads `DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY`,
-are set).
+`DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY` and
+`DAILY_EPUB_PROVIDERS__GEMINI__API_KEY` (provider names are therefore lowercase
+`a-z0-9_`). Top-level keys are just uppercased: `DAILY_EPUB_LOOKBACK_HOURS=30`.
+As a convenience, plain **`DAILY_EPUB_SECRET`** is accepted as an alias for
+`server.hmac_secret` (the explicit key wins if both are set).
 
-Secrets belong in the environment file, never in the TOML.
+Secrets belong in the environment file, never in the TOML. Stale configuration
+fails at startup rather than silently curating without a provider: a
+`[deepseek]` or `[anthropic]` table, a top-level `max_daily_usd`, a batch-size
+or temperature key outside `[llm]`, or a `DAILY_EPUB_DEEPSEEK__*` /
+`DAILY_EPUB_ANTHROPIC__*` environment variable is an error naming the new key.
+
+### Switching providers
+
+The roles are names, the providers are tables. To run the editor on Gemini:
+
+```toml
+[llm]
+editor = "gemini"          # [providers.gemini] is already declared in config.example.toml
+```
+
+and put `DAILY_EPUB_PROVIDERS__GEMINI__API_KEY=…` in the env file. A one-off
+A/B without touching the file, since every key is also an env var:
+
+```sh
+DAILY_EPUB_LLM__EDITOR=gemini daily-epub generate --dry-run --date 2026-09-02
+```
+
+The same works for `bulk`. Both roles may name one provider (they then share
+one client and one `max_daily_usd`), `editor = ""` runs everything on bulk, and
+a new endpoint is a new `[providers.<name>]` table: `kind = "openai"` for any
+OpenAI-compatible chat-completions API (DeepSeek, Gemini, OpenAI, a local
+server), `kind = "anthropic"` for the Messages API. `daily-epub config check`
+prints what resolved.
 
 ### Reference
 
@@ -106,31 +232,57 @@ Secrets belong in the environment file, never in the TOML.
 |---|---|---|
 | `timezone` | `America/New_York` | Day boundaries and `--date` interpretation. |
 | `lookback_hours` | `26` | Size of the ingest window ending at the issue day's end (clamped to now). |
-| `target_article_count` | `20` | Lineup size the selector aims for. `--max-articles` overrides it. |
+| `target_article_count` | `20` | Soft target the editor aims for. There is no minimum: a nine-pick issue is published as nine. |
-| `prefilter_keep` | `120` | Candidates surviving the heuristic pre-filter. Must be ≥ `target_article_count`. |
 | `retention_days` | `21` | EPUBs older than this are deleted from `publish.epub_dir`. SQLite history is kept forever. |
 | `xtc_retention_count` | `5` | How many XTC issues to keep in `publish.xtc_dir`. Counted, not dated: each `.xtch` is ~80–100 MB, so the binding constraint is disk, not age. |
-| `max_daily_usd` | `2.0` | Hard ceiling on DeepSeek spend **per day**, not per run — a re-run inherits what earlier runs for that date already spent. Tripping it skips remaining LLM work and degrades to excerpts. |
 | `world_briefing` | `true` | Include the Wikipedia Current Events section. |
 | `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. |
 | `out_dir` | `/var/lib/daily-epub/out` | Where `generate` writes artifacts before publishing. |
-| `interests_opml` | `data/scour-interests.opml` | Scour interest export used to seed the taste profile. |
+| `profile_path` | `data/profile.md` | Hand-maintained reader profile, loaded every run. |
+| `interests_opml` | `data/scour-interests.opml` | Scour interests merged with the profile interests. |
 | `miniflux.base_url` | `http://127.0.0.1:8082` | Miniflux root (no `/v1`). |
 | `miniflux.api_key` | — | **`DAILY_EPUB_MINIFLUX__API_KEY`**. Required. |
 | `miniflux.page_limit` | `250` | Entries per page; Miniflux caps this at 250. |
-| `deepseek.base_url` | `https://api.deepseek.com/v1` | OpenAI-compatible endpoint. |
+| `llm.bulk` | `deepseek` | The `[providers.*]` name that runs triage, deep assessment and every fallback. `""` ⇒ no bulk provider (those stages are skipped). |
-| `deepseek.model` | `deepseek-v4-flash` | Verified 2026-08-15 (DeepSeek-V4-Flash-0731). |
+| `llm.editor` | `anthropic` | The provider that assembles the lineup, writes the summaries and The Brief and rebuilds the profile. `""` or absent ⇒ everything runs on `bulk`. Naming the same provider as `bulk` shares one client and one ceiling. |
-| `deepseek.api_key` | — | **`DAILY_EPUB_DEEPSEEK__API_KEY`**. Absent ⇒ the run curates heuristically. |
+| `llm.triage_batch_size` | `25` | Articles per first-pass triage request. |
-| `deepseek.score_batch_size` | `12` | Articles per stage-A scoring request. |
+| `llm.deep_batch_size` | `8` | Articles per close-reading assessment request. The removed `score_batch_size` key is a startup error. |
-| `deepseek.score_temperature` | `0.3` | Scoring/selection temperature. |
+| `llm.score_temperature` | `0.3` | Scoring temperature, sent only to `openai`-kind providers. |
-| `deepseek.editorial_temperature` | `0.8` | Summaries, intros, front page. |
+| `llm.editorial_temperature` | `0.8` | Summaries and The Brief on an `openai`-kind provider. |
-| `deepseek.price_input_per_mtok` | `0.14` | USD per 1M cache-miss input tokens (cost guardrail arithmetic). |
+| `providers.<name>.kind` | — | `openai` (chat completions at `{base_url}/chat/completions`, bearer key) or `anthropic` (Messages API: `output_config.effort`, a cached system block, `fallbacks = "default"` with the `server-side-fallback-2026-07-01` beta). Shipped entries: `deepseek`, `anthropic`, `gemini`. |
-| `deepseek.price_cached_input_per_mtok` | `0.0028` | USD per 1M prefix-cache-hit input tokens. |
+| `providers.<name>.base_url` | — | Endpoint root. DeepSeek `https://api.deepseek.com/v1`; Anthropic `https://api.anthropic.com`; Gemini `https://generativelanguage.googleapis.com/v1beta/openai`. |
-| `deepseek.price_output_per_mtok` | `0.28` | USD per 1M output tokens. |
+| `providers.<name>.model` | — | `deepseek-v4-flash` (verified 2026-08-15), `claude-opus-5`, `gemini-3.8-flash` (verified 2026-09-02). |
+| `providers.<name>.api_key` | — | **`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY`**, environment only. Absent ⇒ that role is unavailable and degrades (bulk ⇒ heuristic curation, editor ⇒ bulk). |
+| `providers.<name>.effort` | `high` (anthropic, gemini) | `anthropic`: `low`, `medium`, `high`, `xhigh` or `max` → `output_config.effort`. `openai`: passed through as `reasoning_effort` (Gemini takes `minimal`–`high`); omit it for models without one (DeepSeek). |
+| `providers.<name>.max_daily_usd` | `2.0` / `3.0` / `3.0` | That provider's ceiling per **UTC day** of the run's start, not per run — a re-run inherits what earlier runs that day already spent on it (`runs.provider_costs_json`). Tripping it skips that provider's remaining calls; in-flight requests finish and the paper still publishes. `0` disables the guard. |
+| `providers.<name>.max_concurrent_requests` | `4` | Triage and deep-assessment batches in flight on the bulk provider; summaries in flight on the summary provider. |
+| `providers.<name>.price_input_per_mtok` | `0.14` / `5.0` / `0.75` | USD per 1M cache-miss input tokens (cost guardrail arithmetic). |
+| `providers.<name>.price_cache_read_per_mtok` | `0.0028` / `0.5` / `0.075` | USD per 1M cache-hit input tokens. |
+| `providers.<name>.price_cache_write_per_mtok` | `0.0` / `6.25` / `0.0` | USD per 1M tokens written to the prompt cache (implicit caches charge nothing). |
+| `providers.<name>.price_output_per_mtok` | `0.28` / `25.0` / `3.75` | USD per 1M output tokens, thinking tokens included where the provider bills them as output. |
+| `voyage.enabled` | `true` | Embed articles and interests with Voyage AI. `false` ⇒ cached vectors only. |
+| `voyage.base_url` | `https://api.voyageai.com/v1` | `POST {base_url}/embeddings`. |
+| `voyage.model` | `voyage-4-lite` | Embedding model; changing it invalidates the cache. |
+| `voyage.api_key` | — | **`DAILY_EPUB_VOYAGE__API_KEY`**. Absent ⇒ cached vectors only. |
+| `voyage.output_dimension` | `512` | One of 256, 512, 1024, 2048. |
+| `voyage.batch_size` | `32` | Texts per request. |
+| `voyage.max_concurrent_requests` | `4` | Requests in flight. |
+| `voyage.max_input_chars` | `60000` | Per-article cut, on a char boundary. |
+| `voyage.max_daily_usd` | `0.50` | Runaway guard at $0.02/M tokens. |
+| `curation.max_article_count` | `28` | Hard ceiling on issue size. `--max-articles N` lowers it to `min(28, N)` and drags the soft target down with it. Must be ≥ `target_article_count`. |
 | `curation.always_include_feeds` | `[]` | Miniflux feed ids or URL substrings that can never be dropped. |
 | `curation.blocked_domains` | `[]` | Hosts excluded outright. |
 | `curation.paywall_domains` | `[]` | Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, economist, …). |
 | `curation.sections` | 8 sections | The **only** section names the model may use. `World Briefing` is reserved and never offered. |
+| `curation.feedback.loved_value` | `1.0` | Weight for a Loved it verdict. |
+| `curation.feedback.good_value` | `0.35` | Weight for a Good verdict. |
+| `curation.feedback.not_for_me_value` | `-1.0` | Weight for a Not for me verdict. |
+| `curation.feedback.verdicts_in_prompt` | `60` | Recent explicit verdicts included in the system prompt. |
+| `curation.recent_rejection_days` | `7` | Churn window for recent low triage/deep assessments. |
+| `curation.recent_rejection_floor` | `3.0` | Scores below this floor are excluded during the churn window (except auto-includes). |
+| `curation.ranking.*` | see below | Every weight, quota, gate and threshold of the personalized ranker. |
+| `editorial.summary_model` | `editor` | Which `[llm]` role writes the per-article summaries: `editor` (with per-article bulk fallback) or `bulk`. |
+| `editorial.summary_input_tokens` | `3000` | Article text offered to the summary prompt. |
 | `publish.epub_dir` | `/srv/bookorbit/libraries/daily-epub` | Both EPUB editions land here by atomic copy, and this is the directory the OPDS feed lists. The editions are distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. Point a BookOrbit watched folder at it if you want its UI too. **Renamed from `bookorbit_dir`**; the old key is a hard config error. |
 | `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts. **Not** listed in the OPDS feed — CrossPoint cannot acquire them — but downloadable at `/files/xtc/<name>` for sideloading. |
 | `xtc.enabled` | `true` | Set `false` to skip the converter entirely. |
@@ -143,6 +295,29 @@ Secrets belong in the environment file, never in the TOML.
 | `server.hmac_secret` | — | **`DAILY_EPUB_SERVER__HMAC_SECRET`** (or `DAILY_EPUB_SECRET`). Without it, generated links are rejected with 403. |
 | `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/*` and `/files/*`. |
 
+`[curation.ranking]` holds the ranker's tunables. The learned signals are
+gated: `knn` (rated-neighbour preference) ramps from `knn_floor` (8) to
+`knn_full` (25) rated articles with embeddings, `feed` (feed affinity) from
+`feed_floor` (15) to `feed_full` (40) attributable ratings; below the floor the
+signal is absent. Ratings decay with `rating_half_life_days` (60) over
+`rating_lookback_days` (180); `neighbour_k` (5) neighbours per side and
+`negative_coefficient` (0.75) shape the signal. `triage_max` (800),
+`deep_keep` (120), `shortlist_keep` (60), `assessment_reuse_days` (3),
+`semantic_min_words` (300), `exploration_slots` (5), `[curation.ranking.quotas]`
+(`triage` 60 · `interest` 20 · `knn` 20), `[curation.ranking.weights.utility]`
+(`quality` 0.40 · `fit` 0.20 · `knn` 0.15 · `interest` 0.10 · `feed` 0.05 ·
+`triage` 0.05 · `social` 0.03 · `heuristic` 0.02, over the signals present for
+each article of the deep set) and `[curation.ranking.diversity]`
+(`cluster_threshold` 0.85, `per_cluster_cap` 2, `utility_protected` 10) drive
+the LLM triage, deep assessment, utility ranking and diversification stages.
+`[curation.ranking.weights.preliminary]` (`interest` 0.35 · `knn` 0.25 ·
+`heuristic` 0.20 · `feed` 0.10 · `social` 0.10) blends the cheap signals; weights
+are renormalized over the signals present for each article, so they need not sum
+to 1. `embedding_retention_days` (120) and `telemetry_retention_days` (180) are
+what `features prune` enforces. Validation: weights non-negative; `deep_keep ≥
+shortlist_keep ≥ target_article_count`; `*_full > *_floor`; `0 ≤
+cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; Voyage batch size and concurrency ≥ 1.
+
 ---
 
 ## Deployment (systemd)
@@ -165,10 +340,20 @@ sudo install -m0640 -o daily-epub -g daily-epub config.example.toml /etc/daily-e
 sudo -e /etc/daily-epub/config.toml            # set publish dirs, xtc args, public_url
 sudo tee /etc/daily-epub/env >/dev/null <<EOF
 DAILY_EPUB_MINIFLUX__API_KEY=…
-DAILY_EPUB_DEEPSEEK__API_KEY=…
+DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY=…
+DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY=…
+# DAILY_EPUB_PROVIDERS__GEMINI__API_KEY=…     # only if a role names "gemini"
+DAILY_EPUB_VOYAGE__API_KEY=…
 DAILY_EPUB_SERVER__HMAC_SECRET=$(openssl rand -hex 32)
 EOF
 sudo chown daily-epub:daily-epub /etc/daily-epub/env && sudo chmod 0600 /etc/daily-epub/env
+# One key per [providers.<name>] entry a role uses, named after the table
+# (upper-cased). Any may be left unset: that role is then unavailable and the
+# run degrades (editor → bulk, bulk → heuristic curation, Voyage → cached
+# vectors). The pre-registry DAILY_EPUB_DEEPSEEK__API_KEY /
+# DAILY_EPUB_ANTHROPIC__API_KEY names are a startup error, not a silent no-op.
+sudo -u daily-epub bash -c 'set -a; . /etc/daily-epub/env; set +a;
+  daily-epub --config /etc/daily-epub/config.toml config check'   # roles, keys present?, paths
 
 # publish dirs must exist and be writable by the service user
 sudo install -d -o daily-epub -g daily-epub /var/lib/daily-epub/xtc
@@ -327,6 +512,7 @@ Condensed from spec §5. Run it in this order the first time.
 
 ```sh
 # 1. Config and schema
+daily-epub --config /etc/daily-epub/config.toml config check   # roles → providers, keys present?
 daily-epub --config /etc/daily-epub/config.toml db migrate
 
 # 2. Ingest only, no keys spent: does Miniflux answer, and with how much?
@@ -336,12 +522,13 @@ DAILY_EPUB_OUT_DIR=./out daily-epub generate --dry-run --skip-llm --max-articles
 # 3. Inspect the artifacts
 ls -la ./out                       # two .epub files
 epubcheck "./out/The Daily EPUB - $(date +%F).epub"   # expect zero errors
-#    open the standard edition in Calibre / KOReader: cover, From the Editor,
+#    open the standard edition in Calibre / KOReader: cover, The Brief,
-#    In This Issue, sections, discussions, colophon; TOC depth 2
+#    In This Issue, sections, discussions, Behind the paper, colophon; TOC depth 2
 
-# 4. Now with DeepSeek, still not publishing
+# 4. Now with the bulk and editor providers, still not publishing
 daily-epub generate --dry-run --out ./out --max-articles 6
-#    → check the lineup is sane and the printed cost is well under $0.50
+#    → check the lineup is sane (at most 6 picks, each with a "why" line) and the
+#      printed per-provider cost is well under $1
 
 # 5. Full live run
 sudo systemctl start daily-epub-generate
@@ -356,17 +543,17 @@ curl -s https://daily.hallada.net/healthz
 curl -s https://daily.hallada.net/opds/daily.xml | head
 curl -s https://daily.hallada.net/issues.json | jq '.[0]'
 
-# 7. Feedback loop: tap 👍 in KOReader, then
+# 7. Feedback loop: tap Loved it / Good / Not for me in KOReader, then
-sqlite3 /var/lib/daily-epub/daily-epub.db 'select * from ratings;'
+sqlite3 /var/lib/daily-epub/daily-epub.db 'select * from rating_events order by event_at desc;'
-sqlite3 /var/lib/daily-epub/daily-epub.db 'select * from feed_priors;'
 
 # 8. Watch cost and quality for a week
-sqlite3 /var/lib/daily-epub/daily-epub.db \
+daily-epub stats --days 7
-  'select date, status, entries_fetched, candidates, selected, cost_usd from runs order by id desc limit 7;'
+daily-epub explain --date $(date +%F) --near-misses
 ```
 
-Tune `prefilter_keep`, `target_article_count` and `curation.always_include_feeds`
+Tune `curation.ranking.deep_keep`, `target_article_count` and `curation.always_include_feeds`
-from what you see in step 8.
+from what you see in step 8, and read the paper's *Behind the paper* chapter
+each morning: it is the same numbers, on the device.
 
 ### Troubleshooting
 
@@ -395,14 +582,60 @@ cargo fmt
 
 The crate is a library plus a thin binary, so tests drive the pipeline directly.
 `tests/e2e_pipeline.rs` is the capstone: synthetic entries → dedupe → offline
-extraction → prefilter → selection (both the `--skip-llm` route and a
+extraction → signals → triage → admission → selection (both the `--skip-llm` route and a
 `MockBackend` DeepSeek route) → editorial → both EPUB editions → publish → OPDS
-and database rows, with no network access anywhere.
+and database rows, with no network access anywhere. `tests/m4_epub.rs` covers
+the rendered chapters, including *Behind the paper* in both editions; the
+`stats` and lock tests live next to their modules (`curate/telemetry.rs`,
+`lock.rs`).
 
-Layout: `src/pipeline.rs` wires the stages; `src/{dedupe,extract,social,curate,
+### Layout
-comments,world,epub,publish,server}` implement them; `src/auth.rs` owns the rating
+
-token formula used by both the EPUB writer and the server; `src/types.rs` is the
+`src/pipeline.rs` wires the stages; `src/types.rs` is the contract between them;
-contract between stages.
+`src/auth.rs` owns the rating token formula, shared by the EPUB writer and the
+server. The stages themselves:
+
+```text
+miniflux.rs   ingest            curate/       triage, assessment and selection
+dedupe.rs     clustering          prefilter, llm, triage, admit, assess, rank,
+                                  editor, editorial, embedding, signals, telemetry
+extract.rs    body text           profile/    the reader's taste profile
+images/       article images    comments.rs   discussion chapters
+  normalize     usable <img>    world.rs      the world briefing
+  refs          what's there    epub/         the two editions
+  fetch         download          chapters, cover, build, x4
+  encode        re-encode       publish.rs    BookOrbit + XTC
+  embed         into the page   server.rs     ratings, OPDS
+html.rs       markup helpers    db.rs         SQLite
+lock.rs       one writer at a time (flock on <database_path>.lock)
+```
+
+Two modules are worth knowing about before you go looking for their contents.
+`src/html.rs` holds the generic markup helpers — tag scanning, escaping, entity
+decoding, XHTML fixups — that extraction, images, comments and the world briefing
+all need; put anything that works on markup without caring what the markup is
+*about* there. `src/images/` owns every stage of an article's images, which is
+otherwise the kind of concern that smears itself across extraction and EPUB
+building; see its module docs for the order the stages run in.
+
+### Auditing images against real articles
+
+Image handling fails in ways no synthetic fixture predicts, because every
+publisher invents its own lazy-loading scheme. `examples/image_audit.rs` replays
+the extraction and image pipeline over the articles of issues already published
+and counts what actually reaches the page:
+
+```sh
+cargo run --release --example image_audit -- \
+    --cache /tmp/pagecache ~/bookorbit/books/daily-epub/*.epub
+```
+
+It prints per-article `refs / embedded / shown / placeholders`, a tally of loss
+reasons, and totals. Pages are cached on first run, so a change can be measured
+against byte-identical input; `--dump <title substring>` lists the URLs one
+article resolved to, and `--epub-out DIR` writes a readable EPUB of the audited
+articles so the images can be looked at on a device rather than counted in a
+table. It needs the network and is not part of `cargo test`.
 
 ---
 
@@ -425,6 +658,10 @@ From spec §7, plus what implementation turned up:
   falls back to matching the feed title.
 - **Images are downloaded once per edition** (the two editions need different
   resolutions and colour profiles), so an image-heavy issue makes two passes.
+- **An image that cannot be embedded is dropped, not announced,** unless its alt
+  text is a real description — decorative rules, spacers and dead links would
+  otherwise litter the page with `[image: …]` lines. Verify image handling with
+  `examples/image_audit.rs` after touching extraction.
 - **`dc:date` rides inside a `dcterms:date` metadata fragment** because
   `epub-builder` neither exposes `dc:date` nor accepts a non-`chrono` date. The
   OPF output is correct; the mechanism is a workaround.
@@ -433,8 +670,33 @@ From spec §7, plus what implementation turned up:
   stack, so `curate/llm.rs` speaks the OpenAI-compatible wire protocol over the
   shared `reqwest` client instead, behind a `ChatBackend` trait. The dependency
   was removed.
-- **Only DeepSeek is wired.** Another provider means another `ChatBackend` impl.
+- **Two wire protocols are implemented**, `openai` (chat completions) and
-- **No embedding-based personal ranker yet** (spec §3.9 future work); the schema
+  `anthropic` (Messages API), each a `ChatBackend` impl. Providers are config
-  is ready for it once ~200 ratings exist.
+  entries over those two kinds, each with its own `UsageMeter`, price table and
+  ceiling; a protocol that is neither means another impl. Voyage AI embeddings
+  sit behind the analogous `EmbeddingBackend` trait in `curate/embedding.rs`.
+- **Triage and union admission replace the heuristic gate.** Every eligible
+  article gets interest, rated-neighbour, feed-affinity, social and heuristic
+  signals, then DeepSeek reads its opening (up to `triage_max`). The deep set is
+  the union of triage, interest, neighbour, exploration, blend and auto-include
+  retrievers. `explain` shows the assessment and `admitted_by`. Learned signals
+  stay absent until their gates open (8 and 15 ratings respectively).
+- **Deep assessment and diversity are live.** DeepSeek reads a representative
+  beginning/middle/end sample, separates editorial quality from reader fit, and
+  records descriptive facets. Utility is normalized over the deep set; embedding
+  leader clusters cap near-duplicates before the 60-item editor shortlist.
+- **DeepSeek's content filter rejects whole batches.** A `400 Content Exists
+  Risk` refuses the entire request when any one article in it trips the input
+  filter, without saying which. Triage and deep assessment therefore bisect a
+  rejected batch (halves, then quarters, down to single articles) so the other
+  articles keep their assessment. A single article the bulk provider still
+  refuses is retried once on the editor provider with the same prompt when that
+  is a different provider; if that also fails, an `article_assessments` row with
+  `kind = 'provider_rejected'` and a NULL score is written so the article is not
+  sent again for `assessment_reuse_days`, and it is ranked on its other signals.
+  See them with `explain` (`triage: rejected by provider — deepseek: …`), the
+  `triage:` / `assess:` log lines (`… 3 rejected (2 recovered on gemini)`), the
+  ` · N rejected` suffix on the `curation:` line, or
+  `sqlite3 /var/lib/daily-epub/daily-epub.db "select stage, count(*) from article_assessments where kind = 'provider_rejected' group by stage"`.
 - **One reader, one issue per day.** There is no multi-user support and no
   weekly/retrospective edition (spec §6).

config.example.toml

@@ -3,17 +3,19 @@
 # Load order (later wins): built-in defaults ← this file ← `DAILY_EPUB_*` env vars.
 # Nested keys use a double underscore in env vars, e.g.
 #   DAILY_EPUB_MINIFLUX__API_KEY=...
-#   DAILY_EPUB_DEEPSEEK__API_KEY=...
+#   DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY=...
+#   DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY=...
+#   DAILY_EPUB_PROVIDERS__GEMINI__API_KEY=...
+#   DAILY_EPUB_VOYAGE__API_KEY=...
 #   DAILY_EPUB_SERVER__HMAC_SECRET=...
 #   DAILY_EPUB_LOOKBACK_HOURS=30
+#   DAILY_EPUB_LLM__EDITOR=gemini      # one-off role override, no file edit
 
 timezone = "America/New_York"
 lookback_hours = 26
 target_article_count = 20
-prefilter_keep = 120
 retention_days = 21                  # EPUBs, by age
 xtc_retention_count = 5              # XTC issues, by count (~80-100 MB each)
-max_daily_usd = 2.0
 world_briefing = true
 
 # SQLite database file. Parent directories are created on demand.
@@ -22,7 +24,8 @@ database_path = "/var/lib/daily-epub/daily-epub.db"
 # Default output directory for generated artifacts (overridden by `--out`).
 out_dir = "/var/lib/daily-epub/out"
 
-# Path to the Scour interests OPML used to seed the taste profile (§3.6).
+# Hand-maintained reader profile and Scour interests merged into the system prompt.
+profile_path = "data/profile.md"
 interests_opml = "data/scour-interests.opml"
 
 [miniflux]
@@ -30,19 +33,85 @@ base_url = "http://127.0.0.1:8082"
 # api_key via DAILY_EPUB_MINIFLUX__API_KEY env
 page_limit = 250
 
-[deepseek]
+# The two LLM roles, each assigned to a provider declared in [providers.*]
+# below. Switching the editor to Gemini is `editor = "gemini"` plus its key in
+# the env file; nothing else changes. An empty editor ("") runs everything on
+# the bulk provider. Every editor call degrades to bulk when the editor's key
+# is missing, its daily ceiling is hit, or the API refuses/fails.
+[llm]
+bulk = "deepseek"                    # triage, deep assessment, and every fallback
+editor = "anthropic"                 # lineup, summaries, The Brief, the weekly profile rebuild
+triage_batch_size = 25               # articles per first-pass triage request
+deep_batch_size = 8                  # articles per close-reading assessment request
+score_temperature = 0.3              # sent only by providers that take a temperature (kind = "openai")
+editorial_temperature = 0.8          # summaries and The Brief on an openai-kind provider
+
+# The provider registry. Any number of entries; a role above names one by its
+# table name. Keys never live here: DAILY_EPUB_PROVIDERS__<NAME>__API_KEY.
+# `max_daily_usd` is a per-provider runaway guard per UTC day (0 = none), not
+# accounting — set hard spend limits in each provider's dashboard as well.
+# Prices are USD per 1M tokens and only feed the guard's arithmetic.
+[providers.deepseek]
+kind = "openai"                      # openai | anthropic
 base_url = "https://api.deepseek.com/v1"
 model = "deepseek-v4-flash"          # DeepSeek-V4-Flash-0731 (confirmed 2026-08-15)
-# api_key via DAILY_EPUB_DEEPSEEK__API_KEY env
+# api_key via DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY env
-score_batch_size = 12
+max_daily_usd = 2.0
-score_temperature = 0.3
+max_concurrent_requests = 4          # triage and deep-assessment batches in flight
-editorial_temperature = 0.8
-# USD per 1M tokens, used for the cost guardrail.
 price_input_per_mtok = 0.14
-price_cached_input_per_mtok = 0.0028
+price_cache_read_per_mtok = 0.0028   # prefix-cache hits
+price_cache_write_per_mtok = 0.0     # DeepSeek caches implicitly, no write charge
 price_output_per_mtok = 0.28
 
+# Claude Opus 5 over the Messages API. Requests carry `output_config.effort`,
+# a cached system block, and `fallbacks = "default"` so a classifier refusal is
+# re-routed server-side; a refusal that still comes back degrades to bulk.
+[providers.anthropic]
+kind = "anthropic"
+base_url = "https://api.anthropic.com"
+model = "claude-opus-5"
+# api_key via DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY env
+effort = "high"                      # low | medium | high | xhigh | max → output_config.effort
+max_daily_usd = 3.0
+max_concurrent_requests = 4          # summaries in flight when this is the summary provider
+price_input_per_mtok = 5.0
+price_cache_read_per_mtok = 0.5
+price_cache_write_per_mtok = 6.25
+price_output_per_mtok = 25.0
+
+# Gemini 3.8 Flash through Google's OpenAI-compatible endpoint (beta, verified
+# 2026-09-02). Declared but unreferenced until a role names it.
+[providers.gemini]
+kind = "openai"
+base_url = "https://generativelanguage.googleapis.com/v1beta/openai"
+model = "gemini-3.8-flash"
+# api_key via DAILY_EPUB_PROVIDERS__GEMINI__API_KEY env
+effort = "high"                      # minimal | low | medium | high → reasoning_effort
+max_daily_usd = 3.0
+max_concurrent_requests = 4
+price_input_per_mtok = 0.75          # promotional through 2026-12-31; $1.50 from 2027-01-01
+price_cache_read_per_mtok = 0.075    # implicit cache hits; $0.15 from 2027-01-01
+price_cache_write_per_mtok = 0.0
+price_output_per_mtok = 3.75         # includes thinking tokens; $7.50 from 2027-01-01
+
+# Voyage AI embeddings behind the interest and rated-neighbour signals. Set
+# `enabled = false` (or leave the key unset) and the paper still builds: the
+# learned signals are simply absent, never a penalty.
+[voyage]
+enabled = true
+base_url = "https://api.voyageai.com/v1"
+model = "voyage-4-lite"
+# api_key via DAILY_EPUB_VOYAGE__API_KEY env
+output_dimension = 512               # 256 | 512 | 1024 | 2048
+batch_size = 32
+max_concurrent_requests = 4
+max_input_chars = 60000              # per article, cut on a char boundary
+max_daily_usd = 0.50                 # runaway guard ($0.02 / M tokens)
+
 [curation]
+max_article_count = 28               # hard ceiling; there is no minimum (§13)
+recent_rejection_days = 7
+recent_rejection_floor = 3.0
 always_include_feeds = []            # miniflux feed ids or site urls
 blocked_domains = []
 # Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, …).
@@ -59,6 +128,65 @@ sections = [
   "From the Blogroll",
 ]
 
+[curation.feedback]
+loved_value = 1.0
+good_value = 0.35
+not_for_me_value = -1.0
+verdicts_in_prompt = 60
+
+# Every weight, quota, gate and threshold of the personalized ranker. The
+# learned signals (`knn`, `feed`) contribute nothing until their gates open:
+# the weight ramps linearly from `*_floor` to `*_full` rated articles.
+[curation.ranking]
+triage_max = 800                     # eligible articles the triage LLM reads
+deep_keep = 120                      # deep-assessment set
+shortlist_keep = 60                  # what the editor sees
+assessment_reuse_days = 3
+rating_lookback_days = 180
+rating_half_life_days = 60
+neighbour_k = 5
+negative_coefficient = 0.75
+knn_floor = 8
+knn_full = 25
+feed_floor = 15
+feed_full = 40
+semantic_min_words = 300
+exploration_slots = 5
+embedding_retention_days = 120       # `features prune`: unrated, unpublished vectors
+telemetry_retention_days = 180       # `features prune`: candidate_runs and article_assessments
+
+[curation.ranking.quotas]
+triage = 60
+interest = 20
+knn = 20
+
+# Weights need not sum to 1; they are renormalized over the present signals.
+[curation.ranking.weights.preliminary]
+interest = 0.35
+knn = 0.25
+heuristic = 0.20
+feed = 0.10
+social = 0.10
+
+[curation.ranking.weights.utility]
+quality = 0.40
+fit = 0.20
+knn = 0.15
+interest = 0.10
+feed = 0.05
+triage = 0.05
+social = 0.03
+heuristic = 0.02
+
+[curation.ranking.diversity]
+cluster_threshold = 0.85
+per_cluster_cap = 2
+utility_protected = 10
+
+[editorial]
+summary_model = "editor"             # editor | bulk — which [llm] role writes the summaries
+summary_input_tokens = 3000          # article text offered per summary
+
 [publish]
 # Where both EPUB editions land, and what the OPDS feed lists. BookOrbit is
 # optional — it just watches this folder if you run it.

data/profile.md

@@ -0,0 +1,30 @@
+# Reader profile
+
+## Who he is
+A software engineer in the Boston area who reads on e-ink in the morning. He would
+rather read six excellent long pieces than thirty adequate short ones. He reads across
+an unusually wide range of subjects and does not need a topic to be professionally
+useful to enjoy it.
+
+## What he wants
+- Long-form and high-effort above all: essays, deep dives, post-mortems, field notes,
+  annotated experiments, thorough explainers, personal narratives with real specificity.
+  Length is a proxy, not the goal.
+- Any topic, if the writing is excellent.
+- Social proof is evidence a critical audience read it, not a verdict.
+- Boston and New England local news: city government, transit, universities, civic stories.
+- Ultra-niche community news: small scenes with their own vocabulary.
+- World and US news kept light and neutral (the World Briefing covers it separately).
+
+## What he does not want
+Press releases and funding announcements dressed as news; SEO listicles; link roundups;
+changelogs without analysis; sponsored content; crypto and engagement bait; rewrites of a
+story he can read at the source; culture-war outrage; one paragraph stretched to five.
+
+## How to judge
+Would he still be glad he read this an hour later? Reward specificity, first-hand
+experience, honest uncertainty, and prose with a human behind it. Penalize padding,
+unsourced confidence, and summaries of other people's work.
+
+## Interests
+(optional: one per line; merged with data/scour-interests.opml)

docs/plans/2026-08-15-implementation-notes.md

@@ -1,6 +1,8 @@
 # Implementation notes (shared brief for all implementation agents)
 
 Authoritative spec: `docs/plans/2026-08-15-the-daily-epub.md`. Read it fully before writing code.
+For curation (§3.5, §3.6 and §3.9 of that spec) the authority is now
+`docs/plans/2026-09-02-personalized-curation-v2.md`; see "Curation v2" below.
 This file records implementation-time decisions and verified external facts. Follow both.
 
 ## Verified external facts (2026-08-15)
@@ -68,6 +70,47 @@ This file records implementation-time decisions and verified external facts. Fol
   walks back up to `MAX_LOOKBACK_DAYS` and the section is datelined with the day it actually
   covers, not the masthead date.
 
+## Verified external facts (2026-09-02, curation v2)
+
+- **Anthropic Messages API** (verified 2026-09-02 against the bundled Claude API reference):
+  `POST https://api.anthropic.com/v1/messages` with headers `x-api-key`,
+  `anthropic-version: 2023-06-01`, `content-type: application/json` and
+  `anthropic-beta: server-side-fallback-2026-07-01`. Model id `claude-opus-5`; pricing
+  **$5.00 / M input, $25.00 / M output**, cache reads 0.1× input ($0.50/M), cache writes
+  1.25× ($6.25/M); the minimum cacheable prefix is 512 tokens. **No sampling parameters**
+  (`temperature`, `top_p`, `top_k` are a 400) and no `thinking` block — adaptive thinking is on
+  by default and depth is set with `output_config: {"effort": "high"}` (`low | medium | high |
+  xhigh | max`). The system prompt goes in `system: [{type: "text", text, cache_control:
+  {type: "ephemeral"}}]`; no assistant prefill, JSON is asked for in the prompt and parsed
+  tolerantly. `"fallbacks": "default"` (with the beta header) routes a request the safety
+  classifiers would refuse to a fallback model server-side; a response can still end with
+  `stop_reason: "refusal"` on HTTP 200, which the code treats as an error that degrades the
+  call to DeepSeek. Usage fields: `input_tokens` (uncached remainder),
+  `cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`. Timeout 300 s;
+  retry 429/5xx/network, never 400. Key only from `DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY`
+  (the provider registry below; the pre-registry `DAILY_EPUB_ANTHROPIC__API_KEY` is a startup
+  error).
+- **Gemini 3.8 Flash over the OpenAI-compatible endpoint** (beta, verified 2026-09-02):
+  `POST https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` with
+  `Authorization: Bearer <key>`, the standard `messages` / `temperature` /
+  `response_format: {"type": "json_object"}` body. Model id `gemini-3.8-flash`. Reasoning depth
+  is the OpenAI `reasoning_effort` field, which Google maps onto Gemini 3.x's `thinking_level`
+  (`minimal | low | medium | high`; `none` is not accepted by 3.x models). Usage: implicit
+  cache hits are reported in `prompt_tokens_details.cached_tokens` (the same field DeepSeek
+  now fills), and `completion_tokens` already includes the thinking tokens that
+  `completion_tokens_details.reasoning_tokens` breaks out — so output is priced from
+  `completion_tokens` alone, never the sum. Prices per 1M tokens (promotional through
+  2026-12-31): **$0.75 input, $0.075 cache read, $3.75 output** (thinking included); from
+  2027-01-01 **$1.50 / $0.15 / $7.50**. No cache-write charge. Key only from
+  `DAILY_EPUB_PROVIDERS__GEMINI__API_KEY`. Shipped as `[providers.gemini]`, unreferenced until
+  a role names it.
+- **Voyage AI embeddings** (verified 2026-09-02): `POST https://api.voyageai.com/v1/embeddings`
+  with `Authorization: Bearer <key>`; body `{input: [...], model: "voyage-4-lite", input_type:
+  "document" | "query", truncation: true, output_dimension: 512, output_dtype: "float"}`. Up
+  to 1,000 inputs and 1M tokens per request, 32k tokens per input. Vectors are
+  unit-normalized, so dot product = cosine. **$0.02 / M tokens** after a 200M-token free
+  allocation. Key only from `DAILY_EPUB_VOYAGE__API_KEY`.
+
 ## Cross-cutting implementation decisions
 
 1. **sqlx usage**: use *runtime* queries (`sqlx::query(...).bind(...)`) and manual row mapping
@@ -80,12 +123,32 @@ This file records implementation-time decisions and verified external facts. Fol
    Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
 4. **HTTP**: one shared `reqwest::Client` (rustls, gzip, no cookies, 10s timeouts, UA
    `the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`), passed by clone.
-5. **LLM**: `async-openai` with custom base URL. All LLM calls go through `curate/llm.rs`
+5. **LLM**: a hand-rolled `reqwest` client, not `async-openai` (the published crate exposes
-   `LlmClient` which tracks token usage into a shared `UsageMeter` (input/cached/output tokens,
+   neither `Client` nor `CreateChatCompletionRequest` at the pinned version). Every LLM call
-   cost usd) and enforces `max_daily_usd`.
+   goes through `curate/llm.rs`: `LlmClient { provider, system_prompt, model, effort,
+   max_concurrent_requests, meter, backend, retry }` over the `ChatBackend` trait, with two
+   wire protocols — `OpenAiCompatibleBackend` (`{base_url}/chat/completions`, bearer key,
+   `response_format: json_object`, `reasoning_effort` when the provider has an `effort`) and
+   `AnthropicBackend` (Messages API, facts above). **Providers are config, not code**: the
+   `[providers.<name>]` registry (`kind = openai | anthropic`, `base_url`, `model`, `effort`,
+   `max_daily_usd`, `max_concurrent_requests`, `price_*`) is a `BTreeMap<String,
+   ProviderConfig>`, and `[llm] bulk = "<name>"` / `editor = "<name>"` assign the two roles by
+   name (`editor = ""` means everything runs on bulk; both roles on one provider share one
+   client and one ceiling). `LlmClient::for_provider(name, &cfg, ..)` dispatches on `kind`;
+   `Llms::from_config(&config, prompt, &meters)` builds the roles; `editor_or_bulk()` degrades
+   to bulk when the editor client is missing or its meter is tripped. One `UsageMeter` per
+   *referenced* provider (`llm::provider_meters`), keyed by provider name — the same key used
+   for `runs.provider_costs_json`, the UTC-day spend preload and the log lines — plus Voyage's
+   own. `LlmClient.provider` is the config name, never the kind. The system prompt is sent
+   first and byte-identical within a run so every provider's prefix cache hits. Keys come only
+   from `DAILY_EPUB_PROVIDERS__<NAME>__API_KEY` (figment lower-cases the path, so provider
+   names are `[a-z0-9_]+`); `daily-epub config check` prints the resolved roles without
+   opening the database.
 6. **Testing**: unit tests inline per module; integration tests in `tests/` over fixture JSON in
-   `tests/fixtures/`. Never hit the network in tests. LLM stage mockable via `--skip-llm`
+   `tests/fixtures/`. Never hit the network in tests: `MockBackend` (`ChatBackend`) and the
-   (prefilter order used for selection, feed excerpts as summaries).
+   embedding mock (`EmbeddingBackend`) stand in for all three providers. `--skip-llm` makes
+   zero LLM calls (admission by cheap signals, `select_without_llm` by utility, excerpt
+   summaries); `--skip-embeddings` makes zero Voyage calls.
 7. **Style**: rustfmt defaults, `cargo clippy` clean-ish, no `unwrap()` outside tests, tracing
    spans per pipeline stage.
 8. **File ownership**: waves of agents work in parallel on disjoint files. Do not edit files
@@ -99,3 +162,62 @@ This file records implementation-time decisions and verified external facts. Fol
     `style-x4.css`). Askama 0.12+ configured via `askama.toml` if needed.
 12. **Determinism**: chapter ids `art-{entry_id}`, stable filenames, issue regeneration for the
     same date replaces prior rows/files (idempotent upsert everywhere).
+
+## Curation v2 (2026-09-02)
+
+The personalized ranker is specified in `docs/plans/2026-09-02-personalized-curation-v2.md`
+(§0 settled decisions, §3 target pipeline, §19 configuration, §21 the seven landed steps);
+`docs/plans/2026-09-02-curation-v2-progress.md` records per-step deviations. Facts an
+implementer needs that are easy to get wrong:
+
+- **Tables** (`migrations/0002_curation_v2.sql`, `0003_drop_scores.sql`; never edit
+  `0001_init.sql`): `rating_events` (append-only; the current verdict is the latest
+  `explicit` event), `article_embeddings` and `interest_embeddings` (f32 little-endian BLOBs,
+  `input_hash` = sha256 of the embedded text), `article_assessments` (`stage IN ('triage',
+  'deep')`, reused while `model` and `prompt_version` match and `assessed_at` is within
+  `assessment_reuse_days`; `--rescore` ignores the cache), `candidate_runs` (one row per
+  considered article per run, upserted with every column set on each stage transition),
+  `runs.config_json` / `runs.provider_costs_json`, `issue_articles.why`. `ratings`,
+  `feed_priors` and `scores` are dropped; `kv` keeps `ingest_watermark`, `taste_profile`,
+  `taste_profile_learned`, `profile_version`.
+- **Feedback**: `Vote` is `loved | good | down` (`NotForMe`); the HMAC message stays
+  `{issue_date}/{article_id}/{vote}`. `Vote::parse("up")` → `Loved` and `auth::verify_token`
+  still accepts tokens signed over the literal `up` segment because published issues carry
+  those links. Keep both.
+- **Budget day**: each provider's `UsageMeter` is preloaded with the spend of earlier runs on
+  the **UTC date of the run's `started_at`**, summed from `runs.provider_costs_json`
+  (`db::spend_for_date` by nominal issue date is gone). A tripped meter skips that provider's
+  remaining calls; the paper always publishes.
+- **Lock**: `src/lock.rs` takes `libc::flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`
+  for `generate`, `profile rebuild`, `features backfill` and `backfill-social`; a second
+  writer exits with "<command> is already running". `serve`, `explain`, `stats`, `ratings`,
+  `features prune` and `db migrate` never take it.
+- **Retention**: `telemetry::prune` removes `article_embeddings` of unrated, unpublished
+  articles older than `embedding_retention_days` (120) and `candidate_runs` rows plus
+  `article_assessments` older than `telemetry_retention_days` (180). `features prune` runs it
+  on demand; `generate` runs it once after publishing, best effort.
+- **Keys**: `DAILY_EPUB_PROVIDERS__<NAME>__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` map onto
+  `ProviderConfig.api_key` / `VoyageConfig.api_key` through figment; the fields exist only
+  for that mapping and are never documented in TOML, logged, or stored
+  (`Config::providers_redacted()` is what reaches `runs.config_json`).
+- **Provider registry** (2026-09-02, after step 7): the `[deepseek]` and `[anthropic]` tables
+  and the top-level `max_daily_usd` are gone. `[llm]` holds the role names and the role-level
+  knobs (`triage_batch_size`, `deep_batch_size`, `score_temperature`,
+  `editorial_temperature`); `[providers.deepseek]`, `[providers.anthropic]` and
+  `[providers.gemini]` ship in `config.example.toml` and are `Config::default()` key for key.
+  Stale shapes fail at load, naming the new key: a `[deepseek]`/`[anthropic]` header, a
+  top-level `max_daily_usd`, any of the four role keys outside `[llm]`, or a
+  `DAILY_EPUB_DEEPSEEK__*` / `DAILY_EPUB_ANTHROPIC__*` environment variable. Batching
+  concurrency (`triage`, `assess`) is the bulk provider's `max_concurrent_requests`; the
+  summaries fan out at the summary provider's. `Models { bulk, editor, summaries }` in the
+  colophon and Behind the paper stay model ids taken from the built clients.
+- **Provider rejections** (2026-09-02): `curate::batch` bisects a triage or deep batch that
+  fails with a non-transient error (`Api`, `Refusal`, `EmptyResponse`, or a response that
+  parses to zero items) down to single articles; a rejected single is retried once on the
+  editor client when it is another provider. An article both refuse gets an
+  `article_assessments` row with `kind = 'provider_rejected'`, `score`/`fit` NULL,
+  `rationale = "<provider>: <message ≤ 200 chars>"`, the bulk `model` and the stage's
+  `prompt_version`; the cache loaders skip it (leaving the assessment absent) while it is within
+  `assessment_reuse_days`, `--rescore` ignores it, and `admit::hygiene` never treats it as a low
+  score. Because a recovered article's row carries the editor's model, reuse accepts rows whose
+  `model` is either configured model (`triage::reusable_models`).

docs/plans/2026-09-02-curation-v2-progress.md

@@ -0,0 +1,139 @@
+# Personalized Curation v2 — implementation progress and handoff
+
+**Updated:** 2026-09-02 (end of session 2)
+**Plan:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§21 is the step sequence)
+**Branch:** `curation-v2` (branched from `main` at `a599745`; not merged, not pushed)
+**Briefs:** `docs/plans/curation-v2-briefs/` — `00-preamble.md` + one `stepN.md` per step. Each
+brief was handed to an implementation agent as `cat 00-preamble.md stepN.md`.
+
+## Where things stand
+
+| Step (§21) | Status | Commit |
+|---|---|---|
+| 1. Feedback and profile | **done**, reviewed (codex review: no actionable issues) | `3a9f4b9` |
+| 2. Claude editor and editorial | **done** | `57efbb4` |
+| 3. Embeddings, signals, telemetry | **done** (branch `curation-v2-step3`, merged) | `ea3b141` + merge commit |
+| 4. Triage replaces the gate | **done** (Codex plugin, high effort), reviewed | `10f4afd` |
+| 5. Deep assessment, utility, diversity | **done** (Codex started, cut off by quota; Claude agent finished), reviewed | `05a74a0` |
+| 6. Paper telemetry, stats, lock | **done** (Claude agent), reviewed | `d261cd4` |
+| 7. Cleanup + implementation notes | **done** (Claude agent), reviewed | `d403c51` |
+
+**Post-plan addition (same day):** a provider-agnostic LLM registry — `[llm]` roles (`bulk`,
+`editor`) over named `[providers.*]` entries of `kind = openai | anthropic`, keys from
+`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY`, a `gemini` entry (Gemini 3.8 Flash via Google's
+OpenAI-compatible endpoint) declared but unreferenced, and a `daily-epub config check` subcommand.
+The old `[deepseek]`/`[anthropic]` tables, top-level `max_daily_usd` and the old key env vars
+fail loudly. The server upgrade is written up in `docs/runbooks/curation-v2-migration.md`.
+
+All seven plan steps are implemented. `cargo fmt --check`, `cargo clippy --all-targets`
+(including `-W dead_code`) and `cargo test` (329 lib tests + 6 bin tests + the 7 integration suites) are green at HEAD.
+
+Nothing has been run against the production database or the real providers. No local
+`config.toml`, database, or API keys exist on the dev box, so verification so far is tests only.
+
+## Decisions and deviations made while implementing
+
+Steps 1–3 (session 1):
+
+- **Legacy `up` links.** `Vote::parse("up")` → `Loved`, and `auth::verify_token` also accepts tokens
+  signed over the literal `up` segment so already-published issues keep working. Keep both.
+- **Same-date regeneration** no longer excludes its own picks (`published_before` uses issue dates
+  strictly before the run date), per plan §8.1.
+- **`VoyageConfig.api_key`** exists as a field (figment maps `DAILY_EPUB_VOYAGE__API_KEY` into it;
+  `deny_unknown_fields` would otherwise reject the env var). Never document it in TOML.
+- **`anthropic.max_concurrent_requests`** is validated but not consumed; summary concurrency is
+  the constant `SUMMARY_CONCURRENCY = 4` in `editorial.rs`.
+- `RATINGS_LOOKBACK_DAYS` in `profile/mod.rs` is effectively unbounded (36,500) for the prompt
+  verdict block and the weekly rebuild; the knn/feed preference state uses
+  `curation.ranking.rating_lookback_days` (180) as the plan says.
+- Footer CSS: `.rating` has no `white-space: nowrap` (it would clip on narrow e-ink screens).
+
+Step 4:
+
+- Hygiene order is blocked → published_before → recently_rejected, all skipped for auto-includes.
+- `admitted_by` records every retriever whose own top-N would have taken the article (first entry
+  is the admitting one); exploration is flagged only when `exploration` admitted it.
+- Under the pool cap (`triage_max`) the surplus gets `stage='eligible'`, `excluded_reason='not_admitted'`.
+- Migration `0003_drop_scores.sql` drops `scores`; the churn rule reads `article_assessments`.
+
+Step 5:
+
+- `Candidate` is the only flow type; `ScoredArticle`, `combined_score`, `score.rs`, `select.rs` are gone.
+- The editor omits the `facets:` line when every facet is unknown (plan renders it unconditionally).
+- Articles cut at the shortlist keep the stage they reached (`admitted` if never deep-assessed,
+  else `assessed`) with `excluded_reason` `cluster_suppressed` (hit the base cap) or `shortlist_cap`.
+- `Curator::assess` is a no-op under `--skip-llm`; cached deep rows are still reused when DeepSeek
+  is merely down.
+- `--near-misses` and the Behind-the-paper list order by utility, falling back to the preliminary
+  blend per row.
+- Fixtures: `tests/fixtures/deepseek_deep_batch{,_messy}.json` replace the old score fixtures;
+  `deepseek_triage_batch.json` added in step 4.
+
+Step 6:
+
+- `BehindThePaper` hangs off `Issue.behind` (filled after `build_issue`), not a `build_issue` argument.
+  Generation time and cost shown in the chapter are measured at issue assembly (they exclude the
+  EPUB build and publish, a few seconds); the cost includes Voyage.
+- Voyage spend is recorded under `provider_costs["voyage"]`; `finish()` counts it once and keeps its
+  tokens out of the LLM aggregate. Voyage spend is not preloaded into its meter for the UTC day.
+- `src/lock.rs` uses `libc::flock` (`libc` was already in `Cargo.lock`; now a direct dependency).
+  The holder's command name is written into the lock file for the "X is already running" message.
+- The mid-run `admission:` line is `debug`; the four-line §15.4 block is logged once after
+  `finish()` and printed by `print_report`. Timings `summaries` and `brief` replace `editorial`.
+
+Step 7:
+
+- `features prune` sweeps `article_embeddings`, `candidate_runs` and now `article_assessments`;
+  `generate` runs it once after a published (non-dry) issue, best effort.
+- The Brief chapter's TOC/title string is now "The Brief" (was still "From the Editor").
+- Kept on purpose: `UsageMeter::new(&DeepseekConfig, ..)` (a convenience over `with_prices`, 12
+  call sites); `StageCounts.candidates` (feeds `runs.candidates` and the colophon); the
+  `prefilter_keep`/`score_batch_size` removed-key startup errors (operator guards); the SQL comment
+  "filled in by step 4/5" inside the applied migration `0002` (sqlx checksums); six v1-era
+  unreferenced `pub` items (`enrich_one`, `issues_before`, `today_in_tz`, `to_json_pretty`,
+  `with_retry_policy`, `db::issues_before`) as pre-v2 API surface.
+- `config::tests::shipped_example_config_matches_the_defaults_key_for_key` compares the example
+  against `Config::default()` in both directions (numbers within 1e-6).
+
+## Operator to-dos before the first real run
+
+1. Set `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` in the systemd env file.
+2. Set hard spend limits in the DeepSeek, Anthropic and Voyage dashboards (the meters are runaway
+   guards, not accounting).
+3. Copy `data/profile.md` to wherever `profile_path` points on the server (default is relative to
+   `WorkingDirectory=/var/lib/daily-epub`, like `data/scour-interests.opml`).
+4. Back up the DB, then `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`, 0003 drops `scores`).
+5. `daily-epub features backfill --rated-only` then `--days 30` to warm the embedding cache.
+6. `daily-epub generate --dry-run`, read the paper (including the new Behind-the-paper chapter),
+   then `explain --date … --near-misses` and `stats`.
+7. Watch the first few real runs' `providers:` log line against the ~$0.80/day estimate (plan §3).
+
+## How the work was run (so the next session can repeat it)
+
+- Orchestrator: Claude Code (this repo), one implementation agent per step, review + commit by the
+  orchestrator after independent `cargo fmt/clippy/test`.
+- **Codex via the `openai-codex` Claude Code plugin (v1.0.6) works on this host as of session 2**
+  (the operator fixed bubblewrap). Launch:
+  `node ~/.claude/plugins/cache/openai-codex/codex/1.0.6/scripts/codex-companion.mjs task --background --write --effort high "$(cat brief.md)"`,
+  poll `… status <job-id> --json` (`.job.status`), read `… result <job-id>`. Inside its sandbox
+  `bind()` on 127.0.0.1 is forbidden, so exactly ten pre-existing loopback tests fail there
+  (`curate::llm::tests::anthropic_*` ×4, `extract::tests::relative_urls_resolve_against_the_url_we_landed_on`,
+  `server::tests::*` ×5, plus the two `m7_server` integration tests); the brief must tell the agent
+  to ignore those and the orchestrator runs the full suite outside. The old "host quirk" paragraph
+  in `00-preamble.md` is obsolete.
+- Codex usage budget is roughly one large step per ChatGPT usage window: in session 2 it finished
+  step 4 (~26 min) and was cut off 21 minutes into step 5 ("try again at 7:44 PM"). The job status
+  becomes `failed` with the partial work left in the tree; a Claude `general-purpose` agent with a
+  "finish, don't restart" brief (known failures, scope audit checklist) completed it. Steps 6 and 7
+  ran on Claude agents directly.
+- Steps 4 → 5 → 6 → 7 were sequential, each reviewed against the plan sections named in its brief
+  before committing.
+
+## Next session: what remains
+
+1. `git checkout curation-v2 && cargo test` (expect green). Follow
+   `docs/runbooks/curation-v2-migration.md` on the server.
+2. Optional: a Codex `review --background --scope branch --base main` pass over the whole branch.
+3. `git merge --no-ff curation-v2` into `main`, build, deploy, and do the operator to-dos above.
+4. After a week of real runs: read `stats`, tune `[curation.ranking]` from what `explain` shows,
+   and revisit the deferred items in plan §23.

docs/plans/curation-v2-briefs/00-preamble.md

@@ -0,0 +1,58 @@
+# Implementation agent brief — The Daily EPUB, "Personalized Curation v2"
+
+You are implementing ONE step of a multi-step plan in the Rust repository at
+`/home/thallada/workspace/the-daily-epub` (branch `curation-v2`, already checked out).
+
+The orchestrator hands you `docs/plans/curation-v2-briefs/00-preamble.md` (this file) followed by one
+`stepN.md` from the same directory; `docs/plans/2026-09-02-curation-v2-progress.md` records what has
+landed so far.
+
+The plan is `docs/plans/2026-09-02-personalized-curation-v2.md`. Read it in full before
+writing code: §0 records settled decisions (do not reopen them), §1 lists the files to
+read first, §21 is the implementation sequence. Also read
+`docs/plans/2026-08-15-implementation-notes.md` §"Cross-cutting implementation decisions"
+(note: item 5 is stale — the code uses a hand-rolled `reqwest` client in `src/curate/llm.rs`,
+keep that).
+
+## Hard rules
+
+- Implement ONLY the step assigned below. Later steps land separately; do not start them.
+  Where this step needs a type or table that a later step fills in, create it now exactly as
+  the plan specifies and leave it empty/unused with a short `// filled in by step N` comment.
+- Conventions: sqlx *runtime* queries (`sqlx::query(...).bind(...)`, never the `query!` macros),
+  `jiff` for time, RFC3339 UTC strings in SQLite, `thiserror`/`anyhow` error style, no `unwrap()`
+  outside tests, tracing spans per stage, rustfmt defaults, askama templates.
+- Tests never touch the network. Use mock backends following `MockBackend` in `src/curate/llm.rs`.
+- Follow the plan's names for modules, functions, config keys, table and column names, enum
+  strings and prompt constants exactly. If the plan is internally contradictory or impossible
+  for this step, pick the closest behaviour, keep the paper publishable, and describe the
+  deviation in your final report. Do not invent scope beyond the plan.
+- API keys only from env vars; never in config files, logs, tests or the database.
+- Prefer existing dependencies in `Cargo.toml`. Add a new crate only if there is no reasonable
+  way without it, and say so in the report.
+- Do NOT commit and do NOT create branches. Leave all changes in the working tree; the
+  orchestrator reviews and commits. Do not edit anything under `docs/`.
+- Keep `config.example.toml` and README in sync with any config keys you add or remove
+  (the test `shipped_example_config_parses` must pass).
+- Before finishing, run and make pass: `cargo fmt`, `cargo clippy --all-targets` (no new
+  warnings), `cargo test`. Fix what you broke; do not delete or `#[ignore]` tests to get green
+  unless the plan removes the feature they cover (then move/replace the tests as §18 says).
+
+## Final report
+
+End with a concise report (this is what the orchestrator reads): what you implemented, every
+deviation from the plan and why, anything from the step you could not finish, and the exact
+`cargo test` summary line(s). Keep it under 60 lines.
+
+## Host notes
+
+- The `openai-codex` Claude Code plugin sandbox works on this host (bubblewrap fixed 2026-09-02):
+  `apply_patch`, the shell, `cargo build/test/clippy` and the warm `~/.cargo/registry` all work.
+- Inside that sandbox `bind()` on 127.0.0.1 is forbidden, so ten pre-existing tests fail there for
+  environmental reasons: the four `curate::llm::tests::anthropic_*` tests,
+  `extract::tests::relative_urls_resolve_against_the_url_we_landed_on`, and the five
+  `server::tests::*` (the two `tests/m7_server.rs` tests likewise). They are out of scope; do not
+  touch them. Every other test must pass; the orchestrator runs the full suite outside the sandbox.
+- Cargo's registry is already warm; do not add dependencies that would need a network fetch.
+- Work autonomously to completion. Do not stop to ask questions; make the closest-to-plan choice
+  and record it in the final report.

docs/plans/curation-v2-briefs/step1.md

@@ -0,0 +1,73 @@
+
+---
+
+# YOUR STEP: 1 — Feedback and profile (plan §21 step 1)
+
+Scope (plan §6, §7, §8; tests in §20 under Vote, Rating events, Migration, and the prompt):
+
+1. **Migration `migrations/0002_curation_v2.sql`** (§7). Create `rating_events`,
+   `article_embeddings`, `interest_embeddings`, `article_assessments`, `candidate_runs` with
+   the exact columns/indexes in §7; add `runs.config_json`, `runs.provider_costs_json`,
+   `issue_articles.why`. Copy `ratings` rows into `rating_events` per §6.2 (`vote=1 → 'loved',1.0`;
+   `vote=-1 → 'not_for_me',-1.0`; `source='migration'`; `kind='explicit'`; `event_at=rated_at`),
+   then `DROP TABLE ratings; DROP TABLE feed_priors;`.
+   **Orchestrator decision:** do NOT drop `scores` in this step. Stage A scoring
+   (`db::upsert_score`, `recently_low_scored_ids`) still writes/reads it until step 4 replaces
+   it with `article_assessments`; step 4 will add a `0003` migration dropping it. Do not edit
+   `0001_init.sql`. Migration is plain SQL, no Rust bootstrap.
+2. **Three-way `Vote`** (§6.1) in `src/types.rs`: `Loved | Good | NotForMe`, `as_str` →
+   `"loved" | "good" | "down"`, `parse` also accepts legacy `"up"` → `Loved`, `value(&FeedbackConfig)`
+   → configured 1.0 / 0.35 / −1.0. Add `[curation.feedback]` config
+   (`loved_value`, `good_value`, `not_for_me_value`, `verdicts_in_prompt = 60`). Remove `Vote::as_i64`,
+   `Rating`, `FeedPrior`, and everything that depended on `ratings`/`feed_priors`
+   (`db::upsert_rating`, `ratings_with_feed`, `upsert_feed_prior`, `feed_priors`,
+   `profile::rebuild_feed_priors`, the feed-prior term in `prefilter.rs` and `ScoredArticle.feed_prior`
+   — set the prefilter's feed-prior contribution to nothing; the field can go). HMAC links
+   (`src/auth.rs`) must verify for all three votes; message stays `{issue_date}/{article_id}/{vote}`.
+3. **Footer + confirmation page** (§6.1): `RatingLinks` in `src/epub/chapters.rs` and
+   `src/epub/templates/chapter.xhtml` render three links on one line sized for e-ink:
+   `Was this a good pick?   [ Loved it ]   [ Good ]   [ Not for me ]      Read online ↗`.
+   X4 edition still renders no links. `server::handle_rating` appends one `rating_events` row
+   (`kind='explicit'`, `source='epub'`, label `loved|good|not_for_me`, value from config) and
+   returns "Recorded: Loved it — thanks." (etc.) plus the other two links so a mis-tap can be
+   corrected. No feed-prior rebuild, no provider call.
+4. **`rating_events` access** (§6.2) in `src/db.rs`: `append_rating_event(...)`,
+   `db::current_ratings(lookback_days) -> Vec<RatedArticle>` implementing the latest-explicit-event
+   rule (`ORDER BY event_at DESC, id DESC`, `cleared` removes the article from the learned set),
+   joining `articles`, the best entry's feed title, and the most recent `issue_articles.summary`.
+   Move `RatedArticle` to `src/types.rs` with `summary: Option<String>`, `facets: Option<Facets>`
+   (define `Facets` per §12.1 now, all `Option`s, unused until step 5), `note: Option<String>`,
+   plus `RatingEvent`. Also define `RatedArticle.value: f64` and `label`.
+5. **Ratings CLI** (§6.3) in `src/main.rs`: `ratings list [--days 90] [--label loved|good|down|cleared]`,
+   `ratings set --article ID|--url URL --label loved|good|down [--note "..."]`,
+   `ratings clear --article ID|--url URL`. `set`/`clear` append rows with `source='cli'` and
+   `issue_date` from the latest `issue_articles` row for the article if any; `--url` canonicalizes
+   with `dedupe`'s canonical URL function then `db::article_id_for_url`. Print something useful.
+6. **`data/profile.md`** (§8.2): create the file with the initial content shown in §8.2 verbatim;
+   add `profile_path` config (default `data/profile.md`); loader that parses any `## Interests`
+   section (one per line, leading `- ` stripped) and unions it case-insensitively with the OPML
+   interests, passing everything else through verbatim. Remove `STATED_PREFERENCES` and the
+   reader/wants/does-not-want/how-to-judge parts of `PROFILE_PREAMBLE` from code (they now live
+   in the file); the editor-in-chief framing paragraph stays in code. If the file is missing,
+   warn and continue with the OPML interests only.
+7. **System prompt** (§8.4): rebuilt every run in this exact order: framing → profile.md verbatim
+   (minus Interests) → standing interests grouped by `profile::themes::group_into_themes` →
+   learned adjustments → **recent verdicts**: up to `verdicts_in_prompt` most recent explicit
+   ratings, newest first, `LOVED | title | feed | one-line summary` (labels `LOVED`/`GOOD`/`NOT FOR ME`),
+   cleared and duplicate articles removed. Byte-identical within a run. `profile_version` bumps only
+   on the weekly rebuild, as today. `kv` keeps `ingest_watermark`, `taste_profile`,
+   `taste_profile_learned`, `profile_version` (§7.8).
+8. **Weekly rebuild** (§8.3): still on the DeepSeek client this step (the Claude editor client
+   arrives in step 2). Each rated line carries
+   `LOVED | title | feed | summary | facets: format/depth/evidence/technicality/topic_group | note: …`
+   (omit empty parts), up to 200 most recent explicit ratings, cleared excluded; the softened
+   "strong prior, not a rule" instruction; the required "Diversity check:" bullet.
+9. Pipeline: remove the `rebuild_feed_priors` call and every `feed_priors` reference; the paper
+   must still build. `handle_rating` and `profile rebuild` must not touch dropped tables.
+
+Tests to add/adapt (§20): Vote parse/serialize incl. `up`→Loved; HMAC for all three; template
+renders three links standard / none X4; latest explicit event wins; `cleared` removes from the
+learned set; migration on a temp DB through 0001 then 0002 copies old ratings with the right
+labels/values and `ratings`/`feed_priors` are gone; CLI `set`/`clear` append `source='cli'` rows
+(test the db-level function); profile.md Interests parsing and union; system prompt section order
+and verdict block content. Update `tests/m7_server.rs` and `tests/m4_epub.rs` as needed.

docs/plans/curation-v2-briefs/step2.md

@@ -0,0 +1,86 @@
+
+---
+
+# YOUR STEP: 2 — Claude editor and editorial (plan §21 step 2)
+
+Step 1 has landed (three-way votes, `rating_events`, `data/profile.md`, the rebuilt system
+prompt). Read `git log -3` and the current `src/curate/llm.rs`, `select.rs`, `editorial.rs`,
+`pipeline.rs`, `config.rs`, `report.rs`, `db.rs`, `src/epub/chapters.rs` + templates before coding.
+
+Scope (plan §4.1, §4.2, §5, §7.6, §13, §14, §15.1 chapter/in-this-issue `why`, §18 types, §19 config):
+
+1. **`AnthropicBackend`** in `src/curate/llm.rs` implementing `ChatBackend` exactly per §4.2:
+   `POST {base_url}/v1/messages`; headers `x-api-key`, `anthropic-version: 2023-06-01`,
+   `content-type: application/json`, `anthropic-beta: server-side-fallback-2026-07-01`;
+   body `{model, max_tokens: 16000, system: [{type:"text", text, cache_control:{type:"ephemeral"}}],
+   messages:[{role:"user", content}], output_config:{effort}, fallbacks:"default"}`.
+   **Never** send `temperature`, `top_p`, `top_k`, `thinking`, or an assistant prefill. Ask for JSON
+   in the prompt text and parse tolerantly (reuse `strip_code_fence` + the existing tolerant parsing).
+   Response: concatenate `content[]` blocks with `type == "text"`. Usage: `input_tokens`,
+   `cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`; cost = input×price_input
+   + cache_creation×price_cache_write + cache_read×price_cache_read + output×price_output (per M).
+   `stop_reason == "refusal"` (HTTP 200) surfaces as a distinct `LlmError` variant that triggers the
+   fallback; 429/5xx/network are `Transient` (retried by the existing `RetryPolicy`); 400 is never
+   retried. Request timeout 300 s. API key only from `DAILY_EPUB_ANTHROPIC__API_KEY`.
+   `ChatRequest` gains `effort: Option<String>`; the Anthropic backend ignores `temperature`/`json`
+   and maps `system` to the cached system block. `TokenUsage` needs a cache-write counter (keep the
+   DeepSeek path's cost identical to today). Generalize `UsageMeter` to take a per-provider price
+   table instead of `&DeepseekConfig`.
+2. **`Llms { bulk: Option<LlmClient>, editor: Option<LlmClient> }`** with `editor_or_bulk()` (the
+   editor when configured and its meter is not tripped, else bulk). Both clients share the exact same
+   system prompt string (§8.4). `LlmError::MissingApiKey` and friends must name the provider.
+3. **Config** (§4.2, §19): `[anthropic]` with `enabled`, `base_url`, `model = "claude-opus-5"`,
+   `api_key` (env only), `effort = "high"`, `price_input_per_mtok = 5.0`,
+   `price_cache_write_per_mtok = 6.25`, `price_cache_read_per_mtok = 0.5`, `price_output_per_mtok = 25.0`,
+   `max_daily_usd = 3.0`, `max_concurrent_requests = 4`; `[deepseek].max_concurrent_requests = 4`;
+   `[curation].max_article_count = 28`; `[editorial]` with `summary_model = "editor"` (`editor|bulk`)
+   and `summary_input_tokens = 3000`. Validate `max_article_count >= target_article_count`, effort ∈
+   {low, medium, high, xhigh, max}, batch/concurrency ≥ 1. Startup logs the resolved models and whether
+   each provider is enabled (§19). Update `config.example.toml` and the README (prerequisites, cost
+   line, the note that server-side fallback is enabled, dashboard spend limits as the real backstop).
+4. **Budget and concurrency** (§5): one `UsageMeter` per provider, each with its own `max_daily_usd`.
+   The budget day is the **UTC date of the run's `started_at`**, preloaded by summing
+   `runs.provider_costs_json` for earlier runs that UTC day; this replaces `db::spend_for_date`
+   (remove it). Write `runs.provider_costs_json` (`{"deepseek": {...usage, cost_usd}, "anthropic": {...}}`)
+   and `runs.config_json` (the resolved `[curation]`, `[editorial]`, model names and prompt versions,
+   as JSON) in `finish_run`. Keep `runs.cost_usd` as the total across providers. Existing Stage A
+   scoring batches run through `futures::stream::iter(...).buffer_unordered(max_concurrent_requests)`
+   with the budget check before each request is spawned; a tripped meter skips remaining calls,
+   lets in-flight finish, records the number of unscored candidates in the report, and continues.
+5. **The editor** (§13): `EDITOR_INSTRUCTIONS` replaces `SELECT_INSTRUCTIONS` verbatim from the plan
+   with `{soft_target}`/`{hard_max}` substituted. Runs on `editor_or_bulk()`; on refusal/error fall
+   back to the same prompt on the bulk client; if that fails too, `select_without_llm`. Per-item
+   rendering follows §13 with what exists today (Stage A score/category/rationale instead of
+   quality/fit/facets — those arrive in step 5; render `flags: always-include | excerpt only`;
+   `opening:` first 60 words). Do not put the numeric blend in the prompt. `assemble()` keeps section
+   validation, unique lead, auto-include reinsertion, duplicate-id defence, malformed-response
+   fallback and the `hard_max` trim (by today's ranking key, utility arrives in step 5).
+   **Delete the "too few: top up" branch.** `--max-articles N` is a ceiling:
+   `hard_max = min(config.curation.max_article_count, N)`, `soft_target = min(target_article_count, hard_max)`.
+   `select_without_llm` respects `soft_target` as its size. Each pick's `why` (≤14 words) lands on
+   `Pick.why: Option<String>` and in `issue_articles.why` (`replace_issue_articles`).
+6. **Editorial** (§14): summaries run on the editor client per `editorial.summary_model` with the
+   input budget from config (3,000 tokens), concurrency 4 via `buffer_unordered`, fallback per article:
+   bulk client, then the excerpt. `BRIEF_INSTRUCTIONS` replaces `FRONT_PAGE_INSTRUCTIONS` verbatim
+   from §14.2 (JSON `{"brief": "..."}`); input is the lineup with sections, each pick's title, feed,
+   `why`, summary and the Stage A score. `Editorial.section_intros` is removed (or always empty) and
+   `section.xhtml` renders only the section name; `front_page.xhtml` renders the brief under the
+   masthead; fallback stays `fallback_front_page_html`. The weekly profile rebuild runs on
+   `editor_or_bulk()` (pipeline and the `profile rebuild` CLI).
+7. **Paper** (§15.1): `chapter.xhtml` gets a small italic `Why it's here: <why>` line under the meta
+   line; the In-this-issue page shows the `why` line under each summary. `Colophon` gains
+   `provider_costs` and `models`; the colophon template prints per-provider cost lines and the models
+   (editor and summaries model, bulk model). `StageCounts`/`RunReport` carry per-provider usage;
+   `print_report` in `main.rs` prints per-provider cost.
+8. Remove `Vote`-era leftovers you notice only if they are in your files; do not touch embeddings,
+   signals, telemetry or triage (steps 3–5).
+
+Tests to add/adapt (§20 "Anthropic backend", "Editor", parts of "Pipeline"): request body has the
+system block with `cache_control`, no `temperature`, `output_config.effort`, `fallbacks`, the beta
+header; usage fields parsed into cost with cache read/write prices; `stop_reason: refusal` surfaces as
+the fallback-triggering error; 429 retried, 400 not (use a mock backend or a local `axum`/`tokio`
+listener — never the real network); a nine-pick response is published as nine; `hard_max` trims;
+`--max-articles` is a ceiling; auto-includes reinserted; `why` lines land on picks and in
+`issue_articles.why`; refusal/error on the Anthropic mock falls back to the DeepSeek mock with the
+same prompt; the brief is parsed and rendered, section intros gone; `provider_costs_json` and
+`config_json` are written; budget day preload sums by UTC date.

docs/plans/curation-v2-briefs/step3.md

@@ -0,0 +1,103 @@
+
+---
+
+# YOUR STEP: 3 — Embeddings, signals, telemetry (plan §21 step 3)
+
+You are working in a git **worktree** on branch `curation-v2-step3`, which branches from the step 1
+commit. Another agent is implementing step 2 (the Claude editor) in parallel on the main checkout;
+you will not see its changes and must not need them. Read `git log -3` and the current
+`src/curate/`, `src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs` first.
+Do not run `git checkout`, `git stash`, `git worktree` or `git commit`.
+
+Scope (plan §4.3, §7.1, §7.2, §7.4, §7.5, §9, §12.2 normalization only, §12.4 preliminary blend,
+§15.2 `explain`, §15.4 log line parts, §16, §17 Voyage row, §19 `[voyage]` + `[curation.ranking]`).
+The old heuristic prefilter still gates in this step; signals are computed for every article that
+passes hygiene and persisted, nothing about *selection* changes yet.
+
+1. **`src/curate/embedding.rs`** (§4.3, §7.1, §7.2): `EmbeddingBackend` trait mirroring `ChatBackend`
+   (with a mock for tests), `VoyageBackend` (`POST {base_url}/embeddings`, `Authorization: Bearer`,
+   body `{input, model, input_type: "document"|"query", truncation: true, output_dimension, output_dtype: "float"}`,
+   response mapped by index and length-checked, usage `total_tokens` metered), batching by `batch_size`
+   with `buffer_unordered(max_concurrent_requests)`, retry 429/5xx via `RetryPolicy`, f32
+   little-endian BLOB encode/decode with length and finiteness checks, `dot(a, b)` with a dimension
+   check, and cache orchestration: `article_embeddings` keyed by `(model, dimension, input_hash = sha256
+   of the embedded text)`, overwritten on change. Embedded text = `"Title: {title}\n\n{plain body}"`
+   via `curate::prompt_text`, whitespace collapsed, cut at `max_input_chars` on a char boundary —
+   **no feed name, author or scores**. Interests go to `interest_embeddings` with `input_type = "query"`
+   and the bare interest string. A failed batch leaves those articles without embeddings; never fatal.
+   A Voyage `UsageMeter` with `price_per_mtok = 0.02` and `max_daily_usd` (0.50) as a runaway guard.
+   Key only from `DAILY_EPUB_VOYAGE__API_KEY`. Raw vectors never reach logs or reports.
+2. **Config** (§4.3, §19): `[voyage]` (`enabled`, `base_url = "https://api.voyageai.com/v1"`,
+   `model = "voyage-4-lite"`, `output_dimension = 512`, `batch_size = 32`, `max_concurrent_requests = 4`,
+   `max_input_chars = 60000`, `max_daily_usd = 0.50`) and the complete `[curation.ranking]` block from
+   §19 (`triage_max`, `deep_keep`, `shortlist_keep`, `assessment_reuse_days`, `rating_lookback_days`,
+   `rating_half_life_days`, `neighbour_k`, `negative_coefficient`, `knn_floor/full`, `feed_floor/full`,
+   `semantic_min_words`, `exploration_slots`, `embedding_retention_days`, `telemetry_retention_days`,
+   `[curation.ranking.quotas]`, `[curation.ranking.weights.preliminary]`, `[curation.ranking.weights.utility]`,
+   `[curation.ranking.diversity]`) with the plan's defaults and the validation rules in §19 (weights
+   non-negative; `deep_keep ≥ shortlist_keep ≥ target_article_count`; `*_full > *_floor ≥ 0`;
+   `0 ≤ cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; batch sizes ≥ 1; dimension ∈ {256,512,1024,2048}).
+   Steps 4–5 will *use* the ranking keys; you only add and validate them. Keep `prefilter_keep` for now.
+   Startup logs whether Voyage is enabled. Update `config.example.toml` and the README.
+3. **`src/curate/signals.rs`** (§9): a `Signals` type where every signal is `Option<f64>` (absent ≠ 0):
+   `interest` (§9.1 z-scored interest match, top-three interests recorded; raw top-1 cosine fallback
+   under 30 embedded articles, logged), `knn` (§9.2 preference state from `db::current_ratings(rating_lookback_days)`
+   joined to `article_embeddings`, decayed weights `value × 0.5^(age/half_life)`, top-k positives and
+   negatives, `knn = pos − negative_coefficient × neg`, top-three neighbours recorded, gate ramp
+   `clamp((n − knn_floor)/(knn_full − knn_floor), 0, 1)` multiplying the weight, absent at 0),
+   `feed` (§9.3 Beta-smoothed decayed rate credited to distinct direct `SourceKind::Feed` feeds, mean
+   over the article's rated feeds, gate `feed_floor/feed_full`), `social` (existing composite; absent
+   with no rows), `heuristic` (`longform_points(word_count)` − excerpt-only − roundup penalty, from
+   `prefilter.rs` **without** the social, Scour/HN, multi-source terms; expose those pieces as
+   functions). Log once per run:
+   `preference: N rated articles with embeddings → knn gate X; feed gate Y (n=…)`.
+   Also implement the **mid-rank percentile normalizer** of §12.2 and the **preliminary blend** of
+   §12.4 (present-and-active weights renormalized) so `signals_json.norm`/`weights` can be written now.
+4. **Pipeline**: after social enrichment, for every article that passes hygiene, compute embeddings
+   (cached), signals, and the preliminary blend, then run the old prefilter as today. New stage
+   timings `embed` and `signals`. `--skip-embeddings` uses cached embeddings only (zero Voyage calls);
+   `--skip-llm` unchanged. `StageCounts` gains `eligible`, `embedded`, `rated_with_embeddings`, and
+   the report/`print_report` show them. The paper must still build when Voyage is down or unconfigured
+   (§17): `interest`/`knn` absent, never a penalty.
+5. **`src/curate/telemetry.rs`** (§7.4, §7.5): the `candidate_runs` writer. One row per considered
+   article per run, upserted with `INSERT … ON CONFLICT(run_id, article_id) DO UPDATE` setting every
+   column. Hygiene-excluded articles get thin rows (`stage='excluded'`, `excluded_reason` ∈
+   `blocked | published_before | recently_rejected`, `signals_json='{}'`). In this step map the old
+   pipeline onto the stage vocabulary: passed hygiene → `eligible`; cut by the prefilter →
+   `eligible` + `excluded_reason='not_admitted'`; prefilter survivors → `admitted`; Stage-A scored →
+   `assessed`; sent to selection → `shortlisted`; picked → `selected` (+ `editor_why` if a `why`
+   exists on the pick, else NULL); unpicked shortlist → `excluded_reason='not_selected'`. `signals_json`
+   follows §7.5 exactly (`v: 1`, `raw`, `norm`, `present`, `weights` = effective preliminary weights,
+   `top_interests`, `neighbours`, `exploration: false`, `auto_include`, `notes`). `utility` and
+   `rank_utility` stay NULL until step 5.
+6. **CLI** (§15.2, §16) in `src/main.rs`:
+   `explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]` and
+   `explain --date YYYY-MM-DD --near-misses [N]` printing the persisted `candidate_runs` row for the
+   latest non-dry run of that date (or `--run-id`): stage and reason; every raw/normalized signal with
+   presence and effective weight; top interests with z; nearest rated neighbours; assessments from
+   `article_assessments` when present (empty until step 4); admitted_by; the editor's `why`. `--url`
+   canonicalizes (`dedupe`) and looks the article up; if absent from `articles`, print that it was never
+   ingested. `--near-misses` lists the top N by preliminary blend (utility from step 5 when present)
+   that were not selected. `features backfill [--days 30] [--rated-only] [--all] [--yes]` embeds rated
+   and published articles first, then interests, then other recent articles only under `--all`, prints
+   an estimate and asks for confirmation above 5M tokens unless `--yes`, and is idempotent (warm cache
+   ⇒ zero calls). `features prune` removes `article_embeddings` rows for articles neither rated nor
+   published older than `embedding_retention_days` and `candidate_runs` rows whose run started more
+   than `telemetry_retention_days` ago. `features backfill` takes no lock yet (step 6 adds `lock.rs`).
+7. Types (§18): `Signals`, `Signal` helpers, `RatedArticle` already exists — extend if needed. Do not
+   create `Candidate` yet (step 4/5).
+
+Tests (§20 "Embeddings", "Interest z-scores", "Preference", "Normalization", plus telemetry/explain):
+BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed
+text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the
+others; embedded text contains no feed name or author; a broad interest with uniformly high cosine does
+not dominate while a specific interest with one strong match does; raw fallback under 30 articles; one
+loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high
+(anti-centroid); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate
+0 below `knn_floor`, 1 at `knn_full`, linear between; feed credit sums to 1 across direct feeds; feed
+affinity uses the mean; constant signal → 0.5 for everyone; ties get equal percentiles (400 identical
+zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate
+missing a signal is scored on the rest; `candidate_runs` rows written for every considered article with
+the right stage/reason on a mocked run; `--skip-embeddings` makes zero Voyage calls; a Voyage failure
+still publishes; `explain` renders from persisted rows and reports "never ingested"; `features prune`
+respects rated/published.

docs/plans/curation-v2-briefs/step4.md

@@ -0,0 +1,89 @@
+
+---
+
+# YOUR STEP: 4 — Triage replaces the gate (plan §21 step 4)
+
+Steps 1–3 have landed: three-way votes and `rating_events`; the Claude editor (`Llms { bulk, editor }`,
+`AnthropicBackend`, per-provider `UsageMeter`s, `why` lines, the Brief); Voyage embeddings,
+`signals.rs` (interest/knn/feed/social/heuristic, percentile normalizer, preliminary blend),
+`telemetry.rs` (`candidate_runs` writer), `explain`, `features backfill|prune`, and the full
+`[curation.ranking]` config. Read `git log -6`, then the current `src/curate/` (all files),
+`src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs`, `src/types.rs`.
+
+Scope (plan §7.3 reuse and churn rules, §8.1 hygiene, §10 triage, §11 admission and exploration,
+§12.4 preliminary blend already exists, §15.4 log line parts, §17 DeepSeek row, §19 `deep_keep`):
+
+1. **Migration `migrations/0003_drop_scores.sql`**: `DROP TABLE scores;` (deferred from step 1).
+   Remove `db::upsert_score`, `recently_low_scored_ids` and every `scores` reader. The churn rule
+   now reads `article_assessments`: an article whose latest `triage` `score < recent_rejection_floor`
+   (3.0) or `deep` `score < recent_rejection_floor` within `recent_rejection_days` (7) is
+   `recently_rejected`, unless auto-include (§7.3, §8.1).
+2. **`src/curate/triage.rs`** (§10): `TRIAGE_INSTRUCTIONS` and `TRIAGE_PROMPT_VERSION = 1` verbatim
+   from the plan; per-article block exactly as §10 (id, title, feed (category), author, length ·
+   excerpt only, opening = first 200 words via `prompt_text` + `truncate_words`, `matches interests:`
+   from `top_interests` with z ≥ 1.5 labelled strong (z ≥ 2.5) / weak, `closest rated:` from
+   neighbours with cosine ≥ 0.55; omit lines with nothing). Batches of `deepseek.triage_batch_size`
+   (25) through `buffer_unordered(max_concurrent_requests)` on the **bulk** client, budget check
+   before each batch. Tolerant parsing generalized from `score.rs::parse_score_response`: a
+   malformed item never sinks the batch, ids not in the batch are dropped, unknown `kind` → `other`,
+   `interest` clamped 0–10. Persist every result to `article_assessments (stage='triage', model,
+   prompt_version, profile_version, score, kind, rationale=why, assessed_at)`.
+   **Cache** (§7.3): skip articles with a reusable `triage` or `deep` assessment (same `model` and
+   `prompt_version`, `assessed_at` within `assessment_reuse_days`); `generate --rescore` ignores the
+   cache. A failed batch leaves `triage` absent for its articles.
+   **Pool cap** (§10): if eligible > `triage_max` (800), triage the union of top `0.7 × triage_max`
+   by preliminary blend, top 100 by `interest`, top 100 by `knn` (if active), all auto-includes,
+   filled to `triage_max` by blend; the rest get `stage='eligible'`, `excluded_reason='not_admitted'`.
+3. **`src/curate/admit.rs`** (§8.1, §11): hygiene moves here (`blocked`, `published_before` = in
+   `issue_articles` for any issue date before this run's date, `recently_rejected` per item 1;
+   auto-includes never excluded) writing the thin `candidate_runs` rows, and the union admission
+   into `deep_keep` (120) slots in this order, each retriever taking its top-N by its own signal among
+   not-yet-admitted, not-excluded articles, recording every retriever that would have taken an
+   article in `admitted_by` (first = the one that admitted it):
+   `auto_include` (uncapped) → `triage` (quota 60, floor `interest ≥ 5`) → `interest` (20; floors
+   `word_count ≥ semantic_min_words`, not `looks_like_roundup`, triage `interest ≥ 3` if triaged)
+   → `knn` (20, gate > 0, same floors) → `exploration` (5, §11.1) → `blend` (remaining).
+   Inactive retrievers release their quota to `blend`. Not admitted → `stage='triaged'` (or
+   `'eligible'` if never triaged) + `excluded_reason='not_admitted'`.
+   **Exploration** (§11.1): five slots for articles ranked between `deep_keep` and `deep_keep × 2.5`
+   by the preliminary blend with `word_count ≥ 300`, not roundups, triage `interest ≥ 4`; ordered by
+   `sha256(run_date || article_id)`; flagged `exploration = true` through to the editor prompt.
+4. **Prefilter reduced** (§18): `prefilter.rs` keeps only hygiene helpers (`is_blocked`,
+   `is_auto_include`, `looks_like_roundup`) and the text heuristic pieces `signals.rs` uses. Delete
+   `prefilter::run`, `score_article`'s social/Scour/HN/multi-source terms, `PrefilterContext`, the
+   `prefilter_keep` config key (replace uses with `curation.ranking.deep_keep`; the config validation
+   `deep_keep ≥ shortlist_keep ≥ target_article_count` already exists), and `Curator::prefilter`.
+   A stale `prefilter_keep` key in a config file must fail loudly with a message naming
+   `curation.ranking.deep_keep` (follow the `bookorbit_dir` precedent in `config.rs`).
+5. **Pipeline**: hygiene → embeddings → signals → preliminary blend → triage → admission → the
+   *existing* Stage A scoring (`score.rs`, unchanged this step) over the admitted deep set → the
+   editor → editorial, as today. The deep set (`admitted`) is what Stage A scores and what the editor
+   sees; step 5 replaces Stage A with deep assessment and adds the shortlist. Telemetry stages now:
+   `excluded | eligible | triaged | admitted | assessed | shortlisted | selected` with reasons per §7.4;
+   `admitted_by` and `exploration` written; `signals_json.raw/norm/present` gain `triage` (÷10 for norm).
+   Stage timings `triage` and `admit`; `StageCounts` gains `triaged`, `admitted`, `admitted_by` (map),
+   `exploration_admitted`, `exploration_selected`. Log the admission line from §15.4:
+   `admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0`.
+   `--skip-llm` skips triage; DeepSeek down or tripped ⇒ no triage, admission by
+   `interest`/`knn`/`blend`, editor still runs (§17). The editor prompt (§13 rendering) shows the
+   triage score and `flags: exploration` where present. `explain` prints the triage assessment
+   (score, kind, why) and `admitted_by`.
+6. `types.rs` (§18): introduce `Assessment { triage: Option<Triage>, deep: Option<Deep> }` and
+   `Triage { interest, kind, why, model, prompt_version, assessed_at }` (keep `Deep`/`LlmScore`
+   compatible with the existing Stage A output for now), and `Candidate { article, auto_include,
+   exploration, signals, assessment, utility: Option<f64>, cluster: Option<...>, admitted_by: Vec<String>,
+   stage, excluded_reason }` replacing `ScoredArticle` where the pipeline flows through admission.
+   Keep the change mechanical where `score.rs`/`select.rs` still consume the old shape (step 5
+   retires them); an adapter is acceptable.
+7. `runs.config_json` gains `TRIAGE_PROMPT_VERSION`; `config.example.toml` and README updated
+   (`prefilter_keep` removed, `[deepseek].triage_batch_size`, `--rescore`).
+
+Tests (§20 "Triage and deep parsing" triage half, "Admission", churn/cache): realistic triage
+fixture parsed; malformed items do not sink a batch; unknown kind → other; every `kind` token in the
+prompt round-trips; cached triage reused within `assessment_reuse_days` and ignored with `--rescore`;
+churn rule excludes a recent low triage score and spares auto-includes; a strong-interest,
+weak-heuristic, no-social article reaches the deep set; a 60-word stub with high interest similarity
+is not admitted by `interest`/`knn`; quotas honoured; inactive retrievers release quota; exploration
+deterministic per date and rotating across dates; auto-includes always admitted; excluded articles get
+thin rows with the right reason; pool cap path yields `not_admitted` rows; the mocked full pipeline
+still publishes with DeepSeek failing; migration 0003 drops `scores`.

docs/plans/curation-v2-briefs/step5.md

@@ -0,0 +1,69 @@
+
+---
+
+# YOUR STEP: 5 — Deep assessment, utility, diversity (plan §21 step 5)
+
+Steps 1–4 have landed; triage and union admission now gate the deep set and the old prefilter is
+reduced to hygiene helpers. Read `git log -8`, then all of `src/curate/`, `src/pipeline.rs`,
+`src/types.rs`, `src/config.rs`, `src/report.rs`, `src/main.rs`, `src/db.rs`.
+
+Scope (plan §12 in full, §13 rendering with facets/neighbours, §7.3 deep rows, §18 module retirements):
+
+1. **`src/curate/assess.rs`** replaces `score.rs` (§12.1): `DEEP_INSTRUCTIONS` and
+   `DEEP_PROMPT_VERSION = 1` verbatim from the plan (the section palette is substituted from
+   `curation.sections`). Per article: title, feed, author, length, excerpt-only flag, the triage
+   `why`, the interest and neighbour hint lines from §10, and a **representative sample** of
+   ~2,000 tokens: body under ~1,500 words sent whole; otherwise first 600 words, 500 around the
+   midpoint, last 400, with visible `[BEGINNING]`/`[MIDDLE]`/`[END]` markers, split on word
+   boundaries. Batches of `deepseek.deep_batch_size` (8, replaces `score_batch_size`) through
+   `buffer_unordered`, bulk client, budget check per batch. Auto-includes are assessed too.
+   Tolerant parsing: `quality`/`fit` clamped 0–10, `category` validated against the palette (invalid →
+   `None`, the editor decides), `paywalled_guess`, and `facets` where every unknown enum token degrades
+   to `None` and `specific_topics` is capped at 3. **Every enum token in the prompt must round-trip
+   through the parser (test).** Persist to `article_assessments (stage='deep', score=quality, fit,
+   kind=facets.format, facets_json, rationale, category, paywalled_guess, ...)`; reuse per §7.3 with
+   `--rescore` bypass. Facets are shown to the editor, the profile rebuild and `explain`; **not** a
+   numeric signal.
+2. **`src/curate/rank.rs`** (§12.2–§12.5): normalization (LLM scores ÷ 10; everything else mid-rank
+   percentile over the deep set — reuse/move the step-3 normalizer), the **utility** blend over
+   present signals with `[curation.ranking.weights.utility]` renormalized and learned signals
+   multiplied by their gate ramp first, stored 0–100; the preliminary blend stays for the eligible set.
+   **Diversified shortlist** (§12.5): leader clustering by embedding cosine with `cluster_threshold`
+   (0.85) and `per_cluster_cap` (2): sort by utility desc then article id asc; assign each article to
+   the first existing cluster whose *leader* has cosine ≥ threshold, else make it a new leader;
+   articles without embeddings are singletons; admit in order while the cluster's admitted count is
+   below the cap until `shortlist_keep` (60); the top `utility_protected` (10) by utility and all
+   auto-includes are admitted regardless and still count toward their cluster; exploration picks that
+   reached the deep set get up to 3 reserved slots; if short, relax to cap 3, then uncapped. Persist
+   `cluster_id`, `cluster_rank`, `rank_utility`, `utility`, `excluded_reason ∈ cluster_suppressed |
+   shortlist_cap`, stage `shortlisted`.
+3. **`src/curate/editor.rs`** replaces `select.rs`: the §13 rendering in full —
+   `quality 8.5 · fit 7.0 · triage 8.0 — <deep rationale>`, `facets: format · depth · evidence ·
+   technicality · topic_group`, `matches:`, `closest rated:`, `flags: exploration | always-include |
+   excerpt only`, `opening:` first 60 words. The editor sees the 60-item shortlist. `assemble()`'s
+   `hard_max` trim and `select_without_llm` now order by **utility**, falling back to the preliminary
+   blend. Delete `ScoredArticle::combined_score()` and `ScoredArticle` itself; `Candidate` is the only
+   flow type. Move the tests from `score.rs`/`select.rs` into `assess.rs`/`editor.rs` and delete the
+   old files (update `curate/mod.rs`, `Curator`).
+4. **Editorial inputs**: the Brief and the summaries use quality/fit where the plan says so (§14.2
+   input lists quality/fit). The weekly profile rebuild's rated lines now get real facets.
+5. **Pipeline/report**: stage timings `assess` and `rank`; `StageCounts` gains `assessed`,
+   `shortlisted`, `clusters`; the §15.4 curation line
+   `curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected`.
+   `explain` prints the deep assessment (quality, fit, category, rationale, facets), utility and rank,
+   cluster id and what suppressed it; `--near-misses` orders by utility. `runs.config_json` gains
+   `DEEP_PROMPT_VERSION`. `--skip-llm`/DeepSeek down ⇒ no deep assessment, utility over present
+   signals (triage/interest dominate, §12.3), editor still runs on what it has (§17).
+6. `config.example.toml`/README: `deep_batch_size` replaces `score_batch_size`; a stale
+   `score_batch_size` key fails loudly naming `deep_batch_size`.
+
+Tests (§20 "Triage and deep parsing" deep half, "Normalization", "Clustering", "Editor" utility
+parts): realistic deep fixture; malformed items do not sink a batch; unknown facet tokens → `None`;
+every enum token round-trips; representative sample has the three markers and respects word
+boundaries, short bodies are sent whole; cached deep rows reused / bypassed with `--rescore`; constant
+signal → 0.5; ties equal; absent values do not shift others; effective weights sum to 1; a candidate
+missing a signal is scored on the rest; near-duplicates share a cluster and the third is suppressed;
+protected top-N survive and count; the bridge case (A~C, B~C, A≁B, utility A>B>C) yields two clusters;
+articles without embeddings are never suppressed; `hard_max` trims by utility; `select_without_llm`
+orders by utility; the mocked full pipeline publishes with DeepSeek down and writes `shortlisted`
+rows with cluster ids.

docs/plans/curation-v2-briefs/step6.md

@@ -0,0 +1,62 @@
+
+---
+
+# YOUR STEP: 6 — Paper telemetry, stats, lock (plan §21 step 6)
+
+Steps 1–5 have landed: the full v2 ranking pipeline (hygiene → embeddings → signals → triage →
+admission → deep assessment → utility → cluster-capped shortlist → Claude editor → editorial),
+`candidate_runs` telemetry, `explain`, `features`, `ratings`. Read `git log -10`, then
+`src/pipeline.rs`, `src/report.rs`, `src/main.rs`, `src/curate/telemetry.rs`, `src/epub/chapters.rs`
+and `src/epub/templates/`, `src/types.rs` (`Colophon`), `src/db.rs`, README, `config.example.toml`.
+
+Scope (plan §5 lock, §15.1 Behind the paper + In-this-issue, §15.3 `stats`, §15.4 report/log block,
+§16 lock coverage, §19 startup logging, README/config docs):
+
+1. **Behind the paper** (§15.1): a new short chapter after the World Briefing and before the colophon
+   (`src/epub/templates/behind.xhtml`, `render_behind_the_paper`) with exactly the content shape in
+   the plan: the considered/eligible/triaged/read-closely/shortlisted/selected line; the
+   `Admitted via:` line; the `Learned signals:` line (rated-with-embeddings count and knn percentage,
+   feed affinity state); `Near misses` — the 10 highest-utility non-selected articles as
+   `<title> — <feed> · quality X · fit Y · <stage/reason>`; the `Models:` line with triage/assessment,
+   editor/summaries and embeddings models; cost and generation time. Both editions (X4 gets the same
+   text, no links). The data comes from the run's `RunReport`/`StageCounts` and the run's
+   `candidate_runs` rows (add a `BehindThePaper` struct to `types.rs` filled by the pipeline; keep
+   templates pure). Chapter ids stable (`behind`).
+2. **In this issue** page: verify each entry shows the `why` line under the summary (step 2 was asked
+   to do this; complete it if missing). The colophon keeps its fields and shows per-provider cost
+   lines and models (verify; complete if missing).
+3. **`stats`** (§15.3): `daily-epub stats [--days 14]` printing issues, articles published, explicit
+   ratings by label, ratings per issue, up/down ratio per admitting retriever (`admitted_by[0]` of
+   rated picks, from `candidate_runs` joined to `rating_events` via the latest explicit event),
+   exploration yield (exploration picks selected / admitted, and how many were rated positively),
+   mean issue size, cost per day per provider (from `runs.provider_costs_json`), mean generation time.
+   Plain text, one fact per line, no tables wider than 80 columns.
+4. **Run report and log block** (§15.4): make sure `StageCounts` has every field listed —
+   `eligible`, `embedded`, `triaged`, `admitted`, `admitted_by` (map), `assessed`, `shortlisted`,
+   `clusters`, `exploration_admitted`, `exploration_selected`, `verdicts_in_prompt`,
+   `rated_with_embeddings`, per-provider usage — and that stage timings include `embed`, `signals`,
+   `triage`, `admit`, `assess`, `rank`, `editor`, `summaries`, `brief`. Emit the four-line info block
+   from §15.4 once per run (`curation:`, `admission:`, `preference:`, `providers:`). `print_report`
+   in `main.rs` prints the same lines.
+5. **`src/lock.rs`** (§5): `flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`, taken in `main`
+   for `generate`, `profile rebuild`, `features backfill`, and `backfill-social`; a second invocation
+   exits with `generate is already running` (name the command that holds it if cheap, else the
+   generic message). `serve`, `explain`, `stats`, `ratings`, `db migrate`, `features prune` do not
+   take it. Use `libc`/`rustix` only if already in the dependency tree (check `Cargo.lock`); otherwise
+   `std::fs::File` + the `fs2`-free approach via `libc::flock` is acceptable to add as a tiny dep —
+   say which in the report. Twenty lines, no table, no TTL.
+6. **Docs**: README rewritten where it describes the pipeline, feedback links, costs, prerequisites
+   (Anthropic and Voyage keys, dashboard spend limits as the real backstop, the server-side fallback
+   note), the CLI (`ratings`, `explain`, `stats`, `features`, `--rescore`, `--skip-embeddings`,
+   `--max-articles` as ceiling), and `data/profile.md`. `config.example.toml` carries every key from
+   plan §19 with comments. Startup logs resolved models and per-provider enabled state (verify).
+7. `systemd/daily-epub-generate.service`: no change needed unless an env var name changed; note the
+   two new env vars in the README's systemd section.
+
+Tests (§20 "Lock", "Pipeline" behind-the-paper, stats): two processes/one wins (spawn the binary or
+use two `File` handles from separate threads with `try_lock` semantics — no sleeping longer than a
+second), a killed holder frees the lock, `serve` does not take it (parse-level or by checking no lock
+file is created); the behind-the-paper chapter renders with the counts and near misses on the mocked
+pipeline run and is parseable XHTML in both editions (extend `tests/m4_epub.rs`); `stats` runs on a
+temp DB with a couple of runs, issues and rating events and prints every line; the info block appears
+in the log (capture with `tracing` test subscriber or assert on the formatted strings).

docs/plans/curation-v2-briefs/step7.md

@@ -0,0 +1,29 @@
+
+---
+
+# YOUR STEP: 7 — Cleanup (plan §21 step 7)
+
+Steps 1–6 have landed. This step removes what the plan retired and updates the implementation
+notes. Read `git log -12` and skim every file under `src/` for leftovers.
+
+1. **Dead code**: anything only the old prefilter/scores/select/feed-prior paths used — unused
+   functions, types (`LlmScore` if nothing reads it, `ScoredArticle`, `FeedPrior`, `Rating`),
+   `kv` keys no longer read, config aliases and `#[serde(alias)]`s introduced as transition aids,
+   `// filled in by step N` comments, stale doc comments that still describe 👍/👎, "From the Editor",
+   section intros, prefilter gating, or `async-openai`. `cargo clippy --all-targets -- -W dead_code`
+   must be clean; do not add `#[allow(dead_code)]`.
+2. **Prune paths** (§7.1, §7.4): confirm `features prune` covers `article_embeddings`
+   (`embedding_retention_days`) and `candidate_runs` (`telemetry_retention_days`); add
+   `article_assessments` older than `telemetry_retention_days` to it. Make `generate` call the prune
+   once per run after publishing (best effort, logged).
+3. **Docs**: update `docs/plans/2026-08-15-implementation-notes.md`: fix item 5 (hand-rolled
+   `reqwest` client, not `async-openai`), add the new providers (Anthropic Messages API facts from
+   plan §4.2, Voyage facts from §4.3, each with "verified 2026-09-02"), the new tables, the lock, the
+   budget-day rule, and a short "curation v2" section pointing at
+   `docs/plans/2026-09-02-personalized-curation-v2.md`. This is the one step allowed to edit `docs/`.
+   Do not edit the plan itself.
+4. **Consistency pass**: `config.example.toml` matches `Config::default()` key for key (write a test
+   that loads the example and compares every `[curation.*]`, `[anthropic]`, `[voyage]`, `[editorial]`
+   value to the defaults, since the defaults are the plan's numbers); README CLI section matches
+   `daily-epub --help` output for every subcommand (spot check by running the binary).
+5. Run `cargo fmt`, `cargo clippy --all-targets`, `cargo test`; report the summary lines.

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-18-2.md

@@ -0,0 +1,273 @@
+# Review — Personalized Ranking, Embeddings, Facets, and Feedback
+
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md`
+**Reviewer:** independent blind review (no other review consulted)
+**Date:** 2026-08-18
+**Method:** plan read end to end, then checked against `src/pipeline.rs`, `src/curate/{mod,prefilter,score,select,llm}.rs`, `src/curate/profile/mod.rs`, `src/types.rs`, `src/db.rs`, `src/config.rs`, `src/main.rs`, `migrations/0001_init.sql`, `data/scour-interests.opml`. Voyage AI claims in §3 were independently verified against the live docs.
+
+---
+
+## Verdict
+
+**Not ready to execute as written — but close, and the architecture is right.** The diagnosis in §1 is accurate and well evidenced against the code: `prefilter.rs` really does make the first irreversible cut on word count, social proof, and a max-over-feeds prior (`PrefilterContext::prior_for`, `prefilter.rs:118-129`), and `select.rs:556-579` really does pad the lineup back up to `target - 5` in direct contradiction of the profile's stated editorial philosophy. The union-of-retrievers direction, the quality/reader-fit split, `candidate_rankings` as a durable feature snapshot, and "no forced filler" are all the correct calls, and I would not relitigate them. The Voyage facts in §3 are accurate (I verified `voyage-4-lite` exists at 32K context, dims 256/512/1024/2048, 1,000 inputs and 1M tokens per request, `$0.02`/Mtok with 200M free) — a genuinely unusual level of rigor for a plan.
+
+What blocks execution is a small number of concrete numerical-design defects, not the architecture. Three of them cause the new system to *look* like it is working while producing garbage: percentile normalization with distinct tie-ranks injects article-ID order as ranking signal whenever a signal is sparse or absent (which is the day-one state); the sparse-evidence confidence damping in §11.2 is mathematically cancelled by the very normalization step that consumes it; and the facet vocabulary is roughly 84 values estimated from roughly 63 ratings, so `facet_preference` will be a near-constant for months while holding 22% of the pre-Stage-A blend. Separately, §16's split of `LlmScore.score` silently kills the churn-suppression rule, and the plan doubles the number of sequential DeepSeek round-trips in a job that has a publish deadline without ever mentioning concurrency. Fix Critical and High below — most are a paragraph of spec each — and this is ready.
+
+---
+
+## Critical
+
+### C1. Percentile normalization with ID tie-breaking turns sparse signals into article-ID bias
+
+> §14.1: "convert the daily candidate values to deterministic percentile ranks in `[0,1]` … Tie breaking must be stable by article ID."
+
+Stable-by-ID tie breaking assigns **distinct** percentile ranks to **equal** raw values. That is correct for determinism of output order and wrong for normalization. Consider the two signals most likely to be degenerate:
+
+- **`embedding_preference` on any day with no ratings in the lookback window** (i.e. day one of rollout, and any 90-day gap): §11.2 gives `positive_similarity_or_0 - 0.75 * negative_similarity_or_0` = `0.0` for every candidate. Percentile-ranking 400 tied zeros with an ID tiebreak produces a perfect ascending-article-ID ramp from 0.0 to 1.0. §14.2 then weights that ramp at **0.28** of the pre-Stage-A score and §17 at **0.15** of utility.
+- **`social_score`**: `composite_social_score` returns exactly `0.0` for every article with no `social` rows — the majority on any given day. Same ramp, 0.05 / 0.04 weight.
+
+Article IDs are `AUTOINCREMENT` (`migrations/0001_init.sql:26`) and assigned in `persist_articles` iteration order, so low IDs are systematically older articles and articles from feeds that happened to sort earlier. The failure is silent: rankings look plausible, `candidate_rankings` looks populated, and the reader sees a subtly wrong paper.
+
+**Fix (specify explicitly in §14.1):** equal raw values must receive **equal** normalized values — use mid-rank / average-rank percentiles (`(#below + (#equal + 1)/2) / n`). Article ID may break ties only in the *final output ordering*, never inside the normalizer. Add a unit test: "a signal that is constant across all candidates normalizes to 0.5 for every candidate," alongside the existing §27.6 "percentile normalization stable with ties" case, which as worded would pass with the buggy behavior.
+
+### C2. Splitting `LlmScore.score` silently disables the churn-suppression rule
+
+§16.1 replaces `score` with `quality_score` + `reader_fit_score`. §13.1 says the recently-rejected rule is preserved:
+
+> §13.1: "recently rejected churn rule (LLM < 3 within the configured lookback), except always-includes."
+
+That rule is implemented as `db.recently_low_scored_ids(STALE_LOW_SCORE, since)` (`prefilter.rs:104`), which reads `scores.llm_score` (`db.rs:325-339`), which is written only by `db.upsert_score` from `llm.score` (`db.rs:402`). If Stage A stops producing a field named `score`, nothing writes `scores.llm_score`, `recently_low_scored_ids` returns empty forever, and the rule dies with no error and no test failure. The consequence is not cosmetic: yesterday's rejects re-enter the recall pool every day, they consume facet-extraction and Stage A tokens, and `PENALTY_TITLE_PATTERNS`-class churn recirculates indefinitely.
+
+The plan gestures at this — §16.1 says "Prefer a new type if changing `LlmScore` would make existing persisted `scores.llm_score` ambiguous" — but never states what writes the column going forward. Worth noting that `LlmScore` is **not** persisted as JSON anywhere (I checked `report.rs`, `server.rs`, `publish.rs`; `issue_articles` stores only section/position/summary), so the only real compatibility surface is this one column.
+
+**Fix:** state in §16.1 and §26 that `scores.llm_score` continues to be written with `quality_score`, and add `llm_reader_fit_score` as a new column in `0002`. Add the regression test: "an article with `quality_score < 3` yesterday does not appear in today's recall pool."
+
+### C3. The sparse-evidence confidence damping in §11.2 is a no-op
+
+> §11.2: `confidence = total_decayed_rating_weight / (total_decayed_rating_weight + 6.0)`; `embedding_preference = embedding_preference_raw * confidence`
+
+`confidence` is a **single scalar for the whole run** — it depends only on the rating history, not on the candidate. Multiplying every candidate's raw score by the same positive constant is a strictly monotone transform. It is therefore erased by:
+
+- §14.1's percentile normalization (rank-invariant), which is what consumes `embedding_preference` in the §14.2 and §17 blends; and
+- §13.3's "top `recall_embedding_preference_top` by learned embedding preference" (a pure top-K, also rank-invariant).
+
+So the damping has no effect anywhere it is used. With three upvotes total, the embedding preference signal still contributes its full 0.28 / 0.15 weight, ranking on noise. This is the exact failure the paragraph was written to prevent.
+
+**Fix:** damping must act on the **blend weight**, not the score. Replace with: compute `w_embed_effective = w_embed * confidence`, redistribute the freed weight proportionally across the remaining present signals, and record both in `explanation_json`. Alternatively blend the normalized signal toward the pool mean: `norm' = 0.5 + confidence * (norm - 0.5)`. Either works; the current formulation cannot. Same audit is needed for any other place the plan multiplies a whole-run constant into a per-candidate score.
+
+---
+
+## High
+
+### H1. Signals with no evidence must be dropped from the blend and the weights renormalized
+
+The plan does exactly the right thing *inside* facet preference — §11.3: "Normalize by the sum of weights actually present" — and then does not do it for the **outer** blends in §14.2 and §17. On a cold-start day:
+
+- `embedding_preference` has no evidence (C3) → 0.28
+- `facet_preference` has no evidence (H2) → 0.22, mapped through `(x+1)/2` to a constant 0.5
+
+That is **50% of the pre-Stage-A score** that is either constant or ID-noise, silently compressing the dynamic range of the heuristic, interest, and social signals to half their intended influence. §17's utility has the same problem at 25%.
+
+**Fix:** make the blend a weighted mean over *present* signals with per-signal presence tests (`has_positive_centroid`, `facet_dimensions_with_support > 0`, `social_rows > 0`), renormalizing to the weights actually present, and persist the effective weight vector into `explanation_json` so `explain` and `evaluate` can see it. This is ~15 lines and it is the difference between the system degrading gracefully and degrading invisibly.
+
+### H2. The facet vocabulary is far too large to estimate from the available ratings
+
+§10.1 defines 11 dimensions over roughly **84 controlled values** (14 topic groups + 18 formats + 4 depths + 5 technicality + 4 audience + 10 tones + 8 stances + 9 evidence modes + 4 temporal + 4 locality + 4 commerciality). §11.3 estimates a Beta rate per value from decayed ratings. The plan's own example telemetry in §24 says:
+
+> `personalization: 63 recent ratings, 55 embedding-backed, 51 facet-backed`
+
+63 ratings across 84 values, further split by up/down and diluted by multi-value averaging, means the median facet value will have **zero or one** observation. With `support = (u+d)/(u+d+4)`, a single observation gives `support = 0.2` and `effect = (0.67-0.5)*2*0.2 ≈ 0.067` — indistinguishable from noise. `facet_preference` will hover near zero for months while consuming 0.22 of the pre-Stage-A blend and one DeepSeek call per 12 recall-pool articles.
+
+The plan is aware of the cardinality/evidence tradeoff — §10.1: "Keep controlled enums small enough that ratings accumulate statistical support" — and then does not follow its own rule.
+
+**Fix, pick one:**
+- **(Recommended)** Ship V1 with **four** dimensions and ~5 values each: `format` (collapse 18 → `reported | analysis_essay | tutorial_technical | postmortem_case_study | announcement_roundup`), `depth`, `evidence_mode` (collapse 9 → `first_hand | original_reporting | data_or_experiment | synthesis | speculative`), `commerciality`. That is ~20 parameters against 63 ratings — estimable. Keep the full vocabulary as `schema_version = 2` once there are ~300 ratings. The full enums are still worth extracting into `facets_json` for explanation and for the §12 profile prompt; just don't *score* on the sparse ones.
+- Or gate the whole facet-preference contribution behind a minimum-evidence threshold and let H1's renormalization carry the weight elsewhere until then.
+
+Note this also affects §12: "Enrich the rebuild prompt with saved facet data" is valuable at *any* cardinality, because the LLM is doing pattern recognition, not parameter estimation. That part should ship regardless.
+
+### H3. The high-recall union has no quality floor on three of its four retrievers, and dense retrieval has a strong short-document bias
+
+§13.3 admits candidates by "top 80 by semantic-interest score", "top 80 by learned embedding preference", "top 30 by feed affinity". Only exploration gets a floor (§15: "still above a minimal heuristic-quality floor so the system does not explore obvious junk").
+
+Cosine similarity against a short query concentrates on short documents. A 60-word "Rust 1.94.0 released" changelog stub whose `embedding_document` (§8.1) is `Title: … / Source: … / <60 words>` will score *higher* against the interest query "Rust" than a 3,000-word essay that discusses Rust among other things — because the essay's vector is diluted across many topics. The same holds for the positive centroid. So the two semantic retrievers will systematically over-admit exactly the class §16.2 tells Stage A to punish ("announcements/roundups/vendor marketing are generally low quality"), and which `PENALTY_TITLE_PATTERNS` already exists to catch.
+
+§13.4's "top 20 from each major retriever" protection bounds the damage to ~40 pool slots, but those slots cost facet-extraction tokens and displace real candidates.
+
+**Fix:** apply the cheap existing hygiene to the semantic paths — require `word_count >= ~250` and `!looks_like_roundup(title)` for admission *via the semantic-interest or embedding-preference retrievers only* (an article can still enter via heuristic or auto-include). None of the plan's own motivating examples are affected: §32's "quiet 900-word post" clears 250 comfortably. Add the inverse regression test alongside §27.5's: "a 60-word release-note stub with very high interest similarity does **not** enter the recall pool."
+
+### H4. The plan roughly doubles sequential LLM round-trips and never mentions concurrency
+
+`score_all` batches serially (`score.rs:344`, a plain `for` over `chunks`). The new pipeline adds facet extraction over the ~240-article recall pool at `facet_batch_size` 12–16 — **15 to 20 additional sequential DeepSeek calls** — and simultaneously grows every Stage A prompt from a 200-word excerpt (`score.rs:21`) to a ~450-word beginning/middle/end sample (§10.2), which lengthens each call. Voyage adds ~13 more sequential calls at `batch_size = 32` over ~400 articles.
+
+Token *cost* is not the issue (~240 × 600 tokens ≈ 145K input for facets, well inside `max_daily_usd = 2.0`). **Wall clock is.** This job runs on a 05:30 America/New_York timer and has to produce an EPUB before breakfast; adding 30+ sequential API round-trips to a stage that is already the slowest is a real delivery risk, and the plan's §24 stage-timing list implicitly acknowledges the concern without addressing it.
+
+**Fix:** specify bounded concurrency (`futures::stream::iter(batches).buffer_unordered(4)`; `futures` is already a dependency) for facet extraction, Stage A, and Voyage batching, with the `UsageMeter::check_budget` gate evaluated before each spawn rather than between batches. Add the budget-trip semantics under concurrency to §23 — currently "Once tripped, remaining calls for that provider are skipped for the run" is written assuming a serial loop.
+
+### H5. `candidate_rankings` reruns will interleave two runs' state unless the write is a full replace
+
+> §5.4: "Persist rows incrementally as stages complete. A rerun for the same date should replace/update the day's rows deterministically."
+
+"Replace/update" is ambiguous, and the existing house idiom is the opposite of what is needed here: `db.upsert_score` uses `COALESCE(excluded.x, scores.x)` (`db.rs:394-397`), which deliberately *preserves* prior values. If `candidate_rankings` copies it, a rerun that trips the budget at the recall stage will leave yesterday's `llm_quality_score`, `stage_a_candidate = 1`, and `selected = 1` attached to rows the current run never scored. This table is the ground truth for §21's entire evaluation program and for acceptance criterion 12; silently corrupt evaluation data is worse than no evaluation data.
+
+**Fix:** mandate `DELETE FROM candidate_rankings WHERE run_date = ?` at the start of the recall stage, inside the same transaction as the first batch of inserts — matching `replace_issue_articles` (`db.rs:475-497`), which is the correct existing precedent. Add to §27.8: "rerunning a date that previously reached Stage B, but which now trips the budget at recall, leaves no stale Stage A/B flags."
+
+### H6. `explain` and acceptance criterion 10 cannot be satisfied by the proposed schema
+
+> §30, criterion 10: "`candidate_rankings` records why every eligible daily article did or did not survive each funnel stage."
+
+But §5.4 persists "every **post-hygiene** candidate," and the most common exclusions happen *before* that: already-published, blocked domain, recently-rejected churn (`prefilter.rs:271-287`). Those articles get no row at all, so the single most frequent answer to "why did this article not show up?" is unanswerable from the table. The boolean flag set also cannot distinguish "did not make the recall union" from "made the union but was cut by the cap in §13.4."
+
+**Fix:** add `excluded_reason TEXT` (nullable; `published | blocked | churn | not_recalled | recall_cap | stage_a_cut | mmr_cut | not_selected`) and write a row for every article the run considered, hygiene-excluded ones included, with only that column and the identifying keys populated. Cost is ~400 thin rows/day. Then criterion 10 is actually testable.
+
+---
+
+## Medium
+
+### M1. Backfill has no cost estimate or guard, and embedding storage grows unbounded
+
+§20.1 step 3: "Embed recent articles (e.g. last 90 days) for historical replay/exploration if desired." Nothing prunes `articles` — `publish::prune` only removes EPUB/XTC *files* (`publish.rs:491-547`), and `retention_days = 21` is a file policy. At ~400 articles/day, a system that has been running 90 days holds ~36,000 article rows; embedding all of them is ~47M tokens — a quarter of the lifetime 200M free allocation spent by one command with no confirmation, across ~1,125 requests.
+
+Storage compounds: 1024 × f32 = 4,096 bytes plus row overhead, ~1.7 MB/day, **~600 MB/year** of SQLite BLOB on a VPS, with no retention policy anywhere in the plan.
+
+**Fix:** (a) require `features backfill` to print an estimated token count and USD cost and require `--yes` above a threshold; default `--days` to 30 and default to `--rated-only`. (b) Add a retention rule to §5.1: drop embeddings for articles that are neither rated nor published and are older than N days. (c) Reconsider `output_dimension = 512` as the **default** rather than 1024 — Voyage's Matryoshka training makes 512 near-lossless for retrieval, it halves storage and dot-product cost, and the plan already requires the dimension to be configurable (§3). For a single-reader system on a small host, 512 is the better default and 1024 is the thing you evaluate into.
+
+### M2. `Source: <feed title>` in the embedding document contradicts §8.1's own rule and degrades MMR
+
+§8.1 is emphatic — "Do not include social score, ratings, feed prior, LLM rationale, or other ranking metadata … The vector should represent the article itself" — and then includes `Source: <feed title>` in the document format. Feed title *is* provenance metadata. Two consequences:
+
+1. The positive centroid partly encodes "feeds the reader upvotes," double-counting with the separate `feed_affinity` signal (0.08 pre-Stage-A, 0.05 utility) that the plan went to some trouble to de-bias in §5.5.
+2. **MMR degrades**: two unrelated posts from the same blog become artificially similar, so §18's diversification will suppress the second post from a favored feed as "redundant" when it is not. This is the one calculation where topical purity actually matters.
+
+The effect is small for a 2,000-word article and material for a 200-word one — compounding with H3.
+
+**Fix:** drop `Source:` (and probably `Author:`) from `embedding_document` v1. Keep the `EMBEDDING_DOCUMENT_VERSION` constant so this is an easy A/B later.
+
+### M3. Max-similarity over 230 standing interests will not discriminate, because most interests are broad single words
+
+I parsed `data/scour-interests.opml`: 230 interests, dominated by short generic terms — `Nature`, `History`, `Space`, `Engineering`, `Science`, alongside specific ones like `Gaussian Splatting`, `Writerdeck`, `tmux`. §9.2 takes `0.70 * top1 + 0.30 * mean(top3)` of raw cosine similarity across all of them.
+
+Broad terms have high *average* similarity to everything. So `top1_similarity` will almost always be one of the generic interests, at a value that varies little between articles, and the score mostly measures "how generic is this article" rather than "does this match a stated interest." The genuinely valuable signal — "this article is *unusually* close to Gaussian Splatting" — is exactly what max-of-raw-cosine destroys.
+
+**Fix:** z-score each interest's similarity **across the day's candidate pool** before taking top-1/top-3: `z_i(a) = (sim_i(a) - mean_a sim_i(a)) / std_a sim_i(a)`. This is free — you have already computed the full 230 × 400 matrix — and it converts "close to a broad term" into "unusually close to *this* term," which is what you want for both the score and the top-3 explanation shown to Stage B. Persist raw similarity too, as §9.2 already requires.
+
+Relatedly: `input_type = "query"` already causes Voyage to prepend "Represent the query for retrieving supporting documents" (verified in the live API reference), so the `"Articles about: "` prefix is a second, redundant instruction that is identical across all 230 interests — it pulls all interest vectors toward each other and further compresses the `top1 − top3` gap. Worth testing the bare interest name as v2 of the interest text format.
+
+### M4. Exploration is unbounded at exactly the moment it does the most damage
+
+§15's V1 definition admits candidates "from a feed with low rating evidence **or** semantically outside the dense region of recent positive ratings." On day one there are no ratings, so *every* feed has low evidence and there is no positive region — the predicate is universally true, and 20 slots in the recall pool plus a guaranteed shortlist reservation (§18.2 rule 4) plus Stage B exposure go to articles chosen by `hash(run_date, article_id)`. That is a lot of deliberate noise injected during the phase where you are trying to measure whether the new ranker beats the old one.
+
+"Semantically outside the dense region" is also the one place the plan drops below implementation grade — no definition, no threshold.
+
+**Fix:** make `recall_exploration` scale to zero when `total_decayed_rating_weight` is below a threshold (~15), and define "outside the dense region" concretely as `positive_similarity < 25th percentile of the day's candidate distribution`. Default the reservation to 8, not 20, until Phase D.
+
+### M5. Phase A shadow mode cannot shadow what the plan implies it shadows
+
+> §28 Phase A: "New ranker computes in shadow mode and persists what it _would_ have done. Compare old vs new selections for several days."
+
+The new utility score (§17) is 40% LLM quality + 15% reader fit, and those fields do not exist until Phase C enables the new Stage A. So the Phase A shadow can only compute the non-LLM 45% of utility, and "compare old vs new selections" is not achievable — the shadow shortlist would be ranked on less than half its intended signal.
+
+That is fine, and the honest framing is more useful anyway: **Phase A should shadow the recall and pre-Stage-A stages only**, which is precisely §21.2's metric 6 ("count historical upvoted articles that would have been lost at each proposed stage") — the plan's own "one of the most important metrics." That comparison is fully computable in Phase A and is the single best evidence for whether the recall redesign is justified.
+
+**Fix:** rewrite Phase A's exit criterion as "recall-boundary diagnostics show the union recovers upvoted articles the current top-120 would have dropped," and move selection comparison to Phase C.
+
+### M6. Voyage's daily ceiling will not survive a rerun, unlike DeepSeek's
+
+§23 promises "DeepSeek and Voyage meters trip independently," but §7.4 says "Database columns for Voyage tokens/cost are optional in the first migration if the JSON report is sufficient." DeepSeek's ceiling is day-scoped because `pipeline.rs:379-386` preloads `db.spend_for_date(date)` from the `runs` table. Without an equivalent column, `voyage.max_daily_usd` is per-*invocation*, and `generate --date X` reruns are a normal, documented workflow (idempotency is a stated invariant).
+
+Impact is genuinely low — the embedding cache makes reruns nearly free — but the asymmetry is a trap for whoever debugs a budget trip later.
+
+**Fix:** add `voyage_input_tokens` / `voyage_cost_usd` columns to `runs` in `0002` and preload them the same way. It is four lines and removes a whole class of confusion.
+
+### M7. Replay is reproducible for scalars but not for vectors, and the plan overstates it
+
+> §21.1: "For exact future replay, `candidate_rankings` becomes authoritative."
+
+`article_embeddings`'s primary key is `(article_id, model, dimension)` with `input_hash` as a *non-key* column, so a re-extraction overwrites the vector in place. `upsert_article` overwrites `content_html` on every re-ingest of the same `canonical_url` (`db.rs:272-279`), which happens routinely because the 26h lookback window overlaps consecutive days. So the vectors used for MMR and for §21.2's metric 5 (shortlist diversity) are not recoverable for a past date.
+
+Additionally, percentile normalization is **day-relative**: re-tuning weights on historical data requires recomputing percentiles from the full day's candidate set, which requires a `candidate_rankings` row for every eligible article. §5.4 provides that (and H6 would complete it), so the scalar path works — but only if the evaluator recomputes percentiles from stored *raw* values rather than trusting stored normalized ones.
+
+**Fix:** state explicitly in §21.1 that (a) `evaluate` recomputes normalization from raw columns and never trusts persisted normalized values across code changes, and (b) vector-dependent metrics (diversity, MMR replay) are approximate for historical dates. Do not add embedding versioning to fix this — the storage cost is not worth it; just stop claiming exactness.
+
+### M8. `--max-articles` is not currently a hard ceiling, and the plan assumes it is
+
+> §19.2: "`--max-articles N` should **remain** a hard ceiling/override, not a target that forces filling."
+
+It is not one today. `pipeline.rs:188` assigns `--max-articles` to `target`, and `select.rs:187-193` derives `(target - 5, target + 5)`. So `--max-articles 10` today permits **15** picks and forces a floor of 5. An agent reading "remain" will assume the behavior already exists and not fix it.
+
+Also unaddressed: how `--max-articles` composes with the new `max_article_count = 25`. Presumably `effective_max = min(max_article_count, --max-articles)` with no floor at all.
+
+**Fix:** reword to "`--max-articles N` must **become** a hard ceiling" and specify the composition rule.
+
+---
+
+## Low
+
+- **L1 — `assemble()` loses its sort key.** §26 says to replace `ScoredArticle::combined_score()` with the `rank.rs` utility, but `assemble` uses `combined_score()` in three places (`select.rs:545`, `:609`, and via `sort_by_combined` at `:563`) for oversize trim and intra-section ordering. §19.2 says "keep the max-size trim" without saying what it sorts by. Specify: trim and order by `utility_score`, falling back to `prefilter_score` when utility is absent.
+- **L2 — `select_without_llm` ordering not updated.** §23 says the DeepSeek-unavailable path should "use the enhanced deterministic ranking … rather than reverting all the way to old prefilter order," but §26's `select.rs` bullets don't mention `select_without_llm`, which calls `sort_by_prefilter` directly (`select.rs:660`). Cross-reference the two sections.
+- **L3 — MMR seed rule looks like a slip.** §18.2 rule 1: "Seed with the highest-utility **non-auto** candidate." If an auto-include is the day's best article, it should seed. Also unstated: whether the rule-5 force-preserved top-20 count as `already_selected` for the max-similarity term (they must, or MMR will re-select near-duplicates of them).
+- **L4 — "exploration/novelty bonus" has no definition.** §14.2 gives it 0.05 of the pre-Stage-A blend, but §15 defines exploration as boolean set membership. Either make it a flat additive bonus for `exploration_candidate` articles, or define the continuous novelty measure (e.g. `1 - max similarity to the positive centroid`).
+- **L5 — `article_facets` omits `model` from its primary key** while `article_embeddings` includes `model`. Switching DeepSeek models silently reuses facets extracted by the previous one. Either add `model` to the key or state that facets are deliberately model-agnostic and that `prompt_version` is the invalidation lever.
+- **L6 — `--skip-llm` / `--skip-embeddings` are inconsistent.** §23 recommends that `--skip-llm` also disable Voyage generation "and add `--skip-embeddings` later only if a real operator need appears" — but §20 already specifies `features backfill --embeddings-only`, which *is* that need. Cleaner: `--skip-llm` gates DeepSeek only, `--skip-embeddings` gates Voyage, both shipped in the same commit. One extra boolean.
+- **L7 — `prefilter_keep` validation must move.** `config.rs:348` enforces `prefilter_keep >= target_article_count`. If §25 deprecates or aliases it to `stage_a_keep`, that check needs relocating, and the new constraints (`recall_pool_keep >= stage_a_keep >= shortlist_keep`, `0 <= diversity_lambda <= 1`) need adding. §26's config bullet lists some of this; add the ordering constraints explicitly.
+- **L8 — Correlated "independent" signals.** §16.3 correctly forbids showing numeric preference scores to the reader-fit rubric, but reader-fit *is* shown the taste profile, whose learned-adjustments section is now (§12) enriched with facet data derived from the same ratings that produce `facet_preference`. §17's claim that "30% of the score is direct rating-derived preference" understates the true rating-derived share (~40%) and, more importantly, those components share error. Not a blocker — just note it as a correlation to watch in §21.2 rather than asserting independence.
+- **L9 — Dry-run behavior with `candidate_rankings` unstated.** Articles are persisted even under `--dry-run` (`pipeline.rs:320-332`), so ranking rows will be written on dry runs. Probably desirable for Phase A shadow work; say so.
+
+---
+
+## Nits
+
+- **N1 — Stale file references.** §"read the current curation implementation" lists `src/curate/profile.rs`; it is `src/curate/profile/mod.rs` plus `themes.rs`. §6's module layout repeats the flat `profile.rs` and omits `editorial.rs`, which exists. An agent following §6 literally might collapse the profile module. For a plan that is explicitly "implementation-grade," these should be exact.
+- **N2 — "configured lookback" for the churn rule doesn't exist.** §13.1 says "within the configured lookback"; `STALE_LOOKBACK_DAYS` is a `const` (`prefilter.rs:59`), not config. Either make it config as part of this work or drop the word.
+- **N3 — Acceptance criterion 11 is nearly vacuous.** "`features backfill` can populate at least all historical rated articles **without network calls in tests**" — every test is offline per notes §6. The meaningful criterion is "backfill is resumable and idempotent: re-running it makes zero API calls when the cache is warm."
+- **N4 — Root config section is not typo-protected.** Nested config structs use `#[serde(deny_unknown_fields)]`, but `Config` itself deliberately does not (`config.rs:41-43`, so that bare `DAILY_EPUB_SECRET` passes through). A `[voyages]` typo will therefore be silently ignored and defaults used. Worth a line in §7.1 telling the operator to verify via the startup log rather than assuming.
+- **N5 — `voyage.max_daily_usd = 0.25` is a runaway guard, not a cost ceiling.** At `$0.02`/Mtok that trips at 12.5M tokens/day, ~25× expected volume, and the meter cannot know whether the 200M free allocation is exhausted. Fine as designed — just describe it as a runaway guard so nobody tunes it as if it were a bill.
+- **N6 — `PENALTY_TITLE_PATTERNS` becomes redundant once facets exist.** `ArticleFormat::{roundup, release_notes, announcement}` subsumes the 22-pattern title list (`prefilter.rs:27-50`) with far better recall. §26 says to preserve current heuristic values for evaluation, which is right for V1; add a note that retiring the title-pattern penalty is a Phase E cleanup candidate.
+- **N7 — Verified-facts section deserves a re-verification date.** §3's Voyage facts are correct as of today (I confirmed model, context, dimensions, dtypes, 1,000-input/1M-token request limits, `$0.02`/Mtok, and the 200M free allocation against the live docs). Add a note that this block should be re-checked whenever `model` changes, in the same spirit as the 2026-08-15 notes.
+
+---
+
+## Alternatives
+
+### A1. Skip facets in V1; fit a regularized linear probe on the embeddings instead *(strongest alternative)*
+
+The plan's own §11.2 already computes a positive and a negative centroid. The difference of two class centroids is the closed-form solution of a specific naive classifier — it weights every embedding dimension equally. A **ridge-regularized logistic regression** on the same labels is the same idea done properly: it learns *which* dimensions discriminate, it is convex, it needs no new dependency (a few hundred lines of gradient descent over `Vec<f32>`, or closed-form ridge on 256-d), and it handles the correlated-positives problem in §11.2 that the centroid difference cannot.
+
+Crucially, it addresses H2's evidence problem head-on: with 63 labels you cannot estimate 84 facet parameters, but you *can* fit a heavily-regularized 256-dimensional linear model, because regularization is exactly the tool for the low-n regime — and it costs **zero DeepSeek tokens**.
+
+"Likes first-hand postmortems, dislikes vendor announcements" is substantially linearly separable in a good embedding space; that is what embeddings are for. Facets buy explainability and the §12 profile-prompt enrichment, both real, but neither requires facets to be in the *scoring* path.
+
+**Prefer this when:** rating volume is under ~200 and you want the immediate-feedback property (§30 criterion 4) working on day one. **Prefer the plan's approach when:** rating volume is high enough to estimate facet stats, or when the explainability of "you downvote `commerciality = product_marketing`" is worth more than ranking accuracy — which for a single-reader system it genuinely might be.
+
+**Concrete middle path:** ship facets for extraction, storage, `explain`, and the §12 profile prompt (all cheap and all valuable), but have the *numeric* scoring path use a linear probe on embeddings until facet evidence crosses a threshold. That gets both properties and lets §21 compare them empirically.
+
+### A2. Threshold clustering instead of MMR for diversification
+
+§18's MMR introduces `lambda = 0.82`, a magic number whose meaning is not interpretable in isolation, and it composes awkwardly with rule 5 ("preserve the top ~20 by raw utility regardless of MMR" — at which point you are no longer running MMR, you are running force-include-then-MMR).
+
+**Alternative:** single-linkage cluster the shortlist candidates at cosine > ~0.85, then take the top-N by utility with a cap of 2 per cluster. One interpretable parameter (the similarity threshold, which you can eyeball against real article pairs), trivially composable with auto-includes and the top-20 preservation rule, and dramatically easier to render in `explain` ("suppressed: 3rd article in the cluster led by #4821").
+
+**Prefer MMR when** you want a smooth relevance/diversity tradeoff across the whole ranking. **Prefer clustering when** the actual problem is the one §32 describes — "six articles about the same AI news cycle" — which is a discrete cluster-cap problem, not a continuous one. I would ship clustering first and reach for MMR only if it proves too blunt.
+
+### A3. Two vectors per article: title+lead for interest matching, full body for preference and MMR
+
+Addresses H3 and M2 at the root rather than by filtering. The short-document bias in interest matching is not really a bug — it reflects that interest matching *should* be title-like. The problem is using one vector for two jobs with opposite length preferences.
+
+Embed `Title + first ~100 words` with `input_type = "document"` for standing-interest matching, and the full capped body for the rating centroids and MMR. Cost is one extra Voyage call per batch (~13/day, negligible against a 200M free allocation), storage doubles (mitigated by A4), and both jobs get a vector shaped for them. The `article_embeddings` PK already accommodates this if you add a `kind` column or fold it into `model`.
+
+### A4. Default to `output_dimension = 512`
+
+Voyage's Matryoshka training makes 512-d near-lossless for retrieval on general text. It halves BLOB storage (~300 MB/year instead of ~600), halves every dot product (230 interests × 400 articles × 2 = 184K dot products/day either way — not a bottleneck, but MMR is O(n²)), and the plan already mandates configurability. For a single-reader system where the DB shares a VPS with the EPUB output directory, 512 is the better *default*; 1024 is the thing you evaluate into if §21 shows a measurable difference. This also makes A3's second vector free in storage terms.
+
+---
+
+## Open Questions
+
+These block confident approval; each is answerable quickly and each changes what gets built.
+
+1. **What writes `scores.llm_score` after the Stage A split?** (C2) Without an answer the churn rule dies silently. If the answer is "nothing, migrate `recently_low_scored_ids` to `candidate_rankings`," the `0002` migration and `prefilter.rs` change scope.
+2. **How many ratings exist in the production database today?** If the count is under ~100, H2 stops being a tuning concern and becomes blocking: the entire facet-scoring path would be dead weight for the first several months, and A1 becomes the right V1.
+3. **How many rows are in the production `articles` table?** Determines backfill token spend (M1) and whether the ~600 MB/year embedding storage projection is tolerable on the host. Answerable with one `SELECT COUNT(*)`.
+4. **What is the current wall-clock runtime of `generate`, and what is the hard publish deadline?** (H4) If the run currently takes 4 minutes against a 30-minute window, serial batching is fine and H4 downgrades to a nit. If it takes 20 minutes, concurrency is mandatory before any of this ships.
+5. **Does `[curation.personalization] enabled = false` (§28) disable embedding *generation*, or only the new ranking path?** If it disables generation, Phase A collects no data and the shadow phase is impossible. If it does not, the flag name is misleading and the config docs need to say so.
+6. **Is the reader willing to accept a visibly noisier paper during Phase A–B?** M4's exploration reservation plus H3's short-document admissions will both be most visible exactly when the system has the least evidence. If the answer is no, exploration should start at zero and ramp with rating volume.

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-18.md

@@ -0,0 +1,173 @@
+# Review: Personalized Ranking, Embeddings, Facets, and Feedback
+
+The plan has a strong overall direction—union-based recall, separate descriptive facets, explicit quality/fit scores, pre-editor diversification, and removal of forced filler are all sensible—but it is not ready to execute as written. The principal blockers are that the proposed persistence model cannot provide the promised per-run/exact replay guarantees, facet cache invalidation contradicts its schema, historical runs can leak future feedback, the single positive/negative centroid is too lossy for the stated multi-interest personalization goal, and the rollout/budget and target/ceiling semantics are internally inconsistent. Resolve the High findings before implementation; the remaining items can be handled during staged delivery.
+
+## Critical
+
+No critical findings.
+
+## High
+
+### 1. The facet cache key cannot honor the stated invalidation rules
+
+Evidence: §5.3 defines `PRIMARY KEY (article_id, schema_version)` while also storing `model`, `prompt_version`, and `input_hash`; it says “`prompt_version` changes when instructions change” and “Reuse a facet row only if schema version and content hash match.” §10.4 then describes the cache as `(article_id, schema_version, input_hash)`.
+
+With the proposed primary key, a prompt or model change either reuses stale output or overwrites the previous observation. Worse, §10.3 proposes placing the mutable reader taste profile in the system prompt for a supposedly descriptive extraction task, but neither the profile version nor its text hash participates in invalidation. Facets would therefore be reader/profile-dependent while appearing globally reusable and stable.
+
+Recommendation:
+
+- Remove the taste profile entirely from facet extraction. Use a stable, reader-independent descriptive system prompt and treat article text as untrusted quoted data.
+- Make the cache identity explicit, for example `(article_id, schema_version, model, prompt_version, input_hash)`, or use a surrogate `facet_observation_id` plus a uniqueness constraint over those fields.
+- Define `input_hash` over the exact effective facet input: excerpt-format version, title/author/source fields actually sent, and the representative text. Do not call it merely a “content hash.”
+- Decide whether old observations are retained for replay or superseded; do not silently overwrite them if exact replay remains a goal.
+
+### 2. `candidate_rankings` is neither per-run nor sufficient for exact replay
+
+Evidence: goal 7 promises “replayable from persisted per-run features,” §5.4 calls the table essential, and §21.1 says it becomes authoritative “For exact future replay.” Yet its primary key is `(run_date, article_id)`, despite the existing `runs` table allowing multiple invocations per date. The plan explicitly says reruns replace the date’s rows.
+
+This loses shadow-versus-live results, failed/partial attempts, changed configurations, and the exact feature versions used. It also omits the ranking configuration/profile/model/prompt versions and per-retriever membership/cut reason. Mutable `article_embeddings` and `article_facets` rows can be overwritten, so a later MMR replay cannot reconstruct the original pairwise similarities. `explanation_json` is not a substitute unless its required schema and versioning are specified.
+
+Recommendation:
+
+- Key snapshots by `run_id` (foreign key to `runs`) and `article_id`; keep `run_date` as an indexed denormalization.
+- Add a run-level immutable ranking manifest containing personalization mode (`shadow`/`live`), all normalized weights and thresholds, model/dimension versions, facet schema/prompt version, embedding/excerpt format versions, profile version/hash, feature availability, and deterministic algorithm version/seed.
+- Persist retriever membership and explicit terminal reason (`blocked`, `published_before_cutoff`, `not_in_union`, `recall_cap`, `stage_a_cut`, `mmr_cut`, etc.). The current booleans cannot explain why an article failed to enter a stage.
+- Either retain versioned embedding/facet observations referenced by the run or persist enough pairwise/MMR inputs to reproduce the shortlist. Narrow the claim from “exact replay” to “score diagnostics” if that storage is not desired.
+- Write a run snapshot transactionally or mark its lifecycle (`running`, `complete`, `failed`) so evaluation does not treat a partial run as authoritative.
+
+### 3. Historical generation/evaluation has undefined “as-of” semantics and can leak future data
+
+Evidence: §11 says to build preference state “at the beginning of every generate run” using recent ratings, §20 adds historical backfill/evaluation, and §21 proposes historical replay. The plan does not say that ratings must be bounded by the simulated run time. It also says recall should reuse current prefilter history logic (§13.1/§26), but the current repository’s `previously_published_ids()` returns articles from every issue, including issues after a historical target date, and the current profile code anchors its rating window to `Timestamp::now()`.
+
+A replay of August 1 performed on August 18 could train on August 2–18 votes, use the latest prose profile, and exclude an article because it was published on August 10. That produces optimistic evaluation and non-reproducible historical runs.
+
+Recommendation:
+
+- Define one `as_of` timestamp for every run/evaluation and require all ratings, issue history, feed priors, profile state, and candidate publication/history queries to use it.
+- For an issue dated `D`, exclude only articles published in issues before the chosen cutoff; never use future issue rows.
+- Store or reconstruct profile versions by effective interval. If historical prose profiles are unavailable, explicitly disable that feature in replay and label the limitation.
+- Separate “regenerate an old issue using knowledge available today” from “historical replay as of that day” as distinct CLI modes.
+- Add leakage tests in which future ratings/issues exist but do not affect an as-of replay.
+
+### 4. One global positive and negative centroid collapses the reader’s multi-modal taste
+
+Evidence: §9 describes roughly 220 standing interests, while §11.2 reduces all recent upvotes to one unit-normalized positive centroid and all downvotes to one negative centroid. §32 expects a highly favored niche to be rescued early.
+
+A single average vector is a poor representation of a reader who likes unrelated clusters such as Rust, local Boston reporting, books, and e-ink. Niche vectors may be only weakly similar to the global mean. The negative centroid also conflates topic rejection with format/quality rejection; the plan acknowledges this caveat but still gives the combined embedding preference 15–28% of ranking weight. This can work against the main recall goal before facets or Stage A can rescue the article.
+
+Recommendation: evaluate a signed, time-decayed top-k neighbor signal as the V1 baseline (`top/mean similarity to recent upvotes` minus a configurable downvote term), optionally grouped by topic cluster. At this scale it is simpler than centroid maintenance and preserves multiple modes. If centroids remain, keep several clusters or combine centroid and nearest-neighbor signals, and make a synthetic multi-interest recall test an acceptance criterion—not just a one-topic centroid test.
+
+### 5. Shadow mode can consume or trip the same DeepSeek budget needed by the production selector
+
+Evidence: §28 Phase A says facets and new snapshots run in shadow while “Existing production selection remains authoritative.” §10.3 adds facet extraction for about 240 articles, and §23 says DeepSeek’s meter trips independently once its ceiling is reached. The current pipeline uses a single per-day DeepSeek meter for scoring, selection, profile, and editorial work.
+
+If shadow facet calls run before the existing production stages, they can exhaust the shared daily ceiling and force the supposedly authoritative path into fallback. This is a behavior change, not passive shadowing. Persisting Voyage cost only in report JSON also cannot reliably preload/enforce a provider-specific daily ceiling across reruns or dry runs.
+
+Recommendation:
+
+- Run authoritative production calls first, then shadow work from a separately configured shadow budget, or reserve explicit provider/stage budget slices before shadow calls.
+- Persist provider usage by `run_id` in queryable columns or a `run_provider_usage` table and preload same-date spend, as the current DeepSeek path does. “JSON is sufficient” is not compatible with a durable daily guardrail.
+- State whether failed requests, retries, dry runs, and concurrent runs count toward the budget, and make reservation/accounting atomic enough to prevent two runs overspending simultaneously.
+
+### 6. Soft target, hard ceiling, auto-includes, and `--max-articles` have contradictory precedence
+
+Evidence: §19.2 says target approximately 20, never exceed `max_article_count` 25 “plus unavoidable auto-includes,” then says `--max-articles N` is a “hard ceiling/override.” §18.2 also allows auto-includes and other protected sets to exceed nominal shortlist size. In current code, `--max-articles` replaces `target_article_count`, so merely reusing the existing plumbing would tell Stage B to aim for the ceiling rather than keep the normal soft target.
+
+Recommendation:
+
+- Carry `soft_target` and `hard_max` as separate values through the pipeline and Stage B prompt.
+- Define one precedence rule for auto-includes. Either they can exceed the hard max (then call it a normal-content ceiling and report the exception) or the CLI ceiling is truly hard (then reject conflicting configuration or specify which auto-includes win).
+- Define the exact shortlist-cap precedence among raw-utility preservation, exploration reservation, auto-includes, and MMR; specify whether protected items seed MMR similarity calculations.
+- Add tests for `--max-articles` below, equal to, and above the soft target, including excess auto-includes.
+
+## Medium
+
+### 1. Missing-signal normalization is underspecified and can turn degradation into a penalty
+
+Evidence: §14 percentile-normalizes daily signals, while §23 says missing embedding signals become “neutral, not zero-quality penalties.” The plan does not define the empirical population, neutral value, behavior for all-equal/singleton inputs, or whether missing values participate in percentile ranking.
+
+Recommendation: define a typed normalization contract. Exclude missing values from the empirical CDF, assign missing signals an explicit neutral value (normally 0.5), return 0.5 for degenerate/all-tied distributions, and store availability flags alongside raw and normalized values. Test mixed cached/missing embeddings so an outage does not systematically demote uncached articles.
+
+### 2. The preference evidence model needs vote-time and exposure semantics tightened
+
+Evidence: §11 decays ratings by `age_days` without saying whether age is based on `rated_at` or `issue_date`; §5.5 keeps `included` exposure metadata but §11.3’s support uses only explicit votes. A vote can arrive long after publication, and repeated flips update `rated_at` in the current schema.
+
+Recommendation: use `rated_at` for behavioral recency but document that a flip resets recency, or preserve the initial and last-changed timestamps separately. Keep exposure out of the label, as planned, but persist whether/when an item was shown so evaluation can distinguish unshown, shown-unrated, and rated items. Anchor all calculations to the run’s `as_of` timestamp.
+
+### 3. Feed-prior rebuilding needs an atomic, fully specified source policy
+
+Evidence: §5.5 says split credit among direct `Feed` sources, fall back to `best_entry_id`, then use a “weighted mean” of future source priors without defining weights. §22 allows either recomputation or transactional updates. The current rebuild path performs independent upserts and does not clear obsolete rows.
+
+Recommendation: define deduplication by `feed_id`, the exact candidate-affinity weights, and what happens when the best-entry fallback is itself a discovery feed. Recompute the complete v2 table in one transaction (temporary table plus replace, or delete/upsert under a transaction) so generation never reads a partial rebuild and stale zero-evidence rows disappear.
+
+### 4. LLM inputs need an explicit prompt-injection and data-handling policy
+
+Evidence: §§10.3, 16.4, and 19.1 send extracted third-party article text to DeepSeek, and §8 sends up to 60,000 characters to Voyage. The plan discusses API-key secrecy but not untrusted instructions embedded in article text, provider data handling, or operator opt-out for private/authenticated feeds.
+
+Recommendation: delimit article content as data, explicitly instruct the model to ignore instructions found inside it, validate outputs only against offered IDs/enums, and cap/escape metadata consistently. Document that article text is sent to external providers and add a feed/domain-level “local only / do not send” policy if private feeds are possible. Confirm the providers’ retention/training terms before rollout.
+
+### 5. The new facet stage is expensive but its necessity is not tested against cheaper alternatives
+
+Evidence: §10 adds a DeepSeek call for roughly 240 articles every day before Stage A, while §27 tests parsing and caching but not whether facets are stable or improve ranking. Many proposed fields (depth, technicality, temporal orientation, commerciality) may be derivable cheaply or bundled into Stage A for only 120 candidates.
+
+Recommendation: in shadow mode, measure inter-run facet stability and incremental ranking value. Consider extracting cheap deterministic facets before recall, asking Stage A for descriptive facets alongside quality/fit for its 120 candidates, or using the embedding provider only for topic retrieval and delaying facets until enough ratings justify them. Keep the separate 240-item call only if it measurably rescues candidates at the pre-Stage-A cut.
+
+### 6. The migration and rollout plan needs explicit compatibility behavior for partially deployed code
+
+Evidence: §5 says keep `scores` temporarily, §26 says split `LlmScore`, and §28 phases behavior over several deployments, but the plan does not define which binary versions can safely run against migration 0002 or how Stage A v1/v2 values coexist. `scores.llm_score` becomes semantically ambiguous during Phase B/C.
+
+Recommendation: version the assessment (`assessment_version`, model, prompt version), keep v1 and v2 writes distinguishable, and specify read precedence during each phase. Add forward/backward deployment tests or explicitly require a stop-the-world binary migration for this single-operator service.
+
+## Low
+
+### 1. The provider facts are current, but pricing and compatibility should remain metadata
+
+The choices in §3 match the current official Voyage documentation: `voyage-4-lite` supports the stated context length/dimensions and request limits, and the listed price is currently correct. The provider also states that Voyage 4-series embeddings are mutually compatible, which means the plan’s blanket “Never compare vectors with different model” rule is conservative rather than technically required. Conservative isolation is reasonable for reproducibility; record model IDs in the run manifest and only relax compatibility after an explicit evaluation. See [Voyage embeddings](https://docs.voyageai.com/docs/embeddings), [API reference](https://docs.voyageai.com/reference/embeddings-api), and [pricing](https://docs.voyageai.com/docs/pricing).
+
+### 2. Character caps must be Unicode-safe and do not guarantee the claimed token margin
+
+Evidence: §7.3 uses 60,000 normalized characters per article as an aggregate-token safety proxy, while §8.1 says to cap text but only §10.2 explicitly warns against unsafe UTF-8 splitting.
+
+Recommendation: make every cap Unicode-safe, enforce both per-input and aggregate character budgets before batching, and handle server truncation/token-limit errors explicitly. Log truncation counts without article text.
+
+### 3. MMR should clamp/validate similarities and define the utility scale
+
+Evidence: §18 uses `normalized_utility` but §17 stores utility on a 0–100 scale. Floating-point embeddings loaded from storage are only length-checked, not checked for finite values or unit norm.
+
+Recommendation: define whether MMR uses utility divided by 100 or a percentile, reject non-finite vector values, and normalize or verify vector norms within tolerance before dot products. Clamp small floating-point overshoots when using cosine-like scores.
+
+## Nits
+
+- §5.4 says “every post-hygiene candidate,” while acceptance criterion 10 says “every eligible daily article.” Define whether blocked, previously published, and recently rejected articles receive rows with terminal reasons. Logging them is necessary to explain hard exclusions.
+- Use SQLite integer `0/1` plus `CHECK` constraints for ranking flags if strictness matters; SQLite does not enforce a separate Boolean storage class.
+- `specific_topics: Vec<String> // 0..=4` and other cardinality comments require explicit validation after deserialization; Serde alone will not enforce them.
+- The exploration hash should include an algorithm/version salt in the run manifest so an implementation change does not masquerade as reproducible behavior.
+- Replace approximate terms such as “top ~20” and “small number of exploration candidates” with configuration fields and deterministic defaults before handing the plan to an implementation agent.
+
+## Alternatives
+
+### Alternative A: signed nearest-neighbor preference instead of one centroid
+
+Keep recent rated document embeddings and compute a time-decayed top-k positive similarity and top-k negative similarity for each candidate. This preserves unrelated interest clusters, is explainable (“similar to these three upvotes”), and is trivial at the repository’s scale. Prefer this for V1 when ratings are sparse and multi-modal. Add clustered centroids later if the rating history becomes large enough for neighbor scans to matter.
+
+### Alternative B: immutable feature observations plus run manifests
+
+Store embeddings and facets as immutable observations keyed by model/prompt/input versions, and let each `candidate_ranking` reference the exact observation IDs plus a run manifest. Prefer this when reproducibility and offline tuning are real product requirements. If storage simplicity matters more, retain mutable caches but explicitly downgrade the promise to diagnostic snapshots rather than exact replay.
+
+### Alternative C: fold facets into Stage A initially
+
+Ask the existing 120-candidate Stage A call to return descriptive facets alongside quality and reader fit, then use those facets in final utility and future preference learning. Prefer this during early shadowing to avoid doubling DeepSeek candidate volume. A separate 240-candidate facet stage is preferable only if shadow evaluation shows facet preference materially improves the 240-to-120 cut.
+
+### Alternative D: production-first shadow execution
+
+Run the unchanged authoritative curation path first, reserve enough budget for all publication-critical calls, and execute new facet/ranking work afterward with its own cap. Prefer this during Phase A because it makes “shadow” genuinely non-interfering. Once the new path becomes authoritative, consolidate stage budgets under the provider-level ledger.
+
+## Open Questions
+
+1. Is a historical `generate --date D` supposed to reproduce knowledge available on day D, or intentionally re-curate D using today’s ratings/profile? The database queries and CLI need separate semantics for these two operations.
+2. Is `--max-articles N` an absolute ceiling even when there are more than N auto-includes? Which requirement wins?
+3. Should facet labels be globally descriptive and reader-independent? If yes, confirm that the taste profile will be removed from the facet system prompt.
+4. Is exact replay a hard requirement? If so, is retaining immutable feature observations and run manifests acceptable, including the extra storage?
+5. How much of the DeepSeek daily budget is reserved for publication-critical Stage A/B/editorial work versus shadow facets, profile rebuilds, and evaluation/backfill?
+6. Are any Miniflux feeds private, authenticated, or otherwise unsuitable for sending article text to Voyage/DeepSeek?
+7. What minimum evidence must the shadow evaluation meet before moving between rollout phases (number of runs/ratings and explicit pass/fail thresholds), rather than “several days” or “monitor”?

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-19-2.md

@@ -0,0 +1,184 @@
+# Re-review — Personalized Ranking, Embeddings, Facets, and Feedback (v3)
+
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md`  
+**Date:** 2026-08-19  
+**Scope:** revision 3, after the R3 amendments
+
+## Verdict
+
+Revision 3 resolves every R3 finding thoughtfully, and the main architecture is now coherent: the provider boundary is explicit, run eligibility uses the real lifecycle, profile text is versioned, facets are post-admission, and leader clustering is specified rather than mislabeled. I still would **not begin the full implementation unchanged**. Two remaining correctness defects sit below those amendments: the single global evidence weight can fully activate a learned signal backed by almost no compatible observations, and the stated `as_of`/backfill contract cannot be implemented with the proposed mutable feed-prior and date-keyed score stores. The generation lease also needs fencing and an active heartbeat before it can provide the mutual exclusion the plan claims. These are bounded amendments; the union-admission, utility, facet, and diversification decisions do not need to be revisited.
+
+## Critical
+
+### C1. One global evidence weight does not measure evidence for each learned signal
+
+Evidence:
+
+- §14 defines one `W` as the decayed weight of **all** ratings and uses it to gate `embedding_preference`, `facet_preference`, and `feed_affinity`.
+- §13.4 says a rating is useful for embedding/facet preference only when its article has the corresponding features and that missing historical features “reduce evidence.”
+- §25.1 guarantees protected articles receive no embedding or facets, but then says their ratings still update both `feed_priors_v2` **and the kNN preference state**.
+
+Those statements cannot all hold. A newly protected article has no embedding, so its rating cannot enter kNN. More generally, 19 ratings without compatible embeddings plus one compatible rating produce `W = 20`, fully activating an embedding-preference signal learned from one example. The same issue applies independently to facets. Presence-aware blending does not fix it: once one compatible example makes the signal `Present`, the unrelated global `W` gives it full configured weight.
+
+This recreates the sparse-evidence failure the ladder was designed to prevent, especially after provider opt-outs, partial backfills, model/dimension changes, or facet parse failures.
+
+**Required amendment:** maintain evidence per learned signal:
+
+```text
+W_embedding = decayed weight of ratings with a compatible article embedding
+W_facet     = decayed weight of ratings with usable scored facets
+W_feed      = decayed weight successfully attributable to a feed
+W_global    = decayed weight of all ratings (telemetry/exploration maturity only)
+```
+
+Compute each gate from its own `W_signal`; facet value support remains an additional inner gate. Persist all four values in the finalized manifest/explanation rather than one `rating_evidence_weight`. A protected rating may update feed affinity and `W_global`, but it cannot update kNN or facet preference unless a locally generated compatible feature exists. Add a regression test where 20 total ratings but only one embedding-backed rating leave the embedding gate near its floor rather than fully open.
+
+### C2. The `as_of` and historical-backfill contracts remain unsatisfiable with the proposed stores
+
+Three plan requirements conflict:
+
+1. **Feed priors:** §6.1 requires feed priors recomputed from ratings bounded by `as_of`, while §7.7 defines one singleton `feed_priors_v2` table rebuilt by `DELETE; INSERT`. A historical replay would overwrite today’s materialized priors with a past snapshot. Meanwhile the rating server can rebuild the same table while generation holds its lease, because `serve` is not lease-protected. Atomic replacement prevents partial reads but not “wrong snapshot won the race” or future-data leakage relative to a run’s fixed `as_of`.
+2. **Churn scores:** §6.1 says churn is as-of bounded, but §18.4 keeps it on `scores.llm_score`. That table is keyed only by `(article_id, run_date)` and has no `run_id` or creation timestamp. A recuration performed on August 19 for an August 1 issue writes an August 1 score; a replay as of August 5 cannot tell that the score was created in its future. The proposed `recently_low_scored_ids(threshold, since, until)` bounds the nominal date, not observation time.
+3. **Backfilled features:** §6.1 says replay ignores embedding/facet rows created after `as_of`, while §27.1 says to backfill embeddings/facets now and then replay historical dates. Those newly backfilled rows are, by definition, created after the historical `as_of`, so the replay must ignore the very features the backfill was intended to supply.
+
+Acceptance criterion 13’s leakage test cannot make all three behaviors correct without choosing stronger semantics.
+
+**Required amendment:** separate two operations that v3 still calls `replay`:
+
+- **Fidelity replay:** reconstruct only information actually available then. Ignore later-created features; derive feed affinity in memory from canonical ratings bounded by `as_of`; read churn assessments from run-scoped observations joined to `runs.started_at <= as_of`. Legacy `scores` rows without observation time must be excluded or explicitly treated as unverifiable.
+- **Counterfactual evaluation:** ask how the new algorithm would rank a historical candidate set using features computed later. Permit marked post-hoc embeddings/facets, record `feature_time_policy = counterfactual`, and label results approximate because `content_html` is mutable.
+
+Do not rebuild the global `feed_priors_v2` table for replay. Either compute a run-local `HashMap<FeedId, FeedPriorV2>` directly from bounded ratings, or persist a run-scoped snapshot. Keep the singleton table only as a current/live cache if the rating endpoint still needs it. Move the churn rule to `candidate_rankings.llm_quality_score` joined through eligible runs/manifests, or add `run_id`/actual `scored_at` provenance to `scores`; nominal `run_date` is insufficient.
+
+## High
+
+### H1. The expiring generation lease is not fenced and can expire during a live stage
+
+Evidence:
+
+- §7.4c gives the lease a 30-minute default TTL and refreshes it only “at each stage boundary.”
+- §24.1 applies the same lease to `features backfill`, which may run for many batches, and acknowledges that Stage A wall clock is currently unmeasured.
+- §31.11 tests racing acquisition and expired recovery, but not a live operation lasting longer than the TTL or an old holder acting after reclamation.
+
+If one stage lasts longer than 30 minutes, a second process may reclaim the lease while the first is still running. Both then proceed. Worse, an RAII guard from the original process can later delete or refresh the replacement owner’s row unless every operation is conditional on an unforgeable ownership token. The claimed guarantee—“two concurrent `generate` invocations cannot both proceed”—does not hold.
+
+**Required amendment:**
+
+- assign each acquisition a random fencing token/generation;
+- refresh and release only with `WHERE name = ? AND token = ?`;
+- run a background heartbeat at a fraction of the TTL (for example TTL/3), not only at stage boundaries;
+- if heartbeat/refresh loses ownership, abort before the next external call or persistent side effect;
+- make a stale holder unable to publish even after another process has reclaimed the lease.
+
+Add tests for a stage exceeding one TTL, reclamation followed by the old guard dropping, and a stale holder attempting to refresh/release/publish. An OS file lock is a simpler alternative on one host (§ Alternatives).
+
+### H2. Phase B’s interleave rule depends on Phase C utility and does not guarantee exposure
+
+Evidence:
+
+- §32 says Phase B enables new admission but keeps the existing final selector.
+- The Stage A quality/fit split and new utility do not become authoritative until Phase C.
+- Phase B nevertheless reserves a slot for the “highest-utility” union-only candidate.
+- The same sentence calls it an “issue slot” while preserving “Stage B’s right to refuse.”
+
+At Phase B there is no v3 utility score to rank this cohort unless Phase C work has silently moved earlier. If Stage B may refuse, the slot is not an issue slot and the seven-run exit window can yield zero interleaved exposures, defeating the stated purpose of collecting real labels.
+
+**Required amendment:** choose one exact Phase B behavior:
+
+- rank candidates using an available Phase B score (preliminary blend or the legacy combined Stage A score), apply an explicit minimum quality threshold, and deterministically reinsert one qualified union-only candidate after Stage B; or
+- call it an interleave nomination, let Stage B refuse, and require a minimum number of actual exposures before the phase can exit rather than “7 runs.”
+
+If v3 utility is computed in Phase B shadow solely to choose the interleave, state which Stage A response fields exist then and move the necessary implementation work out of Phase C.
+
+### H3. Migration `0002` cannot seed a hashed profile row as ordinary SQL without a bootstrap design
+
+§7.4b and §31.13 require migration `0002` to seed `taste_profile_versions` from three existing `kv` values, including:
+
+- parsing the JSON `profile_version` payload to obtain `version` and `built_at`;
+- hashing the existing profile text with SHA-256 for non-null `profile_hash`;
+- preserving learned text when present.
+
+SQLite has no built-in SHA-256 function, and the repository’s migrations are plain SQL. Even if JSON extraction is available, the hash cannot be produced by the shown migration alone. A fake/empty hash would violate the manifest identity contract.
+
+**Required amendment:** specify a Rust bootstrap immediately after schema migration:
+
+1. open one transaction;
+2. read and parse the current `kv` values;
+3. compute the canonical hash in Rust;
+4. insert the first history row idempotently;
+5. commit before any profile load/rebuild.
+
+Alternatively register an explicit SQLite hash function, but that is more machinery for a one-row migration. Test absent, malformed, and already-seeded `kv` states, not only the happy path.
+
+## Medium
+
+### M1. Adjudications are not tied to the run or algorithm that produced the sample
+
+The §7.4d primary key is `(run_date, article_id)`, while all ranking snapshots correctly use `run_id`. A date can have live, shadow, dry-run, and rerun manifests with different candidate sets/configurations. `evaluate --adjudicate --date D` therefore cannot prove which run defined “union-only” and “control,” and a later rerun can change the explanation behind an existing verdict.
+
+The text also says the table ensures an article is “never re-scored twice,” but including `run_date` permits the same overlapping article to be adjudicated again the next day.
+
+**Recommendation:** introduce an adjudication batch keyed to `run_id`, algorithm version, and a persisted deterministic sample seed. Store randomized display order separately from hidden arm, and decide whether deduplication is per run, per article globally, or after a cooldown. The CLI should print the selected run ID before collecting labels.
+
+### M2. A valid zero-candidate run has no “first ranking rows” with which to finalize its manifest
+
+§7.4 finalizes the manifest in the same transaction as the first `candidate_rankings` inserts. Empty ingest windows and all-hygiene-excluded days are valid degraded/empty outcomes, but supply no first row. Such a run remains provisional forever and is excluded from every diagnostic even though its ranking configuration and zero-candidate outcome are meaningful.
+
+**Recommendation:** finalize in one transaction that inserts zero or more initial rows; row existence must not be the trigger. Add a zero-eligible-candidate integration test.
+
+### M3. `stage_completeness_json` is authoritative but nullable and structurally unversioned
+
+Per-metric eligibility now depends on fields inside `stage_completeness_json`, yet the column is nullable, has no schema version, and no final-manifest invariant requires valid stage coverage. A malformed or old-shape JSON object can silently change metric denominators.
+
+**Recommendation:** define a versioned typed structure, require it when `manifest_status = 'final'` in application logic, and treat parse/unknown-version failures as ineligible with an explicit diagnostic. Consider normal columns for the few coverage counts most often queried; JSON is reasonable only if all filtering occurs after typed decoding rather than ad hoc SQL JSON paths.
+
+## Low
+
+### L1. `explain` still says “latest complete run”
+
+§26.3 says `explain` defaults to the “latest complete run for the date,” reintroducing the nonexistent lifecycle term fixed in §7.6. Say “latest eligible run under the typed predicate” and specify whether dry-run/shadow runs are excluded by default.
+
+### L2. The proposed `QueryFragment` return type is not part of the current SQLx design
+
+§7.6 sketches `fn evaluable_runs(kind: EvalKind) -> QueryFragment`, but this repository uses runtime `sqlx::query` and has no query-fragment abstraction. Keep the typed single-source requirement, but specify an implementable shape: a DB method that executes the full query, a `QueryBuilder<Sqlite>` helper, or a typed status/stage predicate applied after loading rows.
+
+### L3. “Failed requests count actual tokens spent” is not always observable
+
+§24 requires failed requests and retries to count actual provider tokens. For transport failures and some 5xx responses, no usage payload exists even if a provider ultimately bills work. Reserve-before-send is still correct, but reconciliation cannot always know “actual.”
+
+Document the conservative rule: retain the estimate when actual usage is unavailable, reconcile only from trustworthy usage responses, and report estimated versus provider-reported usage separately.
+
+## Nits
+
+- Acceptance criteria number `18` appears twice; the final API-key/vector criterion should be 23.
+- The R1 resolution table still points leakage tests to §31.9; they moved to §31.12.
+- §24.1’s parenthetical calls `--dry-run` read-only and then immediately says it persists data. State simply that dry runs acquire the lease.
+- §15.1 says a dedicated facet stage is triggered by “metric 6 restricted to facet-driven admissions,” but facet preference is now forbidden in admission until that stage exists. Frame the trigger as an offline counterfactual evaluation, not an existing admission metric.
+
+## Alternatives
+
+### Alternative A: Per-signal evidence gates
+
+Keep the current linear ramp but instantiate it independently for embedding, facets, and feed affinity using only compatible observations. Use global `W` solely for exploration maturity and general telemetry. Prefer this because it preserves the plan’s simple mathematics while making feature outages, opt-outs, and model migrations honest.
+
+### Alternative B: Run-local derived preference snapshots
+
+Treat `ratings` as canonical and derive kNN examples, facet statistics, and feed priors into one immutable in-memory `PreferenceState` per run, all bounded by `as_of`. Persist only the resulting raw candidate signals and evidence counts in run snapshots. Keep `feed_priors_v2` as an optional current-serving cache, never as replay input. Prefer this over versioning every aggregate table at the project’s scale.
+
+### Alternative C: Separate fidelity replay from counterfactual evaluation
+
+Use `replay` only for “what information was available then?” and add an explicit `evaluate --counterfactual-features` mode for post-hoc embeddings/facets. Prefer this when offline algorithm comparison matters more than exact historical feature provenance. The manifest must record the feature-time policy so results cannot be mixed.
+
+### Alternative D: OS-backed process lock
+
+On one Linux host, an advisory file lock held by an open file descriptor naturally releases on process death and cannot be deleted by a stale RAII guard after another process acquires it. Prefer it if generation never needs to coordinate across hosts. Keep the SQLite fenced lease only if holder identity, waiting, and future multi-host operation justify the extra heartbeat/fencing machinery.
+
+## Open Questions
+
+1. Is `replay` meant to reproduce only features available at `as_of`, or to evaluate the new algorithm using features computed later? Both are useful, but they require different cache rules and labels.
+2. Should protected/missing-feature ratings count toward exploration maturity while remaining excluded from embedding/facet evidence?
+3. Is `feed_priors_v2` an online current-state cache or an input to every run? Historical runs cannot safely mutate or trust one global snapshot.
+4. Will churn move to run-scoped `candidate_rankings`, or must `scores` gain actual observation-time provenance?
+5. Is the Phase B interleave a guaranteed exposure or merely a Stage B nomination? What score exists to choose it before Phase C?
+6. Can any lease-protected stage or backfill exceed 30 minutes? If yes, fencing and an active heartbeat are mandatory; stage-boundary refresh is insufficient.
+

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-19-3.md

@@ -0,0 +1,110 @@
+# Plan Review: Personalized Ranking, Embeddings, Facets, and Feedback (v4)
+
+**Reviewed:** 2026-08-19  
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md` (revision 4)  
+**Verdict:** **Not ready for implementation.** Revision 4 resolves the prior review's mathematical, privacy, and lease defects, but two fidelity guarantees are still built on overwrite-in-place tables and therefore cannot hold. The remaining High findings should also be made explicit before implementation because they affect schema shape, provider accounting, process coordination, and rollout behavior.
+
+## Critical findings
+
+### C1. Adding `run_id` and `scored_at` does not preserve score observation history
+
+Section 7.8 correctly identifies that nominal `run_date` is not an observation timestamp, but the proposed migration leaves the existing primary key unchanged: `scores` remains keyed by `(article_id, run_date)` (plan lines 646-661; `migrations/0001_init.sql:58-63`). The current `db::upsert_score` writes through that key (`src/db.rs:386-402`). Consequently, a recuration of the same nominal date still overwrites the earlier observation; it merely replaces it with a row containing a newer `run_id` and `scored_at`.
+
+Example:
+
+1. Article A receives score 2.0 for the August 1 run, observed August 1.
+2. On August 19, `generate --date 2026-08-01` scores A as 7.0 and overwrites the same `(A, 2026-08-01)` row.
+3. A fidelity replay as of August 5 excludes the surviving row because `scored_at = August 19`, but the valid August 1 score is gone. The churn result has changed because of a future recuration.
+
+This directly contradicts the churn observation-time test in §31.12 and the stated purpose of the v4 fix. `run_id` is provenance only if it participates in an append-only identity.
+
+**Required amendment:** create an append-only score-observation relation keyed at least by `(run_id, article_id)` (or rebuild `scores` with that primary key), then have the churn query select the latest compatible observation with `scored_at <= as_of`. If the legacy date-keyed `scores` table must remain for phased compatibility, treat it as a current-value projection and add a separate authoritative `score_observations` table. Add the destructive case above to §31.12; merely inserting one future score row is not sufficient.
+
+### C2. The broader fidelity contract still queries mutable snapshots as though they were histories
+
+The same issue exists outside `scores`:
+
+- `ratings` is keyed by `(issue_date, article_id)`, and a vote flip overwrites both `vote` and `rated_at` (`migrations/0001_init.sql:100-107`, `src/db.rs:526-543`). Section 13.1 calls the overwrite acceptable for decay (plan line 952), but after a future flip a replay before the flip loses the original vote entirely. Filtering `rated_at <= as_of` cannot recover it.
+- `issues` is keyed only by date, `upsert_issue` overwrites `generated_at`, and `replace_issue_articles` deletes and replaces the lineup for that date (`src/db.rs:441-486`). Republishing an old nominal date therefore erases the publication fact and lineup that existed at an earlier `as_of`. The §6.1 predicate `issues.generated_at <= as_of` then excludes the replacement without restoring the original.
+- Feed-prior reconstruction joins ratings to the article's current `sources_json`. That field is overwritten on re-ingest (`src/db.rs:270-279`), so future source expansion can change historical feed attribution even if the rating row itself did not change.
+
+Persisted `candidate_rankings` makes an original run's scalar output inspectable, but it does not repair `generate --as-of-date`, preference-state reconstruction, churn reconstruction, or previously-published exclusion. The acceptance criterion that replay is unaffected by future ratings or issues is therefore stronger than the schema can satisfy.
+
+**Required amendment:** choose one of these contracts before implementation:
+
+1. **Recommended:** add append-only `rating_events` and publication/issue-version tables, selecting the latest event at or before `as_of`; preserve the feed-attribution/source snapshot needed by each rating event. Together with C1, this makes fidelity a real temporal query.
+2. **Narrower alternative:** remove `generate --as-of-date` and stop promising state reconstruction. Define “fidelity” as inspection/reweighting of already-persisted run snapshots only, explicitly excluding vote state before a later flip, historical publication reconstruction, and historical source attribution.
+
+Whichever contract is chosen, §31.12 needs mutation tests: flip an existing rating in the future, republish the same issue date in the future, rescore the same article/date in the future, and verify that the earlier result is unchanged.
+
+## High findings
+
+### H1. “Daily” provider accounting still omits billing-day identity, non-generate commands, and crash persistence
+
+Sections 7.6 and 24 preload spend from `runs` “for the date” (plan lines 588-589 and 1507 onward), following the existing `spend_for_date(date)` query over `runs.date` (`src/db.rs:688-694`). That date is the nominal issue date, not the provider billing day:
+
+- recuration of August 1 on August 19 charges the August 1 bucket;
+- recurations of several historical dates on one real day each receive a fresh “daily” ceiling;
+- `features backfill` and standalone `profile rebuild` make provider calls but are not specified to create/finalize `runs` rows, so their spend has no durable bucket;
+- reservations and conservative failure estimates live only in process memory until the run is finished. A crash after a request but before `finish_run` leaves zero persisted spend; the kernel correctly releases the file lock, and a retry starts from the understated balance.
+
+Serialization prevents concurrent overspend, but it does not fix omitted or crash-lost spend. This is particularly important because the limits are described as runaway guards.
+
+**Required amendment:** account by actual request/reservation time (with a documented UTC/provider-day boundary), across every provider-using command. Persist the estimate before dispatch and reconcile it afterward, so a crash leaves the conservative reservation rather than zero. A small append-only `provider_usage`/`provider_reservations` table keyed by provider, operation, optional `run_id`, and `reserved_at` is the cleanest design. If the plan intentionally keeps only successful-generate, nominal-date accounting, rename and weaken the guarantee accordingly; it is not a daily provider ceiling.
+
+### H2. The lock scope omits the standalone profile rebuild, which spends budget and races profile versioning
+
+Section 24.1 says “Every mutating command” takes the lock, but lists only `generate`, `features backfill`, and `features prune` (plan lines 1517-1523). `profile rebuild` also calls DeepSeek and writes both `taste_profile_versions` and the `kv` current pointer. Without the same lock, it can overlap generation's weekly rebuild, duplicate spend, choose the same next version, or change the profile pointer while a run is establishing its manifest.
+
+There is also a placement mismatch: §30 says `pipeline.rs` takes the lock, but the current CLI calls `Db::open_and_migrate` before entering `pipeline::generate` (`src/main.rs:105-111`). Thus “before doing any work” does not include startup migration/bootstrap, and different commands are likely to acquire at inconsistent points.
+
+**Required amendment:** define the exact command matrix instead of calling `serve` read-only (the rating endpoint writes `ratings`, though its post-`as_of` writes can safely remain unlocked). At minimum, standalone profile rebuild must take the generation/provider lock. Specify whether the lock is acquired in `main` before command-specific DB work or whether migration/bootstrap has its own short critical section. Add a concurrent generate/profile-rebuild test, including profile-version allocation and provider reservations.
+
+### H3. A manifest becomes `final` before its required stage-completeness data exists
+
+Section 7.4 says the manifest is finalized once preference state and profile selection complete, transactionally with the initial ranking rows (plan lines 386-394). In the target pipeline this is before admission, Stage A, facets, Stage B, and publication. Yet the same section requires every final manifest to contain a complete `stage_completeness_json`, and §7.6 uses that object as authoritative per-metric eligibility.
+
+At that early point the implementation can only write placeholder zeroes and mutate a supposedly final authority later. The plan does not specify those later updates or make them atomic with `runs.status`. A crash or error between the independent writes can therefore leave an `ok`/`degraded` run with stale completeness. In addition, the proposed completeness schema contains `embeddings`, `stage_a`, `facets`, and `stage_b`, but §7.6 says admission metrics require the admission stage to have completed; there is no admission field from which to decide that.
+
+**Required amendment:** separate “ranking definition captured” from “run finalized,” or keep the manifest provisional until `finish_run`. Write final stage completeness and the final `runs.status` in one transaction; candidate rows do not need to be coupled to manifest finalization. Include at least admission, utility/diversification, selection, and publication completion states wherever metrics depend on them. Zero-candidate runs still finalize normally at end of run, so the R4 fix is preserved.
+
+### H4. Guaranteed post-Stage-B insertion and a hard maximum still lack a total precedence rule
+
+Revision 4 says protected auto-includes and the Phase B interleave pick are reinserted after Stage B, both subject to `hard_max` (plan lines 1562 and 2015-2023). It simultaneously calls the interleave a reserved slot and guaranteed exposure. If Stage B returns exactly `hard_max` articles, insertion must either exceed the hard maximum, evict a Stage B pick, or fail to expose the interleave. The same ambiguity appears when protected auto-includes and an interleave compete for the last slot, or when Stage B already selected the intended interleave naturally.
+
+Section 21.2 settles only the case where auto-includes alone exceed the ceiling; it does not define precedence among ordinary auto-includes, protected auto-includes, interleave exposure, and editor picks.
+
+**Required amendment:** specify one deterministic merge order and test it at capacity. For example: deduplicate natural Stage-B selections first; reserve/insert mandatory auto-includes (trimming among auto-includes only if they alone exceed the ceiling); insert the interleave by evicting the lowest-ranked non-mandatory editor pick; then fill remaining slots from Stage B. If auto-includes consume all capacity, explicitly decide whether the interleave is not guaranteed that day or may displace an auto-include. Count `interleave_selected` only for a real final exposure.
+
+## Medium findings
+
+### M1. The malformed-profile bootstrap repairs history but leaves the current version pointer malformed
+
+For malformed `kv[profile_version]`, §7.4b seeds `taste_profile_versions.version = 1` and does `ON CONFLICT DO NOTHING` (plan lines 427-440), but it does not repair the malformed `kv` value. The current `stored_version` treats that value as absent (`src/curate/profile/mod.rs:203-220`), so the next rebuild chooses version 1 again. An append then conflicts with the seeded row; an upsert would overwrite the very history the table is meant to preserve.
+
+**Required amendment:** either repair `kv[profile_version]` to the canonical seeded metadata in the same bootstrap transaction, or allocate the next version from `MAX(taste_profile_versions.version) + 1` and update the pointer transactionally. Extend the malformed-state test from “bootstrap succeeds” to “bootstrap followed by rebuild creates version 2 and preserves version 1.”
+
+## Alternatives worth considering
+
+### One append-only observation layer
+
+Instead of solving scores, ratings, issue publication, and provider spend with separate ad hoc exceptions, introduce a small family of append-only event/observation tables and keep the existing tables as current projections. SQLite is well suited to this volume. It gives `as_of` one consistent meaning and makes crash-safe accounting natural.
+
+### Snapshot-only evaluation
+
+If temporal event storage is judged too much scope, keep per-run `candidate_rankings` and manifests as immutable snapshots and constrain evaluation to those snapshots. This is materially simpler, but the plan must then drop claims that it can reconstruct arbitrary historical state or rerun generation faithfully after mutable inputs have changed.
+
+### Pre-reserved final-selection capacity
+
+Rather than post-selection eviction, calculate Stage B's available capacity after mandatory auto-includes and the active interleave reservation. This makes the prompt ceiling truthful and reduces surprising removal of editor choices, at the cost of giving Stage B a slightly smaller slate on those days.
+
+## Open questions
+
+1. Must fidelity remain stable after an existing vote is flipped and after the same nominal issue date is republished? If yes, append-only rating and publication history is mandatory.
+2. Does “daily provider ceiling” mean the provider's real UTC billing day across generation, backfill, profile rebuild, retries, and crashes? If not, what narrower operational guarantee is intended?
+3. When `hard_max` is full, which has priority: protected auto-includes, ordinary auto-includes, the Phase B interleave exposure, or Stage B's lowest-ranked choice?
+4. Is `manifest_status = final` intended to mean “ranking inputs fixed” or “run outcome complete”? The current plan requires it to mean both at different times.
+
+## Bottom line
+
+The ranking, evidence-gating, privacy wrapper, and single-host lock choices are now implementation-worthy. Implementation should still wait for C1 and C2 because they determine whether migration `0002` needs append-only temporal tables. H1-H4 should be resolved in the same amendment so provider limits, manifest eligibility, locking, and rollout exposure have testable semantics rather than being decided piecemeal during coding.

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-19-4.md

@@ -0,0 +1,172 @@
+# Plan Review: Personalized Ranking, Embeddings, Facets, and Feedback (v5)
+
+**Reviewed:** 2026-08-19  
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md` (revision 5)  
+**Verdict:** **Nearly ready, but not yet safe to implement verbatim.** Revision 5 fixes the prior temporal-history, billing-day, lock-scope, manifest-lifecycle, capacity, and profile-bootstrap blockers. No new architectural rewrite is needed. Three High-severity execution defects remain: the proposed churn query does not implement its stated “latest observation wins” rule, the plan sometimes bypasses its new event authority in live/recurate modes, and the provider ledger cannot enforce the promised shadow budget or conservatively account for retries. Several stale v4 instructions should also be removed so an implementation agent is not given two incompatible authorities.
+
+## Critical
+
+No remaining Critical finding. The append-only observation layer, UTC provider ledger, three-state manifest, expanded lock matrix, and pre-reserved Stage B capacity are sound architectural responses to the previous review.
+
+## High
+
+### H1. The shown churn SQL neither selects the latest observation nor uses observation time for the lookback window
+
+Section 7.8 says “the latest observation per article wins,” but the displayed query returns every qualifying low-score row:
+
+```sql
+SELECT cr.article_id
+FROM candidate_rankings cr
+JOIN runs r ON r.id = cr.run_id
+WHERE cr.llm_quality_score IS NOT NULL
+  AND r.started_at <= :as_of
+  AND cr.run_date >= :since
+  AND cr.llm_quality_score < :floor
+  AND r.status IN ('ok', 'degraded')
+```
+
+There is no grouping, window function, correlated `NOT EXISTS`, or maximum observation selection. If an article scored 2.0 and was later rescored 7.0, the old 2.0 row still satisfies this query and suppresses the article. The new mutation test can therefore pass only if the implementation diverges from the SQL the plan tells it to write.
+
+The lookback also remains anchored to `cr.run_date`, the nominal issue date, even though §7.8 calls `runs.started_at` the “true observation time.” A low score observed today while recurating an old nominal date is immediately outside a seven-day nominal-date window; conversely, a future nominal issue date could enter the window despite when it was observed. Pruning ranking snapshots by nominal date would reproduce the same defect.
+
+**Required amendment:** define whether churn recency means observation recency or nominal issue recency. The v5 rationale strongly implies observation recency, so rank eligible observations with something equivalent to:
+
+```sql
+WITH ranked AS (
+  SELECT cr.article_id,
+         cr.llm_quality_score,
+         ROW_NUMBER() OVER (
+           PARTITION BY cr.article_id
+           ORDER BY r.started_at DESC, cr.run_id DESC
+         ) AS rn
+  FROM candidate_rankings cr
+  JOIN runs r ON r.id = cr.run_id
+  WHERE cr.llm_quality_score IS NOT NULL
+    AND r.started_at >= :observation_since
+    AND r.started_at <= :as_of
+    AND r.status IN ('ok', 'degraded')
+)
+SELECT article_id
+FROM ranked
+WHERE rn = 1 AND llm_quality_score < :floor;
+```
+
+Prune `candidate_rankings` by `runs.started_at`, not `candidate_rankings.run_date`. Add both directions to the regression suite: low→high must not suppress; high→low must suppress. Include a historical-date recuration observed today so the time-axis choice is tested rather than inferred.
+
+### H2. `rating_events` and `publication_events` are declared authoritative, then bypassed in live/recurate modes
+
+Section 6.1 correctly says temporal reads always use the event tables. Section 7.9 then says “`live` and `recurate` may read the projections directly.” That is not equivalent to asking the event layer for the latest state now:
+
+- `ratings` is keyed by `(issue_date, article_id)`. If the same article is republished and rated in two issues, reading the projection can count it twice, while “latest event per article” counts it once.
+- `issue_articles` contains only the current lineup for each nominal date. If a republish removes an article, a live projection read says it was never published, while `publication_events` correctly says it was published earlier. The article can then re-enter the paper despite the hard “already published” exclusion.
+- Maintaining separate projection and event queries gives live and fidelity subtly different product semantics, not merely different time bounds.
+
+There is also an incomplete attribution snapshot. `rating_events.source_feeds_json` stores only distinct **direct** feed IDs, while §7.7 explicitly falls back to the `best_entry_id` feed when no direct feed exists. The event does not store that fallback feed, so a discovery-only rating cannot be reproduced without consulting mutable current article state—the exact dependency the event row was introduced to remove.
+
+**Required amendment:** use the event tables for preference and previously-published reads in **all** modes; live/recurate simply pass `as_of = now`. Keep projections only for serving the current issue/current vote UI. Store the exact local attribution result on each rating event—preferably a versioned `feed_credits_json` map of feed ID to weight, or at minimum the post-fallback attributed feed set—not merely the pre-fallback direct set. Specify `ORDER BY event_at DESC, id DESC` so equal timestamps are deterministic.
+
+Add regressions for:
+
+1. an article removed by same-date republication remains “previously published” in a subsequent live run;
+2. an article rated through two issue dates contributes only its latest vote once;
+3. a discovery-only rating retains its vote-time fallback feed after current article provenance changes.
+
+### H3. The provider ledger lacks the dimensions and attempt semantics required by §24
+
+The append-only `provider_usage` ledger fixes UTC bucketing, cross-command spend, and crash persistence, but two promised controls are not representable yet.
+
+First, §24 says shadow work draws from `shadow_max_daily_usd` and “can never consume the production slice.” `provider_usage` has provider, operation, and optional `run_id`, but no `budget_class`/`slice`. Production and shadow calls can occur within the same run, so joining through `run_manifests.shadow` cannot classify individual requests. The ledger can enforce one provider-wide ceiling or a shadow ceiling, but not both the stated production and shadow slices.
+
+Second, retries are under-specified. The plan reserves before “the request,” retries network/429/5xx failures, and keeps an estimate when a failed attempt has no usage payload. If one ledger row covers a logical request, a failed possibly-billed attempt followed by a successful retry will usually settle that row to the final attempt's actual usage, erasing the failed attempt's conservative estimate. This violates the rationale in §24 precisely on the retry path most likely to lack usage metadata.
+
+**Required amendment:**
+
+- Add a `budget_class` such as `production | shadow | backfill` (or an equivalent explicit allocation key) and define whether the provider-wide ceiling also caps the sum of all classes. State whether `shadow_max_daily_usd` is inside the production provider maximum or additive to it.
+- Reserve and reconcile **per outbound HTTP attempt**, linking attempts with a logical request ID, or reserve the worst-case cost of all allowed attempts and decrement safely as attempts become known not to have been billed. The per-attempt model is easier to audit.
+- Define the conservative estimate formula. For DeepSeek it should include input plus the request's maximum possible output tokens at their respective prices, assuming no cache discount unless known; Voyage is input-only.
+- Add a test where attempt 1 returns a 5xx with no usage and attempt 2 succeeds: the day total must contain attempt 1's standing estimate plus attempt 2's actual usage. Add a mixed production/shadow run proving each slice and the aggregate ceiling.
+
+## Medium
+
+### M1. The new authorities were not propagated through the normative file-by-file instructions
+
+Several later sections still instruct an implementer to build the v4 design:
+
+- §9.4 says Voyage is “preloaded per date” and to keep DeepSeek budget semantics unchanged.
+- §23 says feed priors and flips derive from canonical `ratings`; §25.1 says protected ratings derive from `ratings` and current `sources_json`.
+- §24 says meters preload from `runs` by date, despite §7.6 making `provider_usage` authoritative, and says “rather than build a cross-process reservation ledger” immediately after adding one.
+- §30's `src/db.rs` list still requests `voyage_spend_for_date` and “finalize [the manifest] transactionally with the first ranking rows.” Its `src/pipeline.rs` entry likewise says to “finalize” after preference loading instead of transition to `ranking_fixed`.
+- The `PreferenceState` type comment and `preference.rs` entry still call `ratings` canonical.
+- `RunReport`'s listed completeness block still omits admission, utility, diversification, selection, and publication.
+- §31.8 says every mid-run failure leaves a provisional manifest, although a failure after the new transition must leave at least `ranking_fixed` (or deliberately finalize a failed outcome).
+- Phase A's adjudication paragraph describes the superseded date-keyed table rather than the `adjudication_batches` schema.
+
+These are not cosmetic in an “implementation-grade” plan: §30 is exactly where an implementation agent will derive its worklist.
+
+**Required amendment:** run one terminology/authority pass and replace every stale use with `rating_events`, `publication_events`, `provider_usage`, `ranking_fixed`, and the full completeness schema. Reserve “projection” for explicitly non-temporal UI/compatibility paths. Update the R1/R2/R3 resolution tables where they still describe superseded mechanisms, or label those cells as historical resolutions superseded by R5.
+
+### M2. Observation seeding is coupled to profile bootstrap and does not have an explicit one-time marker
+
+Section 7.4b makes `bootstrap_profile_history()` also seed rating and publication events. Its earlier step says that an absent/empty taste profile does nothing, making it unclear whether event seeding still runs on a database that has issue/rating projections but no profile. These are unrelated migrations and should not share an early-return condition.
+
+“Seed if the event table is empty” is also a state heuristic, not a migration marker. Every command runs the bootstrap after migration, while an already-running `serve` process does not hold the file lock and can append rating events concurrently. At cutover, an emptiness check plus projection copy can race a legitimate first event or make retry behavior ambiguous.
+
+**Required amendment:** split this into `bootstrap_profile_history()` and `bootstrap_observation_history()`. Record completion in a durable bootstrap/migration marker inside the same transaction as seeding, and make the copy idempotent by a stable seed identity. Event seeding must run independently of whether a taste profile exists. Test a projection-only database with no profile, and test restart after a partially completed/rolled-back seed.
+
+### M3. Failure-state manifest semantics remain contradictory
+
+Section 7.4 defines `final` as the state written in the same transaction as `finish_run`, and the current pipeline calls `finish_run` on errors. The eligibility table even permits `explain --run-id` over failed runs with `ranking_fixed` **or final**. But §31.8 requires every run that fails mid-way to retain a provisional manifest.
+
+That cannot hold for failures after the run reached `ranking_fixed`, and it discards useful completeness data if all failed runs are deliberately kept provisional.
+
+**Required amendment:** specify transitions by failure point. A coherent rule would be: failure before preference/profile capture remains `provisional`; failure after it remains `ranking_fixed` or transitions to `final` with terminal completeness in the same status transaction; evaluation always excludes `failed`, while `explain --run-id` accepts all three states with whatever data exists. Add one test for failure before and one after `ranking_fixed`.
+
+### M4. Publication-event authority stops just short of the actual publish boundary
+
+The plan appends `publication_events` transactionally with `replace_issue_articles`, which is necessary, but current execution publishes files first and records the issue afterward (`src/pipeline.rs:513-528`, `557-582`). A crash or SQLite error after the atomic file copy but before the database transaction leaves an issue visible through the publish directory/OPDS without a publication event. Also, current `upsert_issue` and `replace_issue_articles` are separate database transactions; adding events only to the latter can leave the projections half-updated.
+
+**Required amendment:** at minimum, put `issues`, `issue_articles`, and `publication_events` in one database transaction after file publication and define a recovery check for “files published, DB commit missing” on rerun/startup. If strict exposure-time fidelity is not required for that crash window, state that `publication_events` means “successfully published and recorded,” not every instant a file may have been externally visible.
+
+## Low
+
+### L1. The migration-lock release/reacquire introduces an avoidable race
+
+Section 24.1 has mutating commands acquire the file lock for migration, release it, then immediately reacquire it for the command. Another command can win the gap, causing a generation that completed startup successfully to fail before doing work. This is safe but surprising.
+
+For a lock-holding command, retain the same file descriptor after migration/bootstrap and upgrade the guard's diagnostic state once the database is available. Only non-lock-holding commands such as `serve` need to release after the migration section.
+
+### L2. The deferred-options table still says a provider ledger is deferred
+
+Section 35 lists “Fenced SQLite lease or a cross-process provider ledger instead of the file lock” as deferred, but v5 now includes a cross-process provider ledger. The actual deferred choice is a distributed generation lock/fenced multi-host coordinator; rename the row so it does not suggest removing or postponing `provider_usage`.
+
+## Nits
+
+- `source_feeds_json` should have a versioned typed schema and validation just like the other authoritative JSON fields; storing the final credit map makes this natural.
+- Add `CHECK (estimated_usd >= 0)` and `CHECK (actual_usd IS NULL OR actual_usd >= 0)` to `provider_usage`, and validate that `billing_day` is derived internally from `reserved_at` rather than independently supplied by callers.
+- The Stage B merge rule says editor picks are admitted in model order and excess picks are trimmed by `ordering_score`; choose one ordering rule for malformed over-cap responses.
+- §23 says `serve` “mutates nothing that a run reads mid-flight.” It does mutate the event authority; the correct statement is that timestamp-bounded reads make concurrent later events invisible to the run.
+
+## Alternatives
+
+### Always read the observation layer (recommended)
+
+Use `rating_events` and `publication_events` for every ranking/history query, with `as_of = now` for live/recurate. This produces one tested semantic path. The projection tables remain valuable for the current issue page and current vote state, but never decide ranking history.
+
+### Maintain query-equivalent projections
+
+If event scans ever become measurably expensive, introduce purpose-built current projections such as one row per article's latest rating and a durable “ever published” set, updated transactionally from events. Those projections must be defined and tested as exact query-equivalent caches; the existing `ratings` and `issue_articles` shapes are not equivalent.
+
+### Retry budget alternatives
+
+Per-attempt ledger rows are the most auditable solution. Reserving the maximum cost of every possible retry up front is simpler but needlessly blocks budget during transient failures and requires careful release semantics; use it only if per-attempt IDs are judged too invasive.
+
+## Open Questions
+
+1. Does “recent rejection” mean recently **observed** by an LLM or associated with a recent nominal issue date? The v5 prose and event-time rationale imply the former, but the SQL implements the latter.
+2. Is `shadow_max_daily_usd` a sub-limit within each provider's overall daily ceiling, or an additive allowance beyond it? Which budget class owns feature collection performed during a shadow run but reusable by production?
+3. Should a failed run after `ranking_fixed` be finalized with partial completeness, or remain `ranking_fixed`? Either is workable; “always provisional” is not.
+4. Does a publication event represent external file visibility or the later successful database commit? What recovery behavior is expected if those diverge?
+
+## Bottom line
+
+The plan's architecture is now fundamentally sound. Resolve H1-H3 before implementation because they affect the correctness of the churn rule and the schemas for `rating_events` and `provider_usage`. The Medium findings should be amended in the same pass; most are consistency work, but leaving them in an implementation-grade document would cause the code to regress toward mechanisms revision 5 explicitly replaced.

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-19-5.md

@@ -0,0 +1,118 @@
+# Plan Review: Personalized Ranking, Embeddings, Facets, and Feedback (v6)
+
+**Reviewed:** 2026-08-19  
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md` (revision 6)  
+**Verdict:** **Close, but not yet safe to implement verbatim.** Revision 6 correctly repairs the churn query, makes the event layer authoritative in every mode, snapshots final feed attribution, adds attempt-level budget accounting, and closes the migration/publication consistency gaps from the previous review. No architectural rewrite remains. Four High-severity contracts still need to be made implementable: protected-feed classification is not actually based on feed identity, the shadow sub-limit does not reserve production capacity, concurrent ledger reservations have no specified SQLite atomicity mechanism, and the legacy churn fallback cannot implement the new observation-time/latest-value semantics. A final normative consistency pass is also still required.
+
+## Critical
+
+No remaining Critical finding. The ranking, observation-history, manifest-lifecycle, and publication designs are now internally sound in their primary paths.
+
+## High
+
+### H1. The provider-policy type prevents bypasses only after classification; its host matcher can misclassify a private feed
+
+Section 25.1 defines `no_external_ai_feeds` as feed IDs or host substrings “matched exactly like `always_include_feeds`” and then gives `externally_processable` only an `Article` and `CurationConfig` (§25.1, §30). That existing matcher does not inspect the feed URL. In `src/curate/prefilter.rs:136-165`, numeric values match any source feed ID, but string values are searched as case-insensitive substrings of `article.url` and `article.canonical_url`. `SourceRef` contains `feed_id`, title, category, and kind, but no feed URL (`src/types.rs:62-70`).
+
+That is insufficient for the stated threat model. A private Miniflux feed may live at `reader.internal/private.xml` while its entries link to public sites. Configuring `reader.internal` appears valid but does not protect those articles; `externally_processable` constructs the wrapper and the strong type then faithfully sends the protected data. Substring matching also has the wrong security properties for a deny policy: it can match unrelated hosts and does not define exact-host/subdomain behavior.
+
+This is especially easy to miss in a deduplicated cluster whose best article URL is public but one secondary `SourceRef.feed_id` is protected. Numeric IDs can classify that case correctly, but the advertised host form cannot.
+
+**Required amendment:** make the privacy configuration identity-safe before relying on the type-level guarantee. The safest V1 is a typed `no_external_ai_feed_ids: Vec<FeedId>` and a startup error for nonnumeric entries; Miniflux feed IDs are already present on every source and survive deduplication. If host policies must remain, separate them into explicitly named fields and pass actual Miniflux feed URL/site metadata into the policy gate. Parse URLs and compare normalized hosts (with a documented exact-host/subdomain rule), never arbitrary substrings of article URLs.
+
+Add tests where:
+
+1. a protected feed URL host differs from the linked article host;
+2. the best source is public but a secondary source feed is protected;
+3. a lookalike hostname does not accidentally match an unrelated protected hostname.
+
+The full-run recording mock should use one of these adversarial classifications rather than only a numeric best-feed match.
+
+### H2. A shadow sub-limit inside a shared ceiling does not preserve a production slice
+
+Section 7.6 says all classes share the provider's `max_daily_usd`, while `shadow_max_daily_usd` merely places an additional cap on shadow (§7.6 lines 672-687). Section 24 then claims shadow “can never consume the production slice” (§24 line 1734). Those statements are not equivalent.
+
+With the documented Voyage defaults, the provider ceiling is $0.25 and the shadow sub-limit is $0.20. A shadow invocation that spends $0.20 first leaves only $0.05 for a later production run. Production-first ordering protects calls only within one invocation; it does nothing across runs or across the UTC day. The same issue arises if a standalone non-publication command runs before the timer. The ledger accurately records the depletion, but it does not reserve capacity for the newspaper.
+
+**Required amendment:** choose and state one of these contracts:
+
+- If publication capacity is guaranteed, add a per-provider `production_reserve_daily_usd` (or explicit class allocations) and admit shadow/backfill only when `provider_total + estimate <= provider_max - production_reserve`. Define which operations may consume the reserve and what happens after the publication run completes.
+- If the ceiling is only an account-wide runaway guard, remove “production slice” and state plainly that shadow is bounded but may reduce capacity available to later production.
+
+The Phase A promise that shadowing does not affect the paper favors the first option. Add an order-sensitive test: spend shadow first, then prove the configured production reserve is still dispatchable. A mixed-class test inside one run, which §31.11b currently requires, does not cover this defect.
+
+### H3. “Atomic check-and-reserve” needs an explicit SQLite serialization mechanism
+
+The plan correctly requires checking and reserving before each concurrently spawned request, but it never defines the transaction primitive that makes the read-sum-insert sequence atomic. `buffer_unordered` can run sibling reservations concurrently inside one process. The process-wide `flock` serializes commands, not async tasks or SQLite connections within that command.
+
+A naive SQLx transaction is deferred in SQLite: two tasks can both read the same pre-reservation total, both decide they fit, and then contend when writing. Depending on timing, that either admits estimates beyond the ceiling or produces `SQLITE_BUSY` at a point the plan currently treats like ordinary provider degradation. A unique key on `(request_id, attempt)` does not serialize different requests, and the spend index does not enforce a sum constraint.
+
+**Required amendment:** specify one implementation-grade reservation path. Viable choices are:
+
+- a provider-scoped in-process async mutex around a short `BEGIN IMMEDIATE` transaction that re-sums and inserts before commit; or
+- a single reservation actor/connection that serializes all check-and-insert operations.
+
+Because every provider-spending command holds the OS lock, an in-process mutex plus `BEGIN IMMEDIATE` is sufficient for the declared single-host deployment. Specify busy-timeout/retry behavior for this short transaction and keep external HTTP work outside it.
+
+Add a barrier-based concurrency test that releases many reservation tasks simultaneously near the ceiling and asserts that the sum of admitted estimates never exceeds either the provider cap or the applicable class cap. Also assert that refusal occurs before the corresponding mock HTTP dispatch.
+
+### H4. The legacy `scores` fallback cannot satisfy the churn rule unless its precedence and lifetime are bounded
+
+Section 7.8 now gives `candidate_rankings` a correct latest-observation query over `runs.started_at`, but then says pre-`0002` dates fall back to `scores` in live/recurate (line 795). `scores` is exactly the mutable, nominal-date-keyed projection the section rejected: it has no observation timestamp and may contain multiple rows for an article across nominal dates. The plan gives no query, merge precedence, or retirement point for that fallback.
+
+A straightforward union of low IDs recreates the v5 defect: one legacy low row can suppress an article despite a newer high `candidate_rankings` observation. Using `scores.run_date` as recency recreates the wrong-time-axis defect. Because new runs continue writing the compatibility projection, “pre-migration score” also cannot be inferred merely from the row's existence.
+
+**Required amendment:** either delete the fallback—the database is effectively at cold start, so this is the cleanest option—or define it as a strictly temporary compatibility bridge:
+
+- capture a migration timestamp/marker;
+- consider legacy `scores` only for articles with no `candidate_rankings` observation at all;
+- document the nominal-date approximation explicitly;
+- disable the fallback after one `recent_rejection_lookback_days` interval from migration, so an unverifiable projection cannot suppress forever.
+
+Do not let a projection row compete with an observed candidate row. Add tests for legacy-low → new-high, legacy-high → new-low, two legacy nominal dates for one article, and fallback expiry. Fidelity should continue to exclude these unverifiable rows.
+
+## Medium
+
+### M1. The normative worklist still contains v5 authorities and one lifecycle contradiction
+
+Revision 6's core sections are clear, but the later implementation instructions still tell an agent to build several superseded forms:
+
+- §23 line 1720 and §30 `src/server.rs` line 2065 say to capture distinct direct-feed IDs. The authoritative event schema requires the completed, versioned `feed_credits_json` after fallback.
+- §25.1 line 1810 says protected feed affinity and `W_global` derive from `ratings` and `sources_json`. They must derive from latest `rating_events` and stored feed credits; using current sources reintroduces the temporal bug §7.9 removes.
+- §30 `src/db.rs` lines 2007-2008 says to load/derive from ratings joined to current sources/facets. The normative source is latest rating events, with article/facet joins only for compatible local signals.
+- §30 `src/db.rs` lists `bootstrap_profile_history()` but omits `bootstrap_observation_history()`.
+- §30 `src/publish.rs` line 2069 says publication events commit with `replace_issue_articles`; the actual contract is one transaction containing `upsert_issue`, `replace_issue_articles`, and events.
+- §30 `src/main.rs` line 2079 still releases and reacquires the lock and runs only the profile bootstrap, contradicting §24.1's same-file-descriptor rule and the two-bootstrap contract.
+- The eligibility table says `explain` accepts only `ranking_fixed`/`final`, while the failure semantics and §31.8 say `explain --run-id` accepts `provisional` too.
+
+These are execution instructions, not historical review tables, and several directly reintroduce bugs v6 says are fixed.
+
+**Required amendment:** update §23, §25.1, §30, and the eligibility table so each has one authority. Add `bootstrap_observation_history` to both the startup sequence and migration concurrency test. Search the normative text for `distinct direct-feed`, `ratings and sources_json`, `load ratings`, `transactionally with replace_issue_articles`, and `release → ... re-acquire`; none should remain except in explicitly labelled historical discussion.
+
+## Low
+
+No separate Low-severity finding. The remaining cleanup belongs in the normative consistency pass above.
+
+## Alternatives
+
+### Prefer typed feed IDs for provider opt-out
+
+Feed IDs are the least ambiguous privacy boundary in this codebase: every clustered source carries one, and the Miniflux account owns the mapping. A separate article-domain rule can be added later for public-domain policy, but it should not masquerade as feed identity.
+
+### Retire legacy churn state at migration
+
+Given the near-empty production history, accepting at most one week without legacy churn suppression is lower risk than maintaining a second, semantically weaker query. If temporary continuity is still desired, snapshot the legacy low set once with an explicit expiry instead of continuing to consult the mutable `scores` projection.
+
+### Serialize ledger admission, parallelize HTTP only
+
+Reservation transactions are tiny. Serialize those locally, commit, then allow HTTP attempts to proceed concurrently. This preserves throughput where it matters while making the monetary invariant easy to state and test.
+
+## Open Questions
+
+1. Is `no_external_ai_feeds` intended to identify Miniflux feed subscriptions, article domains, or both? If both, what are the exact matching semantics for each namespace?
+2. Must Phase A/shadow work be incapable of reducing a later production run's available budget, or is `shadow_max_daily_usd` only an additional runaway cap?
+3. Is preserving pre-migration churn suppression worth a temporary second semantic path, given the production store has approximately one issue of history?
+
+## Bottom line
+
+The main ranking and temporal architecture is ready. Amend H1-H4 before implementation because each concerns a guarantee the current schema/API description cannot actually enforce: privacy classification, production budget availability, atomic budget admission, and latest-observation churn semantics. M1 should be fixed in the same edit so §30 becomes a reliable implementation checklist rather than a source of regressions.

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-19-6.md

@@ -0,0 +1,127 @@
+# Plan Review: Personalized Ranking, Embeddings, Facets, and Feedback (v7)
+
+**Reviewed:** 2026-08-19  
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md` (revision 7)  
+**Verdict:** **The ranking architecture is ready, but the plan is not yet safe to execute verbatim.** Revision 7 resolves all four v6 High findings and the prior normative inconsistencies. Three narrower High-severity guarantees remain: provider protection is not durable across article re-ingestion, non-publication work can still consume the new production reserve by being labelled `production`, and the claimed conservative reservation uses an estimator that is not an upper bound. Two migration/test consistency issues should be corrected in the same pass. These amendments are localized; no redesign of the ranking pipeline is needed.
+
+## Critical
+
+No Critical finding.
+
+## High
+
+### H1. Protected-feed classification is correct for one ingest cluster but is not durable across re-ingestion
+
+Section 25.1 now correctly checks every `SourceRef.feed_id` in the current deduplicated cluster. The plan also explicitly recognizes elsewhere that `db::upsert_article` overwrites `sources_json` on every re-ingest (§7.9, §27.2). The current implementation does exactly that in `src/db.rs:266-279`.
+
+Those facts leave a temporal privacy hole. Consider this sequence:
+
+1. Day 1 ingests canonical article A from protected feed 42. A is correctly withheld from providers but is not selected.
+2. Day 2's overlapping window sees the same canonical URL only through a public mirror/feed. `upsert_article` replaces A's `sources_json`; feed 42 disappears.
+3. The provider-policy constructor sees only the new public `Article.sources`, constructs `ExternallyProcessable`, and sends A.
+
+The type-level gate is again faithfully enforcing an incomplete classification. The same issue affects a later profile rebuild, which reloads current article state. A `rating_events.feed_credits_json` row may preserve old provenance for rated articles, but it does not protect unrated articles and the wrapper does not consult it anyway.
+
+This is not solved by merging historical sources into `articles.sources_json`: §7.7 deliberately uses current provenance for current candidate feed affinity. Privacy provenance and ranking provenance answer different questions.
+
+**Required amendment:** persist durable source membership separately from the current cluster. For example:
+
+```sql
+CREATE TABLE article_feed_observations (
+    article_id  INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
+    feed_id     INTEGER NOT NULL,
+    first_seen  TEXT NOT NULL,
+    last_seen   TEXT NOT NULL,
+    PRIMARY KEY (article_id, feed_id)
+);
+```
+
+Upsert every observed source during article persistence without deleting older observations. Build `ProviderPolicy` from the intersection of this table and the current configured protected IDs, then let the wrapper constructor consult the resulting protected-article set. Removing a feed ID from configuration can deliberately unprotect its articles; merely failing to observe that feed on a later day cannot.
+
+Bootstrap the table from current `sources_json` at migration, state that older already-overwritten provenance is unrecoverable, and require the operator to verify the configured private subscriptions before rollout. Add a two-run regression: ingest A through a protected source, re-ingest the same canonical URL through public sources only, then prove every provider path still rejects A.
+
+### H2. The production reserve is bypassed by classifying reusable shadow work as `production`
+
+The reserve formulas in §7.6 are now correct, but their protection is only as strong as `budget_class`. Immediately after defining the reserve, the plan says embeddings produced during a shadow run are classed `production` because the cache is reusable (line 695). `profile rebuild` is also unconditionally listed as production (line 674), and dry-run classification is not specified.
+
+That reopens the exact cross-invocation failure the reserve was introduced to close:
+
+- Phase A is explicitly a shadow feature-collection phase and primarily performs embeddings.
+- Those embedding attempts are labelled `production`, so they may consume the entire provider ceiling, including the reserve.
+- A later publication run can then be refused despite the claim that Phase A cannot affect the paper.
+
+Cache reuse may make a later run cheaper, but it does not make an evaluation request publication-critical. A shadow run over a different date, a broad feature collection, or an interrupted partial cache fill can spend the reserve without producing the exact artifacts the 05:30 run needs. Likewise, a standalone profile rebuild or dry run can consume DeepSeek's reserve before Stage A/B even though neither invocation publishes an issue.
+
+**Required amendment:** classify by the purpose of the HTTP attempt, not by whether its output might someday be reusable. Prefer naming the privileged class `publication` to make the invariant explicit:
+
+- provider calls required by an active issue-producing `generate` invocation: `publication`;
+- calls made only for a shadow/dry run: `shadow`, including embeddings;
+- standalone profile rebuild and feature backfill: `maintenance`/`backfill`, unless the profile rebuild is an in-run prerequisite for the issue currently being produced.
+
+Only the issue-producing class may use `production_reserve_daily_usd`. Thread an explicit execution/budget context into provider orchestration; do not infer it from operation name or cacheability. Add order-sensitive tests for shadow embeddings, a dry run, and standalone profile rebuild before a live generation. Each must leave the reserve dispatchable.
+
+### H3. The “conservative” reservation is based on a token estimate that can underestimate actual input
+
+Section 7.6 calls the reservation conservative but says input tokens use the existing character approximation (lines 661-670). That helper is `text.len().div_ceil(4)` in `src/curate/mod.rs:159-163`, documented as a crude English-prose average. It is not an upper bound. Punctuation-heavy text, code, unusual Unicode, and provider tokenization can all use materially more than one token per four bytes/characters.
+
+`max_output_tokens` safely bounds the output side, but an underestimated input reservation can be admitted just below the ceiling and then settle to an `actual_usd` above the estimate. At that point the provider call already happened and the day's ledger exceeds a ceiling acceptance criterion 22 says is enforced. The atomic transaction prevents races; it cannot repair an underestimated reservation.
+
+**Required amendment:** either make reservation amounts genuine upper bounds or weaken the contract to a best-effort threshold with a stated maximum overshoot. For the strict contract currently promised, reserve input at a tokenizer-independent upper bound over the exact assembled payload (for example, one token per UTF-8 byte, if verified safe for both providers), plus maximum output at output price. Settle downward only from trustworthy usage. A provider-specific tokenizer is also acceptable if it is available locally and versioned with the model, but an English average plus a safety factor is still not a proof.
+
+Add tests using adversarial prompt text and a mock response whose actual input usage exceeds `approx_tokens`. The admitted reservation must already cover that usage; settlement must never turn an under-ceiling admitted total into an over-ceiling total. If a strict upper bound is operationally too conservative, change the wording and acceptance criterion rather than claiming a hard ceiling.
+
+## Medium
+
+### M1. The observation bootstrap assumes an already-running `serve` process is the new dual-writing binary
+
+Section 7.4b says a concurrent live vote is harmless because it appends an event newer than the seeded rows (line 469). That is true only after `serve` has been upgraded. During the `0002` rollout, an already-running old binary writes only the `ratings` projection. A new `generate` or migration command can acquire the new file lock, seed the event table, and commit; an old `serve` process can then accept a vote into `ratings` without appending `rating_events`. The durable bootstrap marker prevents any later repair, so the new event authority permanently misses that vote.
+
+The generation lock cannot solve this because `serve` intentionally does not take it.
+
+**Required amendment:** add an explicit cutover protocol: stop/drain the existing serve unit, install/start the new binary so migration and both bootstraps complete, then reopen the rating endpoint. Alternatively ship a compatibility release that dual-writes after the new tables exist before making events authoritative, but that is unnecessary complexity for this single-host service. Document the brief downtime and test the migration from a projection-only database; do not claim an old concurrent writer is safe.
+
+### M2. Two migration tests still require the deleted legacy churn fallback
+
+Section 7.8 and §18.4 correctly say pre-`0002` `scores` rows never suppress in any mode. Two later requirements say the opposite:
+
+- §31.12: “Legacy pre-`0002` rows ... are excluded from fidelity and used in `recurate`.”
+- §31.13: confirm “a v1 `scores` row is still usable by the churn rule.”
+
+An implementation cannot satisfy those tests and the no-fallback contract simultaneously.
+
+**Required amendment:** change both tests to assert that the v1 row remains readable as a compatibility projection but is never consulted by churn. The migration test should prove the old row survives schema migration, while a separate churn test proves it does not suppress in live, recurate, or fidelity modes.
+
+## Low
+
+### L1. The normative `VoyageConfig` shape omits the new reserve field
+
+Section 9.1's Rust struct lists `max_daily_usd` and `embedding_retention_days` but not `production_reserve_daily_usd`, while §38 places that field under `[voyage]`. Add it to the struct and §30's file-by-file list. Also state where the shared `shadow_max_daily_usd` lives if it intentionally applies identically to both providers.
+
+## Nits
+
+- Section 24.0 is a subsection placed after section 24; `24.1`/`24.2` would read more naturally, though this has no implementation impact.
+- Historical resolution tables still use `no_external_ai_feeds` in some older-review rows. They are clearly historical, so this is harmless, but adding “superseded by R7-H1” would reduce search noise.
+
+## Alternatives
+
+### Sticky Boolean restriction
+
+Instead of a general source-observation table, add an `external_ai_protected` bit to `articles` and set it monotonically when any protected source is observed. This is smaller, but changing the configured feed list cannot automatically unprotect previously marked articles and loses the audit trail explaining why the bit was set. It is acceptable only if unprotection is intentionally a manual operation.
+
+### Purpose-based budget classes (recommended)
+
+Rename `production` to `publication` and pass a typed `BudgetContext` from the top-level command. This makes misuse difficult: cache code cannot promote itself merely because its output is reusable. The alternative is per-operation allowlists, which are easier to forget when adding a provider call.
+
+### Best-effort budget threshold
+
+If a byte-level upper bound rejects too much useful work, retain the existing estimator but explicitly define the limit as an advisory threshold, reserve with a documented safety factor, trip immediately when settlement exceeds it, and report maximum observed estimation error. This is operationally reasonable, but it is a different guarantee from “ceilings are enforced before dispatch.”
+
+## Open Questions
+
+1. Does an article remain protected after it was ever observed through a protected feed, until the operator removes that feed ID from configuration? The plan's “no field ever leaves” wording implies yes.
+2. Which exact invocations may consume the production reserve: only issue-producing generation, or also dry runs, standalone profile rebuilds, and shadow cache warming?
+3. Is the daily budget intended as a strict pre-dispatch ceiling or a best-effort runaway threshold? The estimator and acceptance criterion currently answer differently.
+
+## Bottom line
+
+Revision 7 resolves the prior review and leaves the core recommendation design in good shape. Amend H1-H3 before implementation because they affect the two strongest operational guarantees: private content never leaves the host, and shadow/maintenance work cannot exhaust publication capacity. M1 is a deployment-order requirement, while M2 and L1 are quick consistency fixes. After those changes, the plan should be ready to implement.

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-19.md

@@ -0,0 +1,211 @@
+# Re-review — Personalized Ranking, Embeddings, Facets, and Feedback (v2)
+
+**Plan:** `docs/plans/2026-08-17-personalized-ranking-and-facets.md`  
+**Date:** 2026-08-19  
+**Scope:** revision 2, after both 2026-08-18 reviews
+
+## Verdict
+
+The revised plan is substantially better: the evidence ladder, presence-aware normalization, union admission, Stage A facet folding, signed nearest neighbors, run-scoped telemetry, as-of semantics, and cluster-cap direction resolve the important defects in both earlier reviews. The core architecture is sound. It is **not ready to execute unchanged**, however. Two issues are blocking: the evaluator filters on a run status that the application never writes, and the promised provider opt-out still leaks opted-out articles through later DeepSeek stages. Before implementation, the plan also needs to repair an impossible Phase A success criterion, persist historical taste-profile versions if replay is meant to use them, and remove cached-only facet preference from admission. These are localized amendments; they do not require redesigning the overall approach.
+
+## Critical
+
+### C1. Evaluation filters on a nonexistent `complete` run status
+
+Evidence:
+
+- §7.5 says evaluation “must ignore rows whose `runs.status != 'complete'`.”
+- §27.1 repeats that `evaluate` “must ignore runs whose `status != 'complete'`.”
+- The current `RunStatus` vocabulary is `running | ok | degraded | failed | dry_run` (`src/report.rs:19-41`), and `Db::finish_run` stores those exact values.
+- Migration `0002` adds no `complete` status and the plan does not change `RunStatus`.
+
+Implemented literally, every run is excluded from evaluation. Phase A can therefore never accumulate its required completed runs, and failed/truncated-run filtering cannot be tested meaningfully.
+
+**Required amendment:** define evaluation eligibility in terms of the real lifecycle. A reasonable default is:
+
+- production outcome metrics: `status IN ('ok', 'degraded')`;
+- shadow diagnostics: the same statuses plus the manifest’s shadow marker;
+- dry-run diagnostics: included only when explicitly requested;
+- always exclude `running` and `failed`.
+
+Use one typed helper/query predicate everywhere rather than copying SQL strings. Add tests covering all five current statuses. If the intent is instead to rename `ok` to `complete`, specify the migration, enum change, compatibility behavior, and existing-row rewrite.
+
+### C2. `no_external_content_feeds` does not actually keep articles away from DeepSeek
+
+Evidence: §25 promises that matching articles are “never sent to Voyage or DeepSeek,” then only specifies “no embedding, no facets, and no Stage A score.” But such an article remains ranked on heuristic signals and can still:
+
+1. enter the Stage B prompt, whose current renderer includes title, feed, and body-derived blurb (`src/curate/select.rs:157`);
+2. be selected and have its body sent to the per-article summary call (`src/curate/editorial.rs:119-170`);
+3. contribute title/feed/facet metadata to a later weekly profile rebuild (§22).
+
+This breaks the explicit privacy contract for the private/authenticated-feed use case that motivated the setting.
+
+**Required amendment:** centralize provider eligibility and apply it at every provider call, not only embedding and Stage A orchestration. Define whether the restriction covers article body only or all article-derived metadata. Under the strict meaning currently documented:
+
+- omit protected candidates from Stage B and reinsert any mandatory protected articles deterministically afterward, subject to `hard_max`;
+- always use local excerpt summaries for protected picks;
+- exclude their rating-history metadata from the DeepSeek profile-rebuild prompt;
+- assert with a recording mock that no Voyage or DeepSeek request contains any protected article field.
+
+If titles/metadata may be sent while bodies may not, rename and document the policy accordingly; the current “never sent” wording is broader.
+
+## High
+
+### H1. Phase A requires a counterfactual upvote that admission-only shadowing cannot observe
+
+Evidence: §32 correctly says Phase A shadows admission only while the old selector remains authoritative. Its exit gate nevertheless requires the union to “admit at least one upvoted article per week that the prefilter would have dropped.”
+
+An article dropped by the authoritative prefilter is not shown in the issue, so it cannot receive an upvote. Historical selected/upvoted articles necessarily survived the old funnel on the day they were shown. The first half of the gate—retaining at least 95% of known positives—is observable; the claimed rescued-positive rate is not. This is selection bias, not something another week of shadow data fixes.
+
+**Required amendment:** replace the impossible half of the Phase A gate with an observable measure. Options include:
+
+- weekly operator adjudication of a fixed sample of union-only candidates;
+- a small, explicitly bounded interleaving bucket that exposes union-only candidates;
+- retrospective labels from an independent source, if one genuinely exists.
+
+Then move measured user upvote yield for rescued candidates to Phase B, after those candidates can actually be exposed. Record impressions/exposure origin so this cohort can be evaluated.
+
+### H2. Replay requires historical profile selection, but the plan stores only profile metadata
+
+Evidence:
+
+- §6.1 requires profile-version selection bounded by `as_of`.
+- §6.2 says replay disables prose profile only “if no profile version was effective at `as_of`.”
+- §7.4 stores only `profile_version` and `profile_hash` in the run manifest.
+- The current implementation overwrites `taste_profile`, `taste_profile_learned`, and `profile_version` singleton keys in `kv` (`src/curate/profile/mod.rs:197-240`).
+- Migration `0002` proposes no profile-history table and the manifest does not store profile text.
+
+After the next weekly rebuild, the profile text that was effective for an earlier date is gone. The evaluator cannot select it by `as_of`, and a hash cannot reconstruct it. Silently disabling the profile would also make replay depend on whether an old version happened to survive in `kv`.
+
+**Required amendment:** add immutable profile history, for example:
+
+```sql
+CREATE TABLE taste_profile_versions (
+    version     INTEGER PRIMARY KEY,
+    built_at    TEXT NOT NULL,
+    profile_hash TEXT NOT NULL,
+    profile_text TEXT NOT NULL
+);
+CREATE INDEX idx_taste_profiles_built_at ON taste_profile_versions(built_at);
+```
+
+Write a new row transactionally whenever the singleton/current pointer changes, select the latest `built_at <= as_of`, and migrate the currently stored profile as the first historical row. If retaining profile text is unwanted, explicitly state that replay always disables the prose profile; do not promise version selection.
+
+### H3. Cached facets create an incumbency-only admission signal
+
+Evidence: §16.4 gives `facet_preference` 0.14 preliminary weight while acknowledging it is absent for new articles and present only for articles previously sent through Stage A. The plan says presence-aware renormalization “handles” the asymmetry.
+
+Presence-aware blending correctly handles outages and genuinely unavailable signals, but it does not make informative missingness fair. Previously admitted recurring articles get an extra positive or negative feature that brand-new articles cannot receive before the same admission cut. Since the 26-hour ingest window overlaps days, this makes prior Stage A admission part of the next day’s ranking and can create self-reinforcing survival. It also makes the admission formula depend on cache history rather than solely on the candidate and declared `as_of` evidence.
+
+**Required amendment:** remove `facet_preference` from the preliminary/admission blend in V1. Use it only in post-Stage-A utility, where all successfully scored candidates have equal opportunity to obtain facets. Promote it into admission only if a later dedicated or deterministic pre-admission facet path provides comparable coverage across the eligible set. Presence-aware normalization should remain for genuine provider/cache failures.
+
+### H4. “Atomic” provider budgeting is underspecified across concurrent processes
+
+Evidence: §24 requires reserve-then-spend accounting to be atomic under concurrency, but §7.6 only persists completed usage on `runs` and preloads same-date spend. An in-process meter can coordinate `buffer_unordered` tasks, but two overlapping `generate` processes can both preload the same balance, reserve locally, publish concurrently, and exceed both provider ceilings.
+
+The same overlap can race issue publication and whole-table feed-prior rebuilds. A systemd timer lowers the probability but does not prevent an operator-triggered rerun from overlapping the scheduled process.
+
+**Required amendment:** choose and specify one model:
+
+- simplest: a database-backed generation lease/mutex, with stale-lease recovery, that permits only one mutating generation process at a time;
+- more flexible: a provider reservation ledger updated in an immediate SQLite transaction, plus explicit publication serialization.
+
+Add a two-process/concurrent-connection test. If operational policy guarantees serialization instead, enforce it in the binary rather than relying on convention.
+
+## Medium
+
+### M1. The exploration ramp formula does not reach full strength at `evidence_full`
+
+§17 defines:
+
+```text
+exploration_reserve =
+  round(exploration_max *
+        clamp((W - exploration_floor) / exploration_full, 0, 1))
+```
+
+and says `exploration_full = evidence_full = 20`, with `exploration_floor = 15`. At `W = 20`, the reserve is only `8 * 5/20 = 2`; it reaches the configured maximum at `W = 35`. That conflicts with the names and with the evidence ladder’s “full at 20” semantics.
+
+**Recommendation:** either use `(W - exploration_floor) / (evidence_full - exploration_floor)`, or introduce a separate explicit `exploration_ramp_width`/full threshold and document that full exploration begins at 35. Add boundary tests at below-floor, floor, full, and above-full values.
+
+### M2. The diversification algorithm is not single-linkage clustering
+
+§20 calls the method “single-linkage cluster caps,” but its algorithm assigns a candidate to the first existing cluster containing a similar member and never merges two existing clusters bridged by a later candidate.
+
+For A similar to C, B similar to C, and A not similar to B, processing A then B then C leaves two clusters; true single linkage produces one connected component. The current method is deterministic greedy threshold assignment, but its cluster caps and explanations can differ materially from the stated design.
+
+**Recommendation:** either:
+
+- implement actual connected components/union-find over the threshold graph (cheap at 120 candidates), accepting single-linkage chaining; or
+- deliberately keep the greedy algorithm, rename it, specify cluster-order semantics, and test bridge cases.
+
+Complete-linkage or leader clustering is also worth considering if chaining entire news cycles into one cluster is undesirable.
+
+### M3. Run-manifest creation conflicts with its required fields
+
+§30 tells `pipeline.rs` to create the manifest “immediately after the run row,” but `run_manifests.rating_evidence_weight` is `NOT NULL` and is only computed when preference state is loaded later in the §5 pipeline. Profile selection and feature availability are also not necessarily final at run creation.
+
+This invites either fake defaults in supposedly authoritative manifests or incremental mutation of a snapshot described as the record needed to interpret a run.
+
+**Recommendation:** distinguish an initial run configuration from a finalized ranking manifest. Either load the bounded profile/rating evidence before inserting the manifest, or allow a clearly defined `running` manifest to be finalized transactionally before candidate rows become evaluable. Evaluation must require finalized manifest state in addition to terminal run status.
+
+### M4. The Stage A example uses an invalid facet enum value
+
+§15.2 defines the scored `format` vocabulary as:
+
+`reported_news | analysis_essay | how_to_technical | first_hand_account | announcement_roundup`.
+
+But §18.1’s canonical response example uses `"postmortem_case_study"`. Under the required tolerant parser, that value is dropped to `None`, so an implementation copied from the example loses precisely the first-hand postmortem signal used throughout the plan’s motivating examples.
+
+**Recommendation:** make the prompt example and all fixtures use the exact enum vocabulary, or add `postmortem_case_study` to the schema and update the claimed cardinality. Add a test that every enum token embedded in prompt examples is accepted by the parser.
+
+## Low
+
+### L1. `article_facets.article_id` lacks the foreign key used by the other feature tables
+
+The proposed `article_facets` schema declares `article_id INTEGER NOT NULL` without `REFERENCES articles(id) ON DELETE CASCADE`. `article_embeddings` has that relationship. Add it so article deletion or future archival cannot leave orphaned facet rows.
+
+### L2. Run mode has two writable sources of truth
+
+§7.4 stores `run_manifests.mode`, while §7.6 also adds `runs.mode`. Without a constraint or one-way derivation they can disagree, undermining the evaluator’s mode filtering. Prefer one authoritative column and expose the other through a join; if both are kept, write them in one transaction and test consistency.
+
+### L3. The lifecycle wording should distinguish “budget-degraded” from “truncated and unusable”
+
+The plan says a Stage A budget trip should exclude the run from metrics, while the current application deliberately marks guardrail trips `degraded` and can still publish a valid fallback issue. Some metrics (provider completion and Stage A accuracy) should exclude such a run, while admission, fallback behavior, issue size, and user ratings remain meaningful.
+
+Use per-stage completeness fields/counts from the report/manifest rather than excluding an otherwise valid run wholesale.
+
+## Nits
+
+- §16.1 says thin hygiene rows make “acceptance criterion 10” testable; in v2 this is acceptance criterion 12.
+- The plan alternates between “completed run” as an English phrase and the nonexistent literal status `complete`; use typed status names consistently.
+- `source TEXT -- 'stage_a' | 'dedicated'` in `article_facets` should have a `CHECK` constraint if the value is used for cache or provenance decisions.
+- `run_manifests.shadow`, `dry_run`, and `mode` should receive the same SQLite `CHECK` treatment already required for candidate flags.
+
+## Alternatives
+
+### Alternative A: Keep facets strictly post-admission in V1
+
+Remove facet preference from preliminary ranking, extract/reuse facets during Stage A, and apply facet preference only to utility. This produces uniform feature opportunity, simplifies Phase A, and preserves facets for explanation, profile rebuilding, and later learning. Prefer this until evaluation justifies a uniform pre-admission facet path.
+
+### Alternative B: Controlled interleaving for counterfactual recall evidence
+
+Reserve one or two issue slots—not merely shortlist slots—for candidates admitted only by the new union, subject to the existing quality floor. Mark their exposure origin and compare their explicit rating rate with baseline picks. Prefer this when the operator accepts a small visible experiment and wants genuine user labels. If not, use blinded operator adjudication during Phase A and postpone user-yield claims to Phase B.
+
+### Alternative C: Central provider-policy wrapper
+
+Represent external-processing permission as a typed policy on each article and require every Voyage/DeepSeek orchestration function to accept only an `ExternallyProcessableArticle` wrapper produced by one central filter. Prefer this over scattered feed checks: it makes accidental Stage B/editorial leakage harder to compile and easier to test.
+
+### Alternative D: Serialize generation rather than building a distributed budget ledger
+
+For this single-reader, single-host service, take a SQLite-backed generation lease before starting a mutating run and release it on terminal completion, with timeout/stale-owner recovery. This is simpler than cross-process provider reservations and also prevents publication and feed-prior races. Prefer the ledger only if overlapping generation is a real requirement.
+
+## Open Questions
+
+1. Which current statuses count as evaluable: `ok` only, `ok + degraded`, and/or `dry_run` for shadow diagnostics?
+2. Does `no_external_content_feeds` prohibit body text only, or titles, feed names, facets, and rating-history metadata as well?
+3. How will Phase A obtain labels for union-only candidates that the authoritative selector never exposes?
+4. Is historical prose-profile fidelity required for replay? If yes, immutable profile text must be stored; if no, replay should always disable that input and say so.
+5. Can two `generate` processes overlap in supported operation? If not, should the application reject the second invocation immediately or wait on a lease?
+6. Is true single linkage intended despite its chaining behavior, or is the current greedy first-matching-cluster algorithm the desired product rule?
+

docs/reviews/2026-08-17-personalized-ranking-and-facets-review-2026-08-20.md

@@ -0,0 +1,83 @@
+# Plan review: personalized ranking and facets, revision 8
+
+## Verdict
+
+Revision 8 is substantially stronger and most earlier architectural defects are now resolved, but it is not quite implementation-ready. Two remaining issues affect hard guarantees rather than tuning: the durable privacy authority is described inconsistently enough that an implementation can still fail open, and the provider reservation formula is not yet a proven upper bound for batched requests with provider-added tokens. Resolve those before implementation; the remaining medium/low items can be folded into the same amendment pass.
+
+## Critical
+
+### C1. The durable privacy authority is not wired into one atomic, fail-closed contract
+
+The new `article_feed_observations` table is the right data model, but four normative parts of the plan disagree about how it becomes authoritative:
+
+- Section 25.1 says `bootstrap_observation_history()` seeds `article_feed_observations`, while §7.4b defines that bootstrap as seeding only `rating_events` and `publication_events` before writing its durable marker. If implemented from §7.4b, the marker can permanently certify an incomplete privacy bootstrap.
+- Section 25.1 says `ProviderPolicy::load` builds the protected set and the wrapper consults it, but §30 specifies `externally_processable(&Article, &CurationConfig)`. That signature has neither the loaded policy nor the durable observation set and invites reimplementation of the v7 current-source check.
+- “Article persistence upserts one row per observed source” does not require those writes to be in the same transaction as `articles`/`sources_json`. If the article upsert commits and an observation insert fails, the newly persisted protected article is indistinguishable from a public one to the next policy load.
+- The general fallback rule says new external stages are non-fatal, but no rule says what happens when `ProviderPolicy::load` or an observation write fails. Treating an error as an empty protected set would disclose content under exactly the failure mode the type is supposed to prevent.
+
+This is a confidentiality boundary, so ambiguity is itself a blocker. Make one normative contract:
+
+1. Put `article_feed_observations` in the migration/observation-layer work item, and have `bootstrap_observation_history()` seed all three observation authorities plus its marker in one transaction.
+2. Persist an article and all feed observations from that ingest in one transaction. A failure rolls back both.
+3. Make the only constructor `externally_processable(&ProviderPolicy, &Article)` (or a method on `ProviderPolicy`); it must not accept configuration alone.
+4. Define failure as closed: if the policy cannot load or privacy provenance cannot be committed, make no Voyage or DeepSeek calls. The issue may continue through the local-only path, but an empty/default policy must never be substituted.
+5. Fix §33 sequencing. Step 0 is called independent, yet durable classification requires the table currently assigned nowhere explicitly and the bootstrap currently placed later. Either move the privacy table/bootstrap into step 0 or split “introduce the wrapper” from “activate provider calls” so no intermediate commit claims the guarantee without its authority.
+
+Add fault-injection tests for failure between article and observation writes, a bootstrap with existing protected `sources_json`, a pre-existing bootstrap marker, and a failed policy query. Each must result in zero external dispatches.
+
+## High
+
+### H1. `bytes + 256` is not yet a strict upper bound on provider-billed input tokens
+
+Section 7.6 correctly rejects `len/4`, but the replacement only bounds caller-visible payload bytes. The plan itself states in §11.1 that Voyage prepends a retrieval instruction server-side for `input_type = "query"`. That text is not in `payload_utf8_bytes`; moreover, a request may contain up to 1,000 inputs, so provider-added framing or instructions can scale per input rather than once per request. DeepSeek chat framing similarly scales with message structure. A fixed `per_request_overhead_tokens = 256` is therefore an assumption, not a demonstrated bound.
+
+The acceptance test is also internally contradictory: it requires settlement never to turn an admitted under-ceiling total into an over-ceiling one, while the next bullet says usage above the reservation merely trips the meter. Tripping after settlement detects that the guarantee failed; it does not preserve the pre-dispatch ceiling.
+
+Define a provider-specific bound over the actual request shape, for example:
+
+```text
+payload/body byte bound
++ request framing bound
++ per-message bound * message_count
++ per-input bound * input_count
++ maximum output tokens at the undiscounted price
+```
+
+The constants must be justified by provider limits or conservatively replaced with a documented maximum-context reservation. If no stable bound exists, weaken the product contract to “conservative guardrail” instead of “strict ceiling”; do not claim both. Add tiny-input/maximum-batch and many-message tests, with the ledger one reservation below the ceiling, so hidden overhead—not only adversarial article text—is exercised. Also add every bound constant, including `per_request_overhead_tokens`, to §38 and the manifest; the appendix currently claims every number is there, but this one is absent.
+
+## Medium
+
+### M1. Existing cached features conflict with the protected-article semantics
+
+Section 25.1 says a protected article has “no embedding, no facets” and that its rating cannot raise `W_embedding` or `W_facet`. Durable classification can, however, discover protection after an embedding/facet was already cached: the operator may add a feed ID later, or the migration may recover a currently visible protected source after an earlier public run processed the article. Nothing currently says whether those cached rows are deleted, ignored, or remain locally usable.
+
+Choose and specify one behavior. The simplest contract matching the current prose is to filter protected article IDs out of embedding/facet loads and all evidence-weight calculations, without requiring destructive deletion. Add a test that caches both features first, then marks the article protected, and verifies zero provider calls plus no contribution to either learned signal.
+
+### M2. Conservative reservations can accumulate beyond the in-flight set
+
+Section 7.6 says the roughly 4× reservation inflation applies only to at most four in-flight attempts and “never to the day’s accumulated total.” That is false for the deliberately conservative crash/retry behavior: `failed_estimated` rows and process-death `reserved` rows retain their estimates for the rest of the billing day. Several failures can therefore consume substantially more apparent budget than the concurrency limit suggests.
+
+The safe accounting rule should stay, but correct the capacity claim and make the operational consequence visible. Report standing estimated reservations separately, including stale `reserved` rows, and state that repeated ambiguous failures may intentionally halt the provider for the day. Do not add an automatic release timeout unless provider billing semantics can prove the request was not charged.
+
+## Low
+
+### L1. One normative ledger test still uses the superseded class name
+
+Section 31.11b says a run mixing “production and shadow” calls should stop shadow while production continues. The actual closed vocabulary is now `publication | shadow | maintenance`. Rename the test wording so an implementation does not recreate or alias a fourth class. Historical resolution-table uses can remain when explicitly marked superseded.
+
+## Nits
+
+- The plan header and the R8 resolution table are dated 2026-08-19, while this revision is being reviewed on 2026-08-20. Updating the revision date would make the review chain easier to audit.
+- Section 29’s example log still prints a single `W=0.0`; use the four evidence weights already required elsewhere so the canonical example does not teach an obsolete field.
+
+## Alternatives
+
+For privacy, the strongest alternative is to make externally processable status a persisted monotone article flag maintained transactionally during ingest. The separate observation table is preferable because it preserves auditability and supports deliberate unprotection by configuration, but only if all provider access goes through a successfully loaded policy and failures close the gate.
+
+For budgets, reserving the provider’s maximum accepted context per request is coarser but easier to prove than maintaining tokenizer/framing constants. It may reduce concurrency near a small ceiling, but it is a sound fallback if provider-specific hidden overhead cannot be bounded from a stable contract.
+
+## Open Questions
+
+1. On a policy-load or privacy-observation write failure, should generation abort entirely, or continue as a local-only degraded issue? The plan should choose; either is safe, while continuing external calls is not.
+2. Are cached embeddings/facets for a newly protected article allowed for local ranking, or must protection also remove their influence? The current prose chooses the latter implicitly, but the storage rules do not enforce it.
+3. Can each provider’s billed-token definition and hidden framing overhead be bounded from a stable API contract? If not, is a conservative guardrail acceptable in place of the stated strict daily ceiling?

docs/runbooks/curation-v2-migration.md

@@ -0,0 +1,283 @@
+# Runbook — upgrading the server from v1 to curation v2
+
+**Written:** 2026-09-02, for the `curation-v2` branch (all seven plan steps plus the provider
+registry). Every command below is meant to be run on the server, as `root` via `sudo`, unless it
+says otherwise. Expect the whole thing to take about an hour, most of it waiting on the embedding
+backfill and one dry run.
+
+What changes for the operator, in one paragraph: the binary is replaced; the SQLite schema gains
+tables and drops `ratings`, `feed_priors` and `scores` (the migration copies your ratings first);
+`config.toml` loses a few keys and gains the `[llm]` / `[providers.*]` registry plus a
+`profile_path`; the env file gains two API keys and renames the DeepSeek one; a hand-maintained
+`profile.md` is installed next to the OPML; the systemd units are unchanged.
+
+## 0. Before touching the server
+
+On the dev box:
+
+```sh
+git checkout curation-v2
+cargo test                      # expect green
+cargo build --release           # or build on the server, as you do today
+```
+
+The crate uses `edition = "2024"` and let-chains, so the server's toolchain must be current
+(`rustup update stable`). Read `data/profile.md` once; it is the reader profile you will be
+editing by hand from now on, and the first thing to tune when the paper feels off.
+
+## 1. Freeze the timer, back everything up
+
+```sh
+sudo systemctl stop daily-epub-generate.timer daily-epub-generate.service
+sudo systemctl stop daily-epub.service          # the server also runs migrations on start
+
+sudo install -d -m0750 -o daily-epub -g daily-epub /var/lib/daily-epub/backup
+sudo -u daily-epub sqlite3 /var/lib/daily-epub/daily-epub.db \
+  ".backup '/var/lib/daily-epub/backup/daily-epub-pre-v2-$(date +%F).db'"
+sudo cp -a /etc/daily-epub/config.toml /etc/daily-epub/config.toml.v1
+sudo cp -a /etc/daily-epub/env         /etc/daily-epub/env.v1
+sudo cp -a /usr/local/bin/daily-epub   /usr/local/bin/daily-epub.v1
+```
+
+`.backup` is the safe way to copy a WAL-mode database; a plain `cp` while anything is open can
+miss the WAL. The migration is one-way (`ratings` is dropped after being copied), so this backup
+is the rollback path.
+
+## 2. Install the new binary
+
+```sh
+sudo install -m0755 target/release/daily-epub /usr/local/bin/
+daily-epub --version
+daily-epub --help                 # confirm `config`, `stats`, `explain`, `ratings`, `features` exist
+```
+
+Do **not** start any service yet: every subcommand except `config check` opens the database and
+applies pending migrations, and you want the config right first.
+
+The units in `systemd/` did not change. If you want to be sure your installed copies match:
+
+```sh
+diff systemd/daily-epub-generate.service /etc/systemd/system/daily-epub-generate.service
+diff systemd/daily-epub.service          /etc/systemd/system/daily-epub.service
+```
+
+## 3. Edit `config.toml` in place
+
+Everything you do not mention keeps its documented default, so the edit is small. Open it with
+`sudo -e /etc/daily-epub/config.toml` and apply this checklist.
+
+**Remove** (each one now fails loudly at startup, naming its replacement):
+
+| Old key | Why |
+|---|---|
+| `prefilter_keep = …` (top level) | replaced by `curation.ranking.deep_keep` (default 120) |
+| `max_daily_usd = …` (top level) | now per provider: `providers.deepseek.max_daily_usd` |
+| the whole `[deepseek]` table | becomes `[providers.deepseek]` + `[llm]` (see below) |
+| any `[anthropic]` table (only if you added one from an interim build) | becomes `[providers.anthropic]` |
+
+**Add** near the top, next to `interests_opml`:
+
+```toml
+profile_path = "/var/lib/daily-epub/data/profile.md"
+```
+
+Use an absolute path. The default is `data/profile.md` *relative to the working directory*, which
+under the unit is `/var/lib/daily-epub`, so the default would resolve to the same place, but an
+explicit path survives running one-off commands from another directory. Point
+`interests_opml` at an absolute path too if it is still relative.
+
+**Add** the LLM registry. Carry over the `base_url`, `model` and `price_*` values from your old
+`[deepseek]` table if you had changed them; the values shown are the defaults.
+
+```toml
+[llm]
+bulk = "deepseek"            # triage, deep assessment, and the fallback for every editor call
+editor = "anthropic"         # lineup, summaries, the Brief, the weekly profile rebuild
+triage_batch_size = 25
+deep_batch_size = 8
+score_temperature = 0.3
+editorial_temperature = 0.8
+
+[providers.deepseek]
+kind = "openai"
+base_url = "https://api.deepseek.com/v1"
+model = "deepseek-v4-flash"
+max_daily_usd = 2.0
+max_concurrent_requests = 4
+price_input_per_mtok = 0.14
+price_cache_read_per_mtok = 0.0028
+price_cache_write_per_mtok = 0.0
+price_output_per_mtok = 0.28
+
+[providers.anthropic]
+kind = "anthropic"
+base_url = "https://api.anthropic.com"
+model = "claude-opus-5"
+effort = "high"
+max_daily_usd = 3.0
+max_concurrent_requests = 4
+price_input_per_mtok = 5.0
+price_cache_read_per_mtok = 0.5
+price_cache_write_per_mtok = 6.25
+price_output_per_mtok = 25.0
+```
+
+Optional, for the Gemini comparison (section 9):
+
+```toml
+[providers.gemini]
+kind = "openai"              # Gemini's OpenAI-compatible endpoint
+base_url = "https://generativelanguage.googleapis.com/v1beta/openai"
+model = "gemini-3.8-flash"
+effort = "high"
+max_daily_usd = 3.0
+max_concurrent_requests = 4
+price_input_per_mtok = 0.75          # $1.50 from 2027-01-01
+price_cache_read_per_mtok = 0.075    # $0.15 from 2027-01-01
+price_cache_write_per_mtok = 0.0
+price_output_per_mtok = 3.75         # includes thinking tokens; $7.50 from 2027-01-01
+```
+
+**Leave alone** `[miniflux]`, `[server]`, `[publish]`, `[xtc]`, `[world]`, `[curation]
+sections`, `always_include_feeds`, `blocked_domains`. `[voyage]`, `[editorial]`,
+`[curation.feedback]` and `[curation.ranking]` all have sensible defaults; copy a section from
+`config.example.toml` only when you want to change a value in it.
+
+## 4. Edit the env file
+
+```sh
+sudo -e /etc/daily-epub/env
+```
+
+| Variable | Action |
+|---|---|
+| `DAILY_EPUB_DEEPSEEK__API_KEY` | **rename** to `DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY` (the old name is rejected at startup so it cannot silently disable the bulk model) |
+| `DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY` | add |
+| `DAILY_EPUB_VOYAGE__API_KEY` | add |
+| `DAILY_EPUB_PROVIDERS__GEMINI__API_KEY` | add only if you configured `[providers.gemini]` |
+| `DAILY_EPUB_MINIFLUX__API_KEY`, `DAILY_EPUB_SERVER__HMAC_SECRET` | unchanged |
+
+Keep it `0600 daily-epub:daily-epub`. Then set hard spend limits in the DeepSeek, Anthropic and
+Voyage dashboards: the in-app `max_daily_usd` meters are runaway guards, not accounting.
+
+## 5. Install the reader profile
+
+```sh
+sudo install -d -m0750 -o daily-epub -g daily-epub /var/lib/daily-epub/data
+sudo install -m0640 -o daily-epub -g daily-epub data/profile.md /var/lib/daily-epub/data/profile.md
+# if the OPML is not already there:
+sudo install -m0640 -o daily-epub -g daily-epub data/scour-interests.opml /var/lib/daily-epub/data/
+```
+
+If the file is missing the run does not fail; it logs a warning and uses the OPML interests only,
+which is a much worse prompt. `config check` in the next step tells you whether it was found.
+
+## 6. Check the config as the service user
+
+The units run as `daily-epub` with the env file loaded, so check the same way. This helper runs
+one command in that identity, with the env file and the working directory the unit uses, and no
+other hardening:
+
+```sh
+de() { sudo systemd-run --quiet --wait --pty --collect \
+        --uid=daily-epub --gid=daily-epub \
+        -p WorkingDirectory=/var/lib/daily-epub -p EnvironmentFile=/etc/daily-epub/env \
+        /usr/local/bin/daily-epub --config /etc/daily-epub/config.toml "$@"; }
+
+de config check
+```
+
+Expected: every line is a fact, no line starts with `!`. Fix anything marked `MISSING` (a key
+name, a path) before continuing. A stale key in the TOML is reported as an error naming its
+replacement; go back to section 3.
+
+## 7. Migrate the schema
+
+```sh
+de db migrate
+sudo -u daily-epub sqlite3 /var/lib/daily-epub/daily-epub.db '.tables'
+#   expect rating_events, article_embeddings, interest_embeddings, article_assessments,
+#   candidate_runs; no ratings, feed_priors or scores
+sudo -u daily-epub sqlite3 /var/lib/daily-epub/daily-epub.db \
+  "select label, count(*) from rating_events group by label;"
+#   your old votes: up → loved, down → not_for_me, source = 'migration'
+```
+
+## 8. Warm the embedding cache, then a dry run
+
+```sh
+de features backfill --rated-only          # the learned set; prints a token estimate first
+de features backfill --days 30             # recent articles, so day one is not all cache misses
+de generate --dry-run --out /var/lib/daily-epub/out-check
+```
+
+The dry run makes real DeepSeek, Claude and Voyage calls but publishes nothing and writes no
+`issues` row. Read the printed lineup and the four report lines (`curation:`, `admission:`,
+`preference:`, `providers:`); the cost should be well under $1. Then read the EPUB it wrote
+(Calibre or KOReader): The Brief, the `Why it's here` line under each headline, and the new
+Behind-the-paper chapter before the colophon. Finally:
+
+```sh
+de explain --date "$(date +%F)" --near-misses
+```
+
+Remove `/var/lib/daily-epub/out-check` when done.
+
+## 9. Go live
+
+```sh
+sudo systemctl daemon-reload
+sudo systemctl start daily-epub.service
+sudo systemctl enable --now daily-epub-generate.timer
+sudo systemctl list-timers daily-epub-generate.timer
+```
+
+If you want today's paper regenerated by the new pipeline now rather than tomorrow at 05:30:
+`sudo systemctl start daily-epub-generate`. A same-date rerun replaces today's issue and is a
+new run id in telemetry.
+
+Afterwards:
+
+```sh
+journalctl -u daily-epub-generate -n 60 --no-pager     # the four-line info block near the end
+de stats --days 14
+```
+
+## 10. Comparing editors (Claude Opus 5 vs Gemini 3.8 Flash)
+
+With `[providers.gemini]` and its key in place, an A/B needs no config edit: environment variables
+override the TOML, so
+
+```sh
+sudo systemd-run --quiet --wait --pty --collect --uid=daily-epub --gid=daily-epub \
+  -p WorkingDirectory=/var/lib/daily-epub -p EnvironmentFile=/etc/daily-epub/env \
+  -E DAILY_EPUB_LLM__EDITOR=gemini \
+  /usr/local/bin/daily-epub --config /etc/daily-epub/config.toml \
+  generate --dry-run --date "$(date +%F)" --out /var/lib/daily-epub/out-gemini
+```
+
+produces the same date's paper with Gemini as the editor. Triage and deep assessments are cached
+per article for three days, so the second run re-requests only articles that have no row yet:
+ones ingested since the first run, and ones whose earlier batch was rejected by the provider's
+content filter (those are bisected, retried once on the editor provider, and then recorded as
+`provider_rejected` so they are not retried daily). The `triage:` and `assess:` log lines say how
+many were reused versus requested. The bulk-side cost of the second run is therefore small; the
+editor, summaries and the Brief are the real spend. Compare the two
+lineups, the `why` lines and the Brief side by side, and the `providers:` cost line. To switch
+for good, set `editor = "gemini"` in `[llm]` (and `summary_model` stays `editor`, so summaries
+move with it). The same trick works for the bulk role: `DAILY_EPUB_LLM__BULK=gemini`.
+
+## Rollback
+
+```sh
+sudo systemctl stop daily-epub-generate.timer daily-epub.service
+sudo install -m0755 /usr/local/bin/daily-epub.v1 /usr/local/bin/daily-epub
+sudo cp -a /etc/daily-epub/config.toml.v1 /etc/daily-epub/config.toml
+sudo cp -a /etc/daily-epub/env.v1         /etc/daily-epub/env
+sudo -u daily-epub cp /var/lib/daily-epub/backup/daily-epub-pre-v2-<date>.db /var/lib/daily-epub/daily-epub.db
+sudo rm -f /var/lib/daily-epub/daily-epub.db-wal /var/lib/daily-epub/daily-epub.db-shm
+sudo systemctl start daily-epub.service daily-epub-generate.timer
+```
+
+The database restore is required, not optional: the v1 binary expects `ratings`, `feed_priors`
+and `scores`, which v2's migrations drop.

examples/image_audit.rs

@@ -0,0 +1,523 @@
+//! Real-world image audit: replay the extraction + image pipeline over the
+//! articles of already-published issues and report what reaches the page.
+//!
+//! Unit tests can only prove the code does what we think on markup we wrote.
+//! This runs the same code over the pages the issues were actually built from,
+//! so a regression in lazy-image handling, URL matching or re-encoding shows up
+//! as a number rather than a hunch.
+//!
+//! ```text
+//! cargo run --release --example image_audit -- \
+//!     --cache /tmp/pagecache ~/bookorbit/books/daily-epub/*.epub
+//! ```
+//!
+//! Pages are cached on disk after the first run, so before/after comparisons
+//! see byte-identical input. Articles are re-fetched and re-extracted through
+//! readability even when the original issue took its body from Miniflux, so the
+//! counts describe the fetch path, not that specific issue's history.
+
+use std::collections::BTreeMap;
+use std::io::Read;
+use std::path::{Path, PathBuf};
+
+use daily_epub::epub::build;
+use daily_epub::extract::{self, Extractor};
+use daily_epub::html;
+use daily_epub::images;
+use daily_epub::types::{
+    Article, Colophon, Editorial, EntryId, ExtractMethod, ImageAsset, Issue, IssueMeta, Lineup,
+    Pick, SourceKind, SourceRef,
+};
+
+/// One article we are auditing, pulled back out of a published EPUB.
+#[derive(Debug, Clone)]
+struct Target {
+    issue: String,
+    entry_id: EntryId,
+    title: String,
+    url: String,
+}
+
+/// What the pipeline did with one article's images.
+#[derive(Debug, Default, Clone)]
+struct Outcome {
+    /// `<img>` elements in the extracted body.
+    refs: usize,
+    /// Images that downloaded and re-encoded successfully.
+    embedded: usize,
+    /// `<img src="images/…">` in the rendered chapter.
+    rendered: usize,
+    /// `[image: …]` paragraphs in the rendered chapter.
+    placeholders: usize,
+    /// Reasons individual images did not make it, most specific first.
+    losses: Vec<String>,
+}
+
+#[tokio::main]
+async fn main() {
+    tracing_subscriber::fmt()
+        .with_writer(std::io::stderr)
+        .with_env_filter(
+            tracing_subscriber::EnvFilter::try_from_default_env()
+                .unwrap_or_else(|_| "error".into()),
+        )
+        .init();
+
+    let mut cache = PathBuf::from("/tmp/daily-epub-audit-cache");
+    let mut dump: Option<String> = None;
+    let mut epub_out: Option<PathBuf> = None;
+    let mut epubs: Vec<PathBuf> = Vec::new();
+    let mut args = std::env::args().skip(1);
+    while let Some(arg) = args.next() {
+        match arg.as_str() {
+            "--cache" => cache = PathBuf::from(args.next().expect("--cache needs a path")),
+            "--dump" => dump = Some(args.next().expect("--dump needs a title substring")),
+            "--epub-out" => {
+                epub_out = Some(PathBuf::from(args.next().expect("--epub-out needs a path")))
+            }
+            other => epubs.push(PathBuf::from(other)),
+        }
+    }
+    if epubs.is_empty() {
+        eprintln!(
+            "usage: image_audit [--cache DIR] [--dump TITLE] [--epub-out DIR] <issue.epub>..."
+        );
+        std::process::exit(2);
+    }
+    std::fs::create_dir_all(&cache).expect("cache dir");
+
+    // One entry per article: passing `*.epub` picks up both editions of every
+    // issue, and the same article twice would collide on its asset ids.
+    let mut seen: std::collections::HashSet<EntryId> = std::collections::HashSet::new();
+    let targets: Vec<Target> = epubs
+        .iter()
+        .flat_map(|p| targets_from_epub(p))
+        .filter(|t| seen.insert(t.entry_id))
+        .collect();
+    eprintln!(
+        "auditing {} articles from {} files",
+        targets.len(),
+        epubs.len()
+    );
+
+    let http = daily_epub::http::build_client(std::time::Duration::from_secs(20)).unwrap();
+    let extractor = Extractor::new(http.clone(), vec![]);
+
+    // Extract every article first, then run one issue-wide image pass, exactly
+    // as `epub::build_edition` does.
+    let mut picks: Vec<Pick> = Vec::new();
+    let mut skipped: Vec<(Target, String)> = Vec::new();
+    for target in &targets {
+        match body_for(&cache, &http, &extractor, target).await {
+            Ok(html) => picks.push(pick_for(target, html)),
+            Err(e) => skipped.push((target.clone(), e)),
+        }
+    }
+
+    let assets =
+        images::collect_for_issue(&http, &picks, daily_epub::types::Edition::Standard).await;
+    eprintln!("embedded {} images", assets.len());
+
+    let by_entry: BTreeMap<EntryId, Vec<&ImageAsset>> = assets.iter().fold(
+        BTreeMap::new(),
+        |mut acc: BTreeMap<EntryId, Vec<&ImageAsset>>, a| {
+            if let Some(id) = entry_of(&a.id) {
+                acc.entry(id).or_default().push(a);
+            }
+            acc
+        },
+    );
+
+    let mut totals = Outcome::default();
+    let mut rows: Vec<(Target, Outcome)> = Vec::new();
+    for (target, pick) in targets_of(&targets, &picks) {
+        let refs = images::extract_img_refs(&pick.article.content_html);
+        let embedded = by_entry.get(&target.entry_id).map_or(0, |v| v.len());
+        let body = build::prepare_body(&pick.article.content_html, &assets);
+        let rendered = body.matches("<img src=\"images/").count();
+        let placeholders = body.matches("image-placeholder").count();
+
+        let embedded_urls: Vec<&str> = by_entry
+            .get(&target.entry_id)
+            .map(|v| v.iter().map(|a| a.source_url.as_str()).collect())
+            .unwrap_or_default();
+        let losses = refs
+            .iter()
+            .filter(|r| !embedded_urls.contains(&r.src.as_str()))
+            .map(|r| classify(&r.src))
+            .collect::<Vec<_>>();
+
+        if dump.as_ref().is_some_and(|d| target.title.contains(d)) {
+            println!("\n== {} [{}]", target.title, target.url);
+            for r in &refs {
+                let state = if embedded_urls.contains(&r.src.as_str()) {
+                    "ok  "
+                } else {
+                    "MISS"
+                };
+                println!("  {state} {}", r.src);
+            }
+            println!("  --- rendered body images:");
+            for cap in body.split("<img src=\"").skip(1) {
+                println!("  {}", cap.split('"').next().unwrap_or_default());
+            }
+        }
+
+        let outcome = Outcome {
+            refs: refs.len(),
+            embedded,
+            rendered,
+            placeholders,
+            losses,
+        };
+        totals.refs += outcome.refs;
+        totals.embedded += outcome.embedded;
+        totals.rendered += outcome.rendered;
+        totals.placeholders += outcome.placeholders;
+        totals.losses.extend(outcome.losses.iter().cloned());
+        rows.push((target.clone(), outcome));
+    }
+
+    println!(
+        "\n{:<12} {:>5} {:>5} {:>5} {:>5}  article",
+        "issue", "refs", "emb", "shown", "ph"
+    );
+    println!("{}", "-".repeat(100));
+    for (t, o) in &rows {
+        let flag = if o.placeholders > 0 || o.rendered < o.refs {
+            "!"
+        } else {
+            " "
+        };
+        println!(
+            "{:<12} {:>5} {:>5} {:>5} {:>5} {} {} [{}]",
+            t.issue,
+            o.refs,
+            o.embedded,
+            o.rendered,
+            o.placeholders,
+            flag,
+            truncate(&t.title, 44),
+            host(&t.url),
+        );
+        for loss in &o.losses {
+            println!("{:>36}   lost: {loss}", "");
+        }
+    }
+
+    println!("\n== totals");
+    println!("  articles           {}", rows.len());
+    println!("  <img> in bodies    {}", totals.refs);
+    println!("  embedded           {}", totals.embedded);
+    println!("  shown in chapters  {}", totals.rendered);
+    println!("  placeholders       {}", totals.placeholders);
+    println!(
+        "  orphaned assets    {}",
+        totals.embedded.saturating_sub(totals.rendered)
+    );
+    let mut kinds: BTreeMap<String, usize> = BTreeMap::new();
+    for loss in &totals.losses {
+        *kinds.entry(loss.clone()).or_default() += 1;
+    }
+    println!("\n== loss reasons");
+    for (kind, n) in &kinds {
+        println!("  {n:>4}  {kind}");
+    }
+    if let Some(dir) = &epub_out {
+        write_audit_epub(dir, &picks, &assets);
+    }
+
+    if !skipped.is_empty() {
+        println!("\n== unfetchable ({})", skipped.len());
+        for (t, e) in &skipped {
+            println!("  {} — {e}", truncate(&t.title, 50));
+        }
+    }
+}
+
+/// Pair each target with the pick built from it (targets that failed to fetch
+/// have no pick and are skipped).
+fn targets_of<'a>(targets: &'a [Target], picks: &'a [Pick]) -> Vec<(&'a Target, &'a Pick)> {
+    picks
+        .iter()
+        .filter_map(|p| {
+            targets
+                .iter()
+                .find(|t| t.entry_id == p.article.best_entry_id)
+                .map(|t| (t, p))
+        })
+        .collect()
+}
+
+fn entry_of(asset_id: &str) -> Option<EntryId> {
+    asset_id
+        .strip_prefix("img-")?
+        .split('-')
+        .next()?
+        .parse()
+        .ok()
+}
+
+/// Why one `<img>` never became an embedded asset — a guess from its URL, good
+/// enough to group the failures.
+fn classify(src: &str) -> String {
+    let lower = src.to_ascii_lowercase();
+    if src.contains('{') || src.contains('}') {
+        format!("unresolved URL template — {}", truncate(src, 90))
+    } else if src.contains(' ') || src.contains("%20") {
+        format!("srcset blob in src — {}", truncate(src, 90))
+    } else if lower.contains(".svg") {
+        format!("svg — {}", truncate(src, 90))
+    } else if src.starts_with("data:") {
+        "data: URI (lazy placeholder)".to_string()
+    } else {
+        format!("download/decode failed — {}", truncate(src, 90))
+    }
+}
+
+/// Fetch + extract one article, using the on-disk page cache.
+async fn body_for(
+    cache: &Path,
+    _http: &reqwest::Client,
+    extractor: &Extractor,
+    target: &Target,
+) -> Result<String, String> {
+    let key = cache.join(format!("{:x}.html", seahash(&target.url)));
+    let url_key = key.with_extension("url");
+    if !key.exists() {
+        // `fetch_readable` does the fetching we want but returns readability's
+        // output; cache the raw page instead so extraction changes are visible.
+        let (raw, final_url) = raw_fetch(&target.url).await?;
+        std::fs::write(&key, raw).map_err(|e| e.to_string())?;
+        std::fs::write(&url_key, final_url).map_err(|e| e.to_string())?;
+    }
+    let bytes = std::fs::read(&key).map_err(|e| e.to_string())?;
+    if bytes.is_empty() {
+        return Err("empty page".into());
+    }
+    // Relative URLs belong to the page we landed on, not the one we asked for.
+    let base = std::fs::read_to_string(&url_key).unwrap_or_else(|_| target.url.clone());
+    let html = String::from_utf8_lossy(&bytes).into_owned();
+    let _ = extractor;
+    let readable = extract::readability(&html, &base).map_err(|e| e.to_string())?;
+    Ok(extract::sanitize_with_base(
+        &images::normalize_img_tags(&readable),
+        &base,
+    ))
+}
+
+/// A plain page fetch with the extractor's desktop UA and gzip handling,
+/// returning the body and the URL the fetch landed on.
+async fn raw_fetch(url: &str) -> Result<(Vec<u8>, String), String> {
+    let client = reqwest::Client::builder()
+        .user_agent(extract::DESKTOP_UA)
+        .gzip(true)
+        .timeout(std::time::Duration::from_secs(30))
+        .build()
+        .map_err(|e| e.to_string())?;
+    let resp = client
+        .get(url)
+        .header(
+            reqwest::header::ACCEPT,
+            "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+        )
+        .send()
+        .await
+        .map_err(|e| e.to_string())?;
+    if !resp.status().is_success() {
+        return Err(format!("HTTP {}", resp.status()));
+    }
+    let final_url = resp.url().to_string();
+    resp.bytes()
+        .await
+        .map(|b| (b.to_vec(), final_url))
+        .map_err(|e| e.to_string())
+}
+
+fn pick_for(target: &Target, content_html: String) -> Pick {
+    let word_count = html::word_count(&content_html);
+    let image_urls = images::collect_image_urls(&content_html, &target.url);
+    Pick {
+        article: Article {
+            id: target.entry_id,
+            canonical_url: target.url.clone(),
+            title: target.title.clone(),
+            best_entry_id: target.entry_id,
+            content_html,
+            word_count,
+            excerpt_only: false,
+            image_count: image_urls.len() as i64,
+            image_urls,
+            sources: vec![SourceRef {
+                entry_id: target.entry_id,
+                feed_id: 1,
+                feed_title: target.issue.clone(),
+                category: None,
+                kind: SourceKind::Feed,
+            }],
+            first_seen: "2026-08-15T05:30:00Z".parse().unwrap(),
+            url: target.url.clone(),
+            author: None,
+            feed_id: 1,
+            feed_title: host(&target.url),
+            category: None,
+            published_at: None,
+            comments_url: None,
+            social: vec![],
+            extract_method: ExtractMethod::Readability,
+        },
+        section: target.issue.clone(),
+        position: 0,
+        is_lead: false,
+        why: None,
+        summary: None,
+        llm: None,
+        discussion: None,
+    }
+}
+
+/// Build a readable EPUB out of the audited articles (`--epub-out DIR`).
+///
+/// Not a regenerated issue — there is no editorial, no discussion chapters and
+/// no world briefing here, and the real thing needs the database. It exists so
+/// the images can be looked at on a device rather than counted in a table.
+fn write_audit_epub(out_dir: &Path, picks: &[Pick], assets: &[ImageAsset]) {
+    let sections: Vec<String> = {
+        let mut seen: Vec<String> = Vec::new();
+        for p in picks {
+            if !seen.contains(&p.section) {
+                seen.push(p.section.clone());
+            }
+        }
+        seen
+    };
+    let issue = Issue {
+        meta: IssueMeta {
+            date: "2026-08-16".parse().unwrap(),
+            issue_number: 0,
+            generated_at: jiff::Timestamp::now(),
+            display_date: "Image audit rebuild".into(),
+            article_count: picks.len() as i64,
+            section_count: sections.len() as i64,
+            total_words: picks.iter().map(|p| p.article.word_count).sum(),
+            reading_minutes: picks.iter().map(|p| p.article.reading_minutes()).sum(),
+        },
+        lineup: Lineup {
+            date: "2026-08-16".parse().unwrap(),
+            picks: picks.to_vec(),
+            section_order: sections,
+        },
+        editorial: Editorial {
+            front_page_html: "<p>Rebuilt from published issues by \
+                 <code>examples/image_audit.rs</code> to check image handling. \
+                 Articles are re-extracted live; editorial, discussions and the \
+                 world briefing are absent by design.</p>"
+                .into(),
+            summaries: Default::default(),
+        },
+        world_briefing: None,
+        colophon: Colophon::default(),
+        behind: Default::default(),
+    };
+    let cfg = daily_epub::config::Config::default();
+    match daily_epub::epub::build_edition_with_images(
+        &issue,
+        daily_epub::types::Edition::Standard,
+        &cfg,
+        out_dir,
+        assets,
+    ) {
+        Ok(artifact) => println!(
+            "\nwrote {} ({:.1} MB, {} images)",
+            artifact.path.display(),
+            artifact.bytes as f64 / 1_048_576.0,
+            assets.len()
+        ),
+        Err(e) => eprintln!("epub build failed: {e}"),
+    }
+}
+
+/// Pull `(entry id, title, url)` out of every article chapter in an EPUB.
+fn targets_from_epub(path: &Path) -> Vec<Target> {
+    let issue = path
+        .file_stem()
+        .map(|s| s.to_string_lossy().replace("The Daily EPUB - ", ""))
+        .unwrap_or_default();
+    let file = std::fs::File::open(path).expect("open epub");
+    let mut zip = zip::ZipArchive::new(file).expect("read epub");
+    let names: Vec<String> = (0..zip.len())
+        .filter_map(|i| zip.by_index(i).ok().map(|f| f.name().to_string()))
+        .filter(|n| n.contains("art-") && n.ends_with(".xhtml"))
+        .collect();
+
+    let mut out = Vec::new();
+    for name in names {
+        let mut xhtml = String::new();
+        if zip
+            .by_name(&name)
+            .and_then(|mut f| f.read_to_string(&mut xhtml).map_err(Into::into))
+            .is_err()
+        {
+            continue;
+        }
+        let Some(entry_id) = name
+            .rsplit('/')
+            .next()
+            .and_then(|f| f.strip_prefix("art-"))
+            .and_then(|f| f.strip_suffix(".xhtml"))
+            .and_then(|f| f.parse::<EntryId>().ok())
+        else {
+            continue;
+        };
+        let Some(url) = between(&xhtml, r#"class="read-online"><a href=""#, '"') else {
+            continue;
+        };
+        let title = between(&xhtml, "<title>", '<').unwrap_or_else(|| "?".into());
+        out.push(Target {
+            issue: issue.clone(),
+            entry_id,
+            title: unescape(&title),
+            url: unescape(&url),
+        });
+    }
+    out
+}
+
+fn between(haystack: &str, prefix: &str, end: char) -> Option<String> {
+    let start = haystack.find(prefix)? + prefix.len();
+    let rest = &haystack[start..];
+    let stop = rest.find(end)?;
+    Some(rest[..stop].to_string())
+}
+
+fn unescape(s: &str) -> String {
+    s.replace("&amp;", "&")
+        .replace("&#39;", "'")
+        .replace("&quot;", "\"")
+        .replace("&lt;", "<")
+        .replace("&gt;", ">")
+}
+
+fn host(url: &str) -> String {
+    url::Url::parse(url)
+        .ok()
+        .and_then(|u| u.host_str().map(str::to_string))
+        .unwrap_or_default()
+}
+
+fn truncate(s: &str, n: usize) -> String {
+    if s.chars().count() <= n {
+        return s.to_string();
+    }
+    s.chars().take(n.saturating_sub(1)).collect::<String>() + "…"
+}
+
+/// Tiny stable hash for cache filenames.
+fn seahash(s: &str) -> u64 {
+    let mut h: u64 = 0xcbf2_9ce4_8422_2325;
+    for b in s.as_bytes() {
+        h ^= u64::from(*b);
+        h = h.wrapping_mul(0x0000_0100_0000_01b3);
+    }
+    h
+}

migrations/0002_curation_v2.sql

@@ -0,0 +1,96 @@
+-- Personalized curation v2 schema (plan 2026-09-02 §7).
+-- `scores` intentionally remains until migration 0003 (step 4).
+
+CREATE TABLE rating_events (
+    id          INTEGER PRIMARY KEY AUTOINCREMENT,
+    article_id  INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
+    issue_date  TEXT,
+    kind        TEXT NOT NULL CHECK (kind IN ('explicit', 'implicit')),
+    source      TEXT NOT NULL,
+    label       TEXT NOT NULL,
+    value       REAL NOT NULL,
+    note        TEXT,
+    event_at    TEXT NOT NULL
+);
+
+CREATE INDEX idx_rating_events_article ON rating_events(article_id, event_at);
+CREATE INDEX idx_rating_events_at ON rating_events(event_at);
+
+-- Filled in by step 3.
+CREATE TABLE article_embeddings (
+    article_id  INTEGER PRIMARY KEY REFERENCES articles(id) ON DELETE CASCADE,
+    model       TEXT NOT NULL,
+    dimension   INTEGER NOT NULL,
+    input_hash  TEXT NOT NULL,
+    embedding   BLOB NOT NULL,
+    created_at  TEXT NOT NULL
+);
+
+-- Filled in by step 3.
+CREATE TABLE interest_embeddings (
+    interest    TEXT PRIMARY KEY,
+    model       TEXT NOT NULL,
+    dimension   INTEGER NOT NULL,
+    embedding   BLOB NOT NULL,
+    created_at  TEXT NOT NULL
+);
+
+-- Triage rows are filled in by step 4; deep rows by step 5.
+CREATE TABLE article_assessments (
+    article_id      INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
+    stage           TEXT NOT NULL CHECK (stage IN ('triage', 'deep')),
+    model           TEXT NOT NULL,
+    prompt_version  INTEGER NOT NULL,
+    profile_version INTEGER,
+    score           REAL,
+    fit             REAL,
+    kind            TEXT,
+    facets_json     TEXT,
+    rationale       TEXT,
+    category        TEXT,
+    paywalled_guess INTEGER NOT NULL DEFAULT 0,
+    assessed_at     TEXT NOT NULL,
+    PRIMARY KEY (article_id, stage)
+);
+
+CREATE INDEX idx_article_assessments_at ON article_assessments(assessed_at);
+
+-- Filled in by step 3.
+CREATE TABLE candidate_runs (
+    run_id          INTEGER NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
+    article_id      INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
+    stage           TEXT NOT NULL,
+    excluded_reason TEXT,
+    admitted_by     TEXT,
+    signals_json    TEXT NOT NULL,
+    utility         REAL,
+    rank_utility    INTEGER,
+    cluster_id      INTEGER,
+    cluster_rank    INTEGER,
+    editor_why      TEXT,
+    PRIMARY KEY (run_id, article_id)
+);
+
+CREATE INDEX idx_candidate_runs_article ON candidate_runs(article_id);
+CREATE INDEX idx_candidate_runs_run_stage ON candidate_runs(run_id, stage);
+
+-- Filled in by later telemetry/provider steps.
+ALTER TABLE runs ADD COLUMN config_json TEXT;
+ALTER TABLE runs ADD COLUMN provider_costs_json TEXT;
+-- Filled in by step 2.
+ALTER TABLE issue_articles ADD COLUMN why TEXT;
+
+INSERT INTO rating_events
+    (article_id, issue_date, kind, source, label, value, note, event_at)
+SELECT article_id,
+       issue_date,
+       'explicit',
+       'migration',
+       CASE vote WHEN 1 THEN 'loved' ELSE 'not_for_me' END,
+       CASE vote WHEN 1 THEN 1.0 ELSE -1.0 END,
+       NULL,
+       rated_at
+FROM ratings;
+
+DROP TABLE ratings;
+DROP TABLE feed_priors;

migrations/0003_drop_scores.sql

@@ -0,0 +1 @@
+DROP TABLE scores;

src/auth.rs

@@ -5,14 +5,14 @@
 //! both sides. It lives here and nowhere else:
 //!
 //! ```text
-//! message = "{issue_date}/{article_id}/{up|down}"
+//! message = "{issue_date}/{article_id}/{loved|good|down}"
 //! token   = hex(hmac_sha256(secret, message))[..16]
 //! link    = {public_url}/r/{issue_date}/{article_id}/{vote}?t={token}
 //! ```
 //!
 //! Pinned test vector, asserted from three places (here, `epub::build`,
 //! `tests/m7_server.rs`): `secret = "test-secret"`, date `2026-08-15`,
-//! article `42`, `up` → `3b314cf7e6d8f50f`.
+//! article `42`, `loved` (with legacy `up` verification).
 
 use hmac::{Hmac, KeyInit, Mac};
 use jiff::civil::Date;
@@ -23,17 +23,26 @@ use crate::types::{ArticleId, Vote};
 /// Characters of the hex HMAC kept in rating links (§3.9).
 pub const TOKEN_LEN: usize = 16;
 
-/// The exact signed string: `{issue_date}/{article_id}/{up|down}` (§3.9).
+/// The exact signed string: `{issue_date}/{article_id}/{loved|good|down}` (§3.9).
 pub fn rating_message(issue_date: Date, article_id: ArticleId, vote: Vote) -> String {
     format!("{issue_date}/{article_id}/{}", vote.as_str())
 }
 
 /// `hex(hmac_sha256(secret, "{issue_date}/{article_id}/{vote}"))[..16]` (§3.9).
 pub fn rating_token(secret: &str, issue_date: Date, article_id: ArticleId, vote: Vote) -> String {
+    rating_token_for_segment(secret, issue_date, article_id, vote.as_str())
+}
+
+fn rating_token_for_segment(
+    secret: &str,
+    issue_date: Date,
+    article_id: ArticleId,
+    segment: &str,
+) -> String {
     // `Hmac` derives a fixed-size key from any input length, so this never fails.
     let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(secret.as_bytes())
         .expect("HMAC accepts keys of any length");
-    mac.update(rating_message(issue_date, article_id, vote).as_bytes());
+    mac.update(format!("{issue_date}/{article_id}/{segment}").as_bytes());
     let digest = hex::encode(mac.finalize().into_bytes());
     digest[..TOKEN_LEN].to_string()
 }
@@ -46,10 +55,16 @@ pub fn verify_token(
     vote: Vote,
     token: &str,
 ) -> bool {
-    constant_time_eq(
+    let current = rating_token(secret, issue_date, article_id, vote);
-        rating_token(secret, issue_date, article_id, vote).as_bytes(),
+    if constant_time_eq(current.as_bytes(), token.as_bytes()) {
-        token.as_bytes(),
+        return true;
-    )
+    }
+    // Already-published `up` links were signed over the literal legacy segment.
+    vote == Vote::Loved
+        && constant_time_eq(
+            rating_token_for_segment(secret, issue_date, article_id, "up").as_bytes(),
+            token.as_bytes(),
+        )
 }
 
 /// Length-independent, data-independent byte comparison.
@@ -67,7 +82,7 @@ pub fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
 }
 
 /// Full rating URL embedded in an article footer:
-/// `{public_url}/r/{date}/{article_id}/{up|down}?t={token}` (§3.9).
+/// `{public_url}/r/{date}/{article_id}/{loved|good|down}?t={token}` (§3.9).
 pub fn rating_url(
     public_url: &str,
     secret: &str,
@@ -92,50 +107,55 @@ mod tests {
     }
 
     #[test]
-    fn token_matches_the_shared_vector() {
+    fn all_three_tokens_verify_and_are_distinct() {
-        assert_eq!(rating_message(date(), 42, Vote::Up), "2026-08-15/42/up");
+        let votes = [Vote::Loved, Vote::Good, Vote::NotForMe];
-        assert_eq!(
+        let tokens: Vec<String> = votes
-            rating_token("test-secret", date(), 42, Vote::Up),
+            .iter()
-            "3b314cf7e6d8f50f"
+            .map(|vote| rating_token("test-secret", date(), 42, *vote))
-        );
+            .collect();
-        assert_eq!(rating_token("test-secret", date(), 42, Vote::Up).len(), 16);
+        assert_eq!(tokens.len(), 3);
+        assert!(tokens.iter().all(|token| token.len() == TOKEN_LEN));
+        assert_ne!(tokens[0], tokens[1]);
+        assert_ne!(tokens[1], tokens[2]);
+        for (vote, token) in votes.into_iter().zip(tokens) {
+            assert!(verify_token("test-secret", date(), 42, vote, &token));
+        }
     }
 
     #[test]
-    fn tokens_are_per_article_and_per_vote() {
+    fn legacy_up_token_still_verifies_as_loved() {
-        let up = rating_token("s", date(), 42, Vote::Up);
+        let legacy = rating_token_for_segment("test-secret", date(), 42, "up");
-        assert_ne!(up, rating_token("s", date(), 42, Vote::Down));
+        assert_eq!(legacy, "3b314cf7e6d8f50f");
-        assert_ne!(up, rating_token("s", date(), 43, Vote::Up));
-        assert_ne!(up, rating_token("other", date(), 42, Vote::Up));
-        let tomorrow: Date = "2026-08-16".parse().unwrap();
-        assert_ne!(up, rating_token("s", tomorrow, 42, Vote::Up));
-    }
-
-    #[test]
-    fn verification_is_exact() {
         assert!(verify_token(
-            "s",
+            "test-secret",
             date(),
             42,
-            Vote::Up,
+            Vote::Loved,
-            &rating_token("s", date(), 42, Vote::Up)
+            &legacy
         ));
-        assert!(!verify_token("s", date(), 42, Vote::Up, "deadbeefdeadbeef"));
+    }
-        assert!(!verify_token("s", date(), 42, Vote::Up, ""));
+
-        assert!(!verify_token("s", date(), 42, Vote::Up, "short"));
+    #[test]
+    fn verification_rejects_tampering() {
+        let token = rating_token("s", date(), 42, Vote::Loved);
+        assert!(!verify_token("s", date(), 42, Vote::Good, &token));
+        assert!(!verify_token("s", date(), 43, Vote::Loved, &token));
+        assert!(!verify_token("other", date(), 42, Vote::Loved, &token));
+        assert!(!verify_token("s", date(), 42, Vote::Loved, "short"));
     }
 
     #[test]
     fn url_shape_matches_the_spec() {
+        let token = rating_token("test-secret", date(), 42, Vote::Good);
         assert_eq!(
             rating_url(
                 "https://daily.hallada.net/",
                 "test-secret",
                 date(),
                 42,
-                Vote::Up
+                Vote::Good
             ),
-            "https://daily.hallada.net/r/2026-08-15/42/up?t=3b314cf7e6d8f50f"
+            format!("https://daily.hallada.net/r/2026-08-15/42/good?t={token}")
         );
     }
 }

src/comments.rs

@@ -9,7 +9,7 @@ use std::collections::HashSet;
 use futures::StreamExt;
 use serde_json::Value;
 
-use crate::epub::images::{text_escape, to_xhtml};
+use crate::html::{text_escape, to_xhtml};
 use crate::types::{Comment, CommentThread, Discussion, Pick, SocialSource};
 
 /// Top-level threads kept per source (§3.7).

src/curate/admit.rs

@@ -0,0 +1,542 @@
+//! Hygiene and union admission into the deep set (plan §8.1, §11).
+
+use std::collections::{BTreeMap, HashSet};
+
+use jiff::Timestamp;
+use jiff::civil::Date;
+use sha2::{Digest, Sha256};
+use sqlx::Row as _;
+
+use super::{prefilter, telemetry};
+use crate::config::{CurationConfig, RankingConfig};
+use crate::db::{Db, fmt_ts};
+use crate::types::{Article, ArticleId, Candidate};
+
+/// Run hygiene before embeddings and write thin telemetry rows for exclusions.
+pub async fn hygiene(
+    db: &Db,
+    run_id: i64,
+    articles: Vec<Article>,
+    date: Date,
+    config: &CurationConfig,
+    now: Timestamp,
+) -> anyhow::Result<Vec<Candidate>> {
+    let published = db
+        .previously_published_ids_before(date)
+        .await?
+        .into_iter()
+        .collect::<HashSet<_>>();
+    let since = now - jiff::Span::new().hours(config.recent_rejection_days.max(0) * 24);
+    let rows = sqlx::query(
+        "SELECT DISTINCT article_id FROM article_assessments
+         WHERE stage IN ('triage', 'deep') AND score IS NOT NULL AND score < ?
+           AND assessed_at >= ?",
+    )
+    .bind(config.recent_rejection_floor)
+    .bind(fmt_ts(since))
+    .fetch_all(db.pool())
+    .await?;
+    let rejected = rows
+        .iter()
+        .map(|row| row.get::<i64, _>("article_id"))
+        .collect::<HashSet<_>>();
+
+    let mut eligible = Vec::new();
+    for article in articles {
+        let auto_include = prefilter::is_auto_include(&article, config);
+        let reason = if auto_include {
+            None
+        } else if prefilter::is_blocked(&article, config) {
+            Some("blocked")
+        } else if published.contains(&article.id) {
+            Some("published_before")
+        } else if rejected.contains(&article.id) {
+            Some("recently_rejected")
+        } else {
+            None
+        };
+        if let Some(reason) = reason {
+            telemetry::thin_excluded(db, run_id, article.id, reason).await?;
+        } else {
+            eligible.push(Candidate::new(article, auto_include));
+        }
+    }
+    Ok(eligible)
+}
+
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct AdmissionSummary {
+    pub admitted: usize,
+    pub admitted_by: BTreeMap<String, usize>,
+    pub exploration_admitted: usize,
+}
+
+pub fn admit(
+    candidates: &mut [Candidate],
+    date: Date,
+    ranking: &RankingConfig,
+) -> AdmissionSummary {
+    for candidate in candidates.iter_mut() {
+        candidate.admitted_by.clear();
+        candidate.exploration = false;
+        if candidate.excluded_reason.as_deref() != Some("not_admitted") {
+            candidate.excluded_reason = None;
+        }
+    }
+    let mut admitted = HashSet::new();
+    for (index, candidate) in candidates.iter_mut().enumerate() {
+        if candidate.excluded_reason.is_none() && candidate.auto_include {
+            candidate.admitted_by.push("auto_include".into());
+            admitted.insert(index);
+        }
+    }
+
+    let capacity = |admitted: &HashSet<usize>| ranking.deep_keep.saturating_sub(admitted.len());
+
+    let triage = ranked(candidates, |candidate| {
+        candidate
+            .assessment
+            .triage
+            .as_ref()
+            .filter(|triage| triage.interest >= 5.0)
+            .map(|triage| triage.interest)
+    });
+    let quota = ranking.quotas.triage.min(capacity(&admitted));
+    take_retriever(
+        candidates,
+        &mut admitted,
+        &triage,
+        ranking.quotas.triage,
+        quota,
+        "triage",
+    );
+
+    let interest = ranked(candidates, |candidate| {
+        semantic_floor(candidate, ranking)
+            .then_some(candidate.signals.interest)
+            .flatten()
+    });
+    if !interest.is_empty() {
+        let quota = ranking.quotas.interest.min(capacity(&admitted));
+        take_retriever(
+            candidates,
+            &mut admitted,
+            &interest,
+            ranking.quotas.interest,
+            quota,
+            "interest",
+        );
+    }
+
+    let knn = ranked(candidates, |candidate| {
+        semantic_floor(candidate, ranking)
+            .then_some(candidate.signals.knn.filter(|score| *score > 0.0))
+            .flatten()
+    });
+    if !knn.is_empty() {
+        let quota = ranking.quotas.knn.min(capacity(&admitted));
+        take_retriever(
+            candidates,
+            &mut admitted,
+            &knn,
+            ranking.quotas.knn,
+            quota,
+            "knn",
+        );
+    }
+
+    let mut by_blend = ranked(candidates, |candidate| candidate.signals.preliminary);
+    let band_end = ((ranking.deep_keep as f64) * 2.5).ceil() as usize;
+    let band_start = ranking.deep_keep.min(by_blend.len());
+    by_blend.truncate(band_end.min(by_blend.len()));
+    let mut exploration = by_blend
+        .into_iter()
+        .skip(band_start)
+        .filter(|index| {
+            let candidate = &candidates[*index];
+            candidate.article.word_count >= 300
+                && !prefilter::looks_like_roundup(&candidate.article.title)
+                && candidate
+                    .assessment
+                    .triage
+                    .as_ref()
+                    .is_some_and(|triage| triage.interest >= 4.0)
+        })
+        .collect::<Vec<_>>();
+    exploration.sort_by_key(|index| exploration_key(date, candidates[*index].article.id));
+    let quota = ranking.exploration_slots.min(capacity(&admitted));
+    take_retriever(
+        candidates,
+        &mut admitted,
+        &exploration,
+        ranking.exploration_slots,
+        quota,
+        "exploration",
+    );
+    for index in &admitted {
+        if candidates[*index]
+            .admitted_by
+            .first()
+            .is_some_and(|name| name == "exploration")
+        {
+            candidates[*index].exploration = true;
+        }
+    }
+
+    let blend = ranked(candidates, |candidate| candidate.signals.preliminary);
+    let quota = capacity(&admitted);
+    take_retriever(candidates, &mut admitted, &blend, quota, quota, "blend");
+
+    for (index, candidate) in candidates.iter_mut().enumerate() {
+        if admitted.contains(&index) {
+            candidate.stage = "admitted".into();
+            candidate.excluded_reason = None;
+        } else if candidate.excluded_reason.is_none() {
+            candidate.stage = if candidate.assessment.triage.is_some() {
+                "triaged".into()
+            } else {
+                "eligible".into()
+            };
+            candidate.excluded_reason = Some("not_admitted".into());
+        }
+    }
+    let mut summary = AdmissionSummary {
+        admitted: admitted.len(),
+        exploration_admitted: admitted
+            .iter()
+            .filter(|index| candidates[**index].exploration)
+            .count(),
+        ..AdmissionSummary::default()
+    };
+    for index in admitted {
+        if let Some(first) = candidates[index].admitted_by.first() {
+            *summary.admitted_by.entry(first.clone()).or_default() += 1;
+        }
+    }
+    summary
+}
+
+fn semantic_floor(candidate: &Candidate, ranking: &RankingConfig) -> bool {
+    candidate.article.word_count >= ranking.semantic_min_words
+        && !prefilter::looks_like_roundup(&candidate.article.title)
+        && candidate
+            .assessment
+            .triage
+            .as_ref()
+            .is_none_or(|triage| triage.interest >= 3.0)
+}
+
+fn ranked(candidates: &[Candidate], signal: impl Fn(&Candidate) -> Option<f64>) -> Vec<usize> {
+    let mut values = candidates
+        .iter()
+        .enumerate()
+        .filter(|(_, candidate)| candidate.excluded_reason.is_none())
+        .filter_map(|(index, candidate)| signal(candidate).map(|value| (index, value)))
+        .collect::<Vec<_>>();
+    values.sort_by(|(left_index, left), (right_index, right)| {
+        right.total_cmp(left).then_with(|| {
+            candidates[*left_index]
+                .article
+                .id
+                .cmp(&candidates[*right_index].article.id)
+        })
+    });
+    values.into_iter().map(|(index, _)| index).collect()
+}
+
+fn take_retriever(
+    candidates: &mut [Candidate],
+    admitted: &mut HashSet<usize>,
+    ranked: &[usize],
+    would_take: usize,
+    admit_quota: usize,
+    name: &str,
+) {
+    // Record overlap among this retriever's own top-N.
+    for index in ranked.iter().take(would_take) {
+        if admitted.contains(index)
+            && !candidates[*index]
+                .admitted_by
+                .iter()
+                .any(|value| value == name)
+        {
+            candidates[*index].admitted_by.push(name.into());
+        }
+    }
+    let mut taken = 0;
+    for index in ranked {
+        if admitted.contains(index) {
+            continue;
+        }
+        if taken >= admit_quota {
+            break;
+        }
+        candidates[*index].admitted_by.push(name.into());
+        admitted.insert(*index);
+        taken += 1;
+    }
+}
+
+fn exploration_key(date: Date, article_id: ArticleId) -> [u8; 32] {
+    let mut hasher = Sha256::new();
+    hasher.update(date.to_string().as_bytes());
+    hasher.update(article_id.to_string().as_bytes());
+    hasher.finalize().into()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::config::{CurationConfig, RankingConfig, RankingQuotas};
+    use crate::curate::prefilter::tests::article;
+    use crate::types::Triage;
+
+    fn candidate(
+        id: i64,
+        words: i64,
+        interest: Option<f64>,
+        knn: Option<f64>,
+        triage: Option<f64>,
+    ) -> Candidate {
+        let mut candidate = Candidate::new(article(id, &format!("article {id}"), words), false);
+        candidate.signals.interest = interest;
+        candidate.signals.knn = knn;
+        candidate.signals.preliminary = Some(id as f64);
+        candidate.assessment.triage = triage.map(|interest| Triage {
+            interest,
+            kind: "essay".into(),
+            why: "specific".into(),
+            model: "mock".into(),
+            prompt_version: 1,
+            assessed_at: "2026-09-02T05:30:00Z".parse().expect("timestamp"),
+        });
+        candidate
+    }
+
+    #[test]
+    fn semantic_retrievers_reject_stubs_and_honor_quotas() {
+        let config = RankingConfig {
+            deep_keep: 4,
+            exploration_slots: 0,
+            quotas: RankingQuotas {
+                triage: 0,
+                interest: 2,
+                knn: 1,
+            },
+            ..RankingConfig::default()
+        };
+        let mut candidates = vec![
+            candidate(1, 60, Some(100.0), Some(100.0), None),
+            candidate(2, 600, Some(9.0), None, None),
+            candidate(3, 600, Some(8.0), None, None),
+            candidate(4, 600, None, Some(0.8), None),
+            candidate(5, 600, None, None, None),
+        ];
+        let summary = admit(
+            &mut candidates,
+            "2026-09-02".parse().expect("date"),
+            &config,
+        );
+        assert_eq!(summary.admitted_by.get("interest"), Some(&2));
+        assert_eq!(summary.admitted_by.get("knn"), Some(&1));
+        assert!(
+            !candidates[0]
+                .admitted_by
+                .iter()
+                .any(|by| by == "interest" || by == "knn")
+        );
+        assert_eq!(summary.admitted, 4);
+        assert_eq!(summary.admitted_by.get("blend"), Some(&1));
+    }
+
+    #[test]
+    fn strong_interest_weak_heuristic_reaches_deep_set_and_auto_always_wins() {
+        let config = RankingConfig {
+            deep_keep: 2,
+            exploration_slots: 0,
+            quotas: RankingQuotas {
+                triage: 0,
+                interest: 1,
+                knn: 0,
+            },
+            ..RankingConfig::default()
+        };
+        let mut candidates = vec![
+            candidate(1, 400, Some(9.0), None, None),
+            candidate(2, 100, None, None, None),
+            candidate(3, 4000, None, None, None),
+        ];
+        candidates[0].signals.heuristic = Some(0.0);
+        candidates[1].auto_include = true;
+        let summary = admit(
+            &mut candidates,
+            "2026-09-02".parse().expect("date"),
+            &config,
+        );
+        assert_eq!(summary.admitted, 2);
+        assert_eq!(
+            candidates[0].admitted_by.first().map(String::as_str),
+            Some("interest")
+        );
+        assert_eq!(
+            candidates[1].admitted_by.first().map(String::as_str),
+            Some("auto_include")
+        );
+    }
+
+    #[test]
+    fn inactive_retrievers_release_slots_to_blend() {
+        let config = RankingConfig {
+            deep_keep: 3,
+            exploration_slots: 0,
+            ..RankingConfig::default()
+        };
+        let mut candidates = (1..=5)
+            .map(|id| candidate(id, 500, None, None, None))
+            .collect::<Vec<_>>();
+        let summary = admit(
+            &mut candidates,
+            "2026-09-02".parse().expect("date"),
+            &config,
+        );
+        assert_eq!(summary.admitted_by.get("blend"), Some(&3));
+    }
+
+    #[test]
+    fn exploration_is_stable_for_a_date_and_rotates() {
+        let config = RankingConfig {
+            deep_keep: 4,
+            exploration_slots: 2,
+            quotas: RankingQuotas {
+                triage: 0,
+                interest: 0,
+                knn: 0,
+            },
+            ..RankingConfig::default()
+        };
+        let base = (1..=12)
+            .map(|id| candidate(id, 500, None, None, Some(5.0)))
+            .collect::<Vec<_>>();
+        let mut first = base.clone();
+        admit(&mut first, "2026-09-02".parse().expect("date"), &config);
+        let ids = |items: &[Candidate]| {
+            items
+                .iter()
+                .filter(|candidate| candidate.exploration)
+                .map(|candidate| candidate.article.id)
+                .collect::<Vec<_>>()
+        };
+        let expected = ids(&first);
+        let mut again = base.clone();
+        admit(&mut again, "2026-09-02".parse().expect("date"), &config);
+        assert_eq!(ids(&again), expected);
+        let mut next = base;
+        admit(&mut next, "2026-09-03".parse().expect("date"), &config);
+        assert_ne!(ids(&next), expected);
+    }
+
+    #[tokio::test]
+    async fn recent_low_triage_is_excluded_but_auto_include_is_spared() {
+        let dir = tempfile::tempdir().expect("tempdir");
+        let db = Db::open_and_migrate(&dir.path().join("hygiene.db"))
+            .await
+            .expect("db");
+        sqlx::query(
+            "INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
+             (1, 'https://example.com/1', 'Rejected', '2026-09-02T00:00:00Z'),
+             (2, 'https://example.com/2', 'Auto', '2026-09-02T00:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .expect("articles");
+        sqlx::query(
+            "INSERT INTO article_assessments
+             (article_id, stage, model, prompt_version, score, assessed_at) VALUES
+             (1, 'triage', 'model', 1, 2.0, '2026-09-02T04:00:00Z'),
+             (2, 'triage', 'model', 1, 1.0, '2026-09-02T04:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .expect("assessments");
+        let run_id = db
+            .start_run(
+                "2026-09-02".parse().expect("date"),
+                "2026-09-02T05:30:00Z".parse().expect("timestamp"),
+            )
+            .await
+            .expect("run");
+        let config = CurationConfig {
+            always_include_feeds: vec!["99".into()],
+            ..CurationConfig::default()
+        };
+        let normal = article(1, "Rejected", 500);
+        let mut auto = article(2, "Auto", 500);
+        auto.feed_id = 99;
+        let eligible = hygiene(
+            &db,
+            run_id,
+            vec![normal, auto],
+            "2026-09-02".parse().expect("date"),
+            &config,
+            "2026-09-02T05:30:00Z".parse().expect("timestamp"),
+        )
+        .await
+        .expect("hygiene");
+        assert_eq!(eligible.len(), 1);
+        assert_eq!(eligible[0].article.id, 2);
+        assert!(eligible[0].auto_include);
+        let reason: String = sqlx::query_scalar(
+            "SELECT excluded_reason FROM candidate_runs WHERE run_id = ? AND article_id = 1",
+        )
+        .bind(run_id)
+        .fetch_one(db.pool())
+        .await
+        .expect("thin row");
+        assert_eq!(reason, "recently_rejected");
+    }
+
+    #[tokio::test]
+    async fn provider_rejected_rows_do_not_mark_an_article_recently_rejected() {
+        let dir = tempfile::tempdir().expect("tempdir");
+        let db = Db::open_and_migrate(&dir.path().join("hygiene.db"))
+            .await
+            .expect("db");
+        sqlx::query(
+            "INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
+             (1, 'https://example.com/1', 'Refused', '2026-09-02T00:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .expect("article");
+        sqlx::query(
+            "INSERT INTO article_assessments
+             (article_id, stage, model, prompt_version, score, fit, kind, rationale, assessed_at)
+             VALUES (1, 'triage', 'model', 1, NULL, NULL, 'provider_rejected',
+                     'deepseek: Content Exists Risk', '2026-09-02T04:00:00Z'),
+                    (1, 'deep', 'model', 1, NULL, NULL, 'provider_rejected',
+                     'deepseek: Content Exists Risk', '2026-09-02T04:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .expect("rejection rows");
+        let run_id = db
+            .start_run(
+                "2026-09-02".parse().expect("date"),
+                "2026-09-02T05:30:00Z".parse().expect("timestamp"),
+            )
+            .await
+            .expect("run");
+        let eligible = hygiene(
+            &db,
+            run_id,
+            vec![article(1, "Refused", 500)],
+            "2026-09-02".parse().expect("date"),
+            &CurationConfig::default(),
+            "2026-09-02T05:30:00Z".parse().expect("timestamp"),
+        )
+        .await
+        .expect("hygiene");
+        assert_eq!(eligible.len(), 1, "a NULL score is not a low score");
+        assert_eq!(eligible[0].article.id, 1);
+    }
+}

src/curate/batch.rs

@@ -0,0 +1,839 @@
+//! Bisecting batch runner shared by triage and deep assessment.
+//!
+//! DeepSeek's input content filter rejects a whole request when any one
+//! article in the batch trips it (`400 Content Exists Risk`) and does not say
+//! which. Sending the batch once and giving up cost every other article its
+//! assessment and retried them all the next day. Instead, a batch that fails
+//! with a non-transient error is split in half and both halves are sent
+//! again, down to single articles. A single article that still fails is
+//! "rejected": it is retried once on the editor client when that runs on a
+//! different provider, and otherwise reported so the caller can persist a
+//! `provider_rejected` row and stop asking (§10, §12.1, §17).
+//!
+//! Transient errors are retried inside [`LlmClient::complete`] and, once
+//! exhausted, leave the batch unassessed as before; a budget trip stops the
+//! stage. Concurrency is the caller's `max_concurrent_requests` across the
+//! original batches; the bisection inside one batch runs sequentially.
+
+use std::collections::HashSet;
+
+use futures::{StreamExt, stream};
+
+use super::llm::{LlmClient, LlmError};
+use crate::types::{ArticleId, Candidate};
+
+/// Longest rejection message kept in `article_assessments.rationale`.
+pub const REJECTION_MESSAGE_CHARS: usize = 200;
+
+/// A parsed per-article item the runner can attribute to a candidate.
+pub trait Assessed {
+    fn article_id(&self) -> ArticleId;
+}
+
+/// One stage's prompt builder, parser and clients.
+pub struct BatchRunner<'a, T> {
+    /// The bulk client every batch goes to first.
+    pub llm: &'a LlmClient,
+    /// The editor client, tried once per rejected article when it runs on a
+    /// different provider than `llm` and its meter is not tripped.
+    pub fallback: Option<&'a LlmClient>,
+    pub temperature: f32,
+    pub build_prompt: &'a (dyn Fn(&[&Candidate]) -> String + Sync),
+    pub parse: &'a (dyn Fn(&str) -> Vec<T> + Sync),
+}
+
+impl<T> BatchRunner<'_, T> {
+    /// The editor client when it is a real alternative to the bulk one.
+    pub fn usable_fallback(&self) -> Option<&LlmClient> {
+        self.fallback
+            .filter(|fallback| fallback.provider() != self.llm.provider())
+            .filter(|fallback| fallback.meter.check_budget().is_ok())
+    }
+
+    /// The name of the provider rejected articles are retried on, if any.
+    pub fn fallback_provider(&self) -> Option<String> {
+        self.fallback
+            .filter(|fallback| fallback.provider() != self.llm.provider())
+            .map(|fallback| fallback.provider().to_string())
+    }
+}
+
+/// An item together with the model that produced it (bulk or editor).
+#[derive(Debug, Clone, PartialEq)]
+pub struct Scored<T> {
+    pub item: T,
+    pub model: String,
+}
+
+/// A single article both the bulk provider and the fallback refused.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Rejection {
+    pub id: ArticleId,
+    /// The bulk provider's name.
+    pub provider: String,
+    /// The bulk provider's message, at most [`REJECTION_MESSAGE_CHARS`] long.
+    pub message: String,
+}
+
+impl Rejection {
+    /// `<provider>: <message>`, the `rationale` of a `provider_rejected` row.
+    pub fn rationale(&self) -> String {
+        format!("{}: {}", self.provider, self.message)
+    }
+}
+
+/// What a set of batches produced.
+#[derive(Debug, Clone, PartialEq)]
+pub struct BatchOutcome<T> {
+    pub items: Vec<Scored<T>>,
+    /// Articles nobody would assess; the caller persists these.
+    pub rejections: Vec<Rejection>,
+    /// Single articles the bulk provider rejected, recovered or not.
+    pub rejected: usize,
+    /// Of `rejected`, those the fallback client assessed.
+    pub recovered: usize,
+    /// Requests made on the bulk client.
+    pub requests: usize,
+    /// True when the bulk budget tripped and work was left undone.
+    pub budget_stopped: bool,
+}
+
+impl<T> Default for BatchOutcome<T> {
+    fn default() -> Self {
+        Self {
+            items: Vec::new(),
+            rejections: Vec::new(),
+            rejected: 0,
+            recovered: 0,
+            requests: 0,
+            budget_stopped: false,
+        }
+    }
+}
+
+impl<T> BatchOutcome<T> {
+    fn absorb(&mut self, other: Self) {
+        self.items.extend(other.items);
+        self.rejections.extend(other.rejections);
+        self.rejected += other.rejected;
+        self.recovered += other.recovered;
+        self.requests += other.requests;
+        self.budget_stopped |= other.budget_stopped;
+    }
+}
+
+/// The per-stage totals behind the `triage:` / `assess:` log line and the
+/// run report's `*_reused` / `*_rejected` counts.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct StageSummary {
+    pub stage: &'static str,
+    /// Articles the stage was responsible for this run.
+    pub pool: usize,
+    /// Served from cached rows within `assessment_reuse_days`.
+    pub reused: usize,
+    /// Skipped because a `provider_rejected` row is still fresh.
+    pub known_rejected: usize,
+    /// Sent to the bulk provider.
+    pub requested: usize,
+    /// Original batches before any bisection.
+    pub batches: usize,
+    /// Assessments written this run (bulk and editor).
+    pub applied: usize,
+    /// Single articles the bulk provider rejected this run.
+    pub rejected: usize,
+    /// Of `rejected`, those the editor client assessed instead.
+    pub recovered: usize,
+    pub fallback_provider: Option<String>,
+}
+
+impl StageSummary {
+    /// Articles carrying this stage's assessment after the run.
+    pub fn assessed(&self) -> usize {
+        self.reused + self.applied
+    }
+
+    /// Articles without an assessment because a provider rejected them:
+    /// this run's unrecovered rejections plus the cached ones.
+    pub fn rejected_total(&self) -> usize {
+        self.known_rejected + self.rejected.saturating_sub(self.recovered)
+    }
+
+    /// `triage: 398 in pool · 210 reused · 188 requested in 8 batches · 3 rejected (2 recovered on gemini)`
+    pub fn info_line(&self) -> String {
+        let mut line = format!(
+            "{}: {} in pool · {} reused · {} requested in {} batches · {} rejected",
+            self.stage, self.pool, self.reused, self.requested, self.batches, self.rejected
+        );
+        if self.recovered > 0 {
+            line.push_str(&format!(
+                " ({} recovered on {})",
+                self.recovered,
+                self.fallback_provider.as_deref().unwrap_or("editor")
+            ));
+        }
+        if self.known_rejected > 0 {
+            line.push_str(&format!(" · {} known rejected", self.known_rejected));
+        }
+        line
+    }
+}
+
+/// Errors the provider will keep returning for the same input.
+fn is_rejection(error: &LlmError) -> bool {
+    matches!(
+        error,
+        LlmError::Api { .. } | LlmError::Refusal { .. } | LlmError::EmptyResponse { .. }
+    )
+}
+
+/// The provider's own words, without the `<provider> request failed:` prefix.
+fn rejection_message(error: &LlmError) -> String {
+    match error {
+        LlmError::Api { message, .. } => message.clone(),
+        LlmError::Refusal { .. } => "returned a refusal".into(),
+        LlmError::EmptyResponse { .. } => "returned an empty completion".into(),
+        other => other.to_string(),
+    }
+}
+
+fn truncate_chars(text: &str, max_chars: usize) -> String {
+    let collapsed = text.split_whitespace().collect::<Vec<_>>().join(" ");
+    if collapsed.chars().count() <= max_chars {
+        return collapsed;
+    }
+    let mut cut = collapsed
+        .chars()
+        .take(max_chars.saturating_sub(1))
+        .collect::<String>();
+    cut.push('…');
+    cut
+}
+
+enum Failure {
+    Rejected(String),
+    NoItems,
+}
+
+struct Work<'c> {
+    batch: Vec<&'c Candidate>,
+    /// A batch that parses to zero items is split once; its halves are not.
+    zero_split: bool,
+}
+
+/// Run every batch on the bulk client, `max_concurrent` in flight.
+pub async fn run_batches<T: Assessed>(
+    runner: &BatchRunner<'_, T>,
+    batches: Vec<Vec<&Candidate>>,
+    max_concurrent: usize,
+) -> BatchOutcome<T> {
+    let outcomes = stream::iter(batches)
+        .map(|batch| run_one(runner, batch))
+        .buffer_unordered(max_concurrent.max(1))
+        .collect::<Vec<_>>()
+        .await;
+    let mut merged = BatchOutcome::default();
+    for outcome in outcomes {
+        merged.absorb(outcome);
+    }
+    merged
+}
+
+/// One original batch, bisected as far as it needs to be.
+async fn run_one<T: Assessed>(
+    runner: &BatchRunner<'_, T>,
+    batch: Vec<&Candidate>,
+) -> BatchOutcome<T> {
+    let original = batch.len();
+    let provider = runner.llm.provider();
+    let mut out = BatchOutcome::default();
+    let mut warned = false;
+    // Explicit LIFO stack instead of recursion: the left half is pushed last
+    // so it runs first, which keeps the request order predictable.
+    let mut stack = vec![Work {
+        batch,
+        zero_split: true,
+    }];
+    while let Some(work) = stack.pop() {
+        if let Err(error) = runner.llm.meter.check_budget() {
+            tracing::warn!(%error, "bulk budget tripped; skipping the rest of the batch");
+            out.budget_stopped = true;
+            return out;
+        }
+        let size = work.batch.len();
+        let prompt = (runner.build_prompt)(&work.batch);
+        let allowed = work
+            .batch
+            .iter()
+            .map(|candidate| candidate.article.id)
+            .collect::<HashSet<_>>();
+        out.requests += 1;
+        let failure = match runner.llm.complete(&prompt, runner.temperature, true).await {
+            Ok(raw) => {
+                let items = (runner.parse)(&raw)
+                    .into_iter()
+                    .filter(|item| allowed.contains(&item.article_id()))
+                    .collect::<Vec<_>>();
+                if items.is_empty() {
+                    Failure::NoItems
+                } else {
+                    out.items.extend(items.into_iter().map(|item| Scored {
+                        item,
+                        model: runner.llm.model.clone(),
+                    }));
+                    continue;
+                }
+            }
+            Err(LlmError::BudgetExceeded { spent, limit }) => {
+                tracing::warn!(
+                    spent,
+                    limit,
+                    "bulk budget tripped; skipping the rest of the batch"
+                );
+                out.budget_stopped = true;
+                return out;
+            }
+            Err(error) if is_rejection(&error) => Failure::Rejected(rejection_message(&error)),
+            Err(error) => {
+                tracing::warn!(%error, size, "batch failed; its articles remain unassessed");
+                continue;
+            }
+        };
+        if size == 1 {
+            let candidate = work.batch[0];
+            let message = match failure {
+                Failure::Rejected(message) => message,
+                Failure::NoItems => "returned no assessment for the article".into(),
+            };
+            resolve_single(runner, candidate, &prompt, message, &mut out).await;
+            continue;
+        }
+        match failure {
+            Failure::Rejected(message) => {
+                if !warned {
+                    tracing::warn!(
+                        provider,
+                        message = %truncate_chars(&message, REJECTION_MESSAGE_CHARS),
+                        "batch of {original} rejected by {provider}; bisecting"
+                    );
+                    warned = true;
+                }
+                push_halves(&mut stack, work.batch, work.zero_split);
+            }
+            Failure::NoItems if work.zero_split => {
+                tracing::warn!(provider, size, "batch parsed to zero items; bisecting once");
+                push_halves(&mut stack, work.batch, false);
+            }
+            Failure::NoItems => {
+                tracing::warn!(
+                    provider,
+                    size,
+                    "half batch parsed to zero items again; its articles remain unassessed"
+                );
+            }
+        }
+    }
+    out
+}
+
+fn push_halves<'c>(stack: &mut Vec<Work<'c>>, batch: Vec<&'c Candidate>, zero_split: bool) {
+    let mut left = batch;
+    let right = left.split_off(left.len() / 2);
+    stack.push(Work {
+        batch: right,
+        zero_split,
+    });
+    stack.push(Work {
+        batch: left,
+        zero_split,
+    });
+}
+
+/// A single article the bulk provider would not assess: try the editor
+/// client once, else report it as rejected.
+async fn resolve_single<T: Assessed>(
+    runner: &BatchRunner<'_, T>,
+    candidate: &Candidate,
+    prompt: &str,
+    message: String,
+    out: &mut BatchOutcome<T>,
+) {
+    let id = candidate.article.id;
+    let provider = runner.llm.provider();
+    out.rejected += 1;
+    if let Some(fallback) = runner.usable_fallback() {
+        let fallback_provider = fallback.provider();
+        match fallback.complete(prompt, runner.temperature, true).await {
+            Ok(raw) => {
+                if let Some(item) = (runner.parse)(&raw)
+                    .into_iter()
+                    .find(|item| item.article_id() == id)
+                {
+                    tracing::info!(
+                        article_id = id,
+                        provider = fallback_provider,
+                        "article rejected by {provider}; assessed on {fallback_provider} instead"
+                    );
+                    out.items.push(Scored {
+                        item,
+                        model: fallback.model.clone(),
+                    });
+                    out.recovered += 1;
+                    return;
+                }
+                tracing::warn!(
+                    article_id = id,
+                    provider = fallback_provider,
+                    "fallback returned no assessment for the rejected article"
+                );
+            }
+            Err(error) => {
+                tracing::warn!(
+                    article_id = id,
+                    provider = fallback_provider,
+                    %error,
+                    "fallback failed for the rejected article"
+                );
+            }
+        }
+    }
+    let message = truncate_chars(&message, REJECTION_MESSAGE_CHARS);
+    tracing::warn!(
+        article_id = id,
+        provider,
+        %message,
+        "article rejected by the provider; recorded so it is not retried"
+    );
+    out.rejections.push(Rejection {
+        id,
+        provider: provider.to_string(),
+        message,
+    });
+}
+
+#[cfg(test)]
+pub(crate) mod tests {
+    use super::*;
+    use crate::config::ProviderConfig;
+    use crate::curate::llm::{
+        ChatBackend, ChatCompletion, ChatRequest, MockBackend, PriceTable, UsageMeter,
+    };
+    use crate::curate::prefilter::tests::article;
+    use crate::curate::triage::{TriageItem, build_batch_prompt, parse_triage_response};
+    use crate::http::RetryPolicy;
+    use crate::types::TokenUsage;
+    use std::future::Future;
+    use std::pin::Pin;
+    use std::sync::{Arc, Mutex};
+    use std::time::Duration;
+
+    /// A backend that answers for every id it finds in the prompt unless the
+    /// prompt mentions a forbidden title, in which case it rejects the whole
+    /// request the way DeepSeek's content filter does.
+    #[derive(Debug)]
+    pub(crate) struct FilterBackend {
+        forbidden: Vec<String>,
+        /// Prompts answered with `{"articles": []}` (by request index).
+        empty_on: Vec<usize>,
+        answer: fn(&[i64]) -> String,
+        seen: Mutex<Vec<String>>,
+    }
+
+    impl FilterBackend {
+        /// Answers in the triage shape.
+        pub(crate) fn new(forbidden: &[&str], empty_on: &[usize]) -> Arc<Self> {
+            Self::with_answer(forbidden, empty_on, answer_for)
+        }
+
+        /// Answers in the deep-assessment shape.
+        pub(crate) fn deep(forbidden: &[&str]) -> Arc<Self> {
+            Self::with_answer(forbidden, &[], deep_answer_for)
+        }
+
+        fn with_answer(
+            forbidden: &[&str],
+            empty_on: &[usize],
+            answer: fn(&[i64]) -> String,
+        ) -> Arc<Self> {
+            Arc::new(Self {
+                forbidden: forbidden.iter().map(|s| s.to_string()).collect(),
+                empty_on: empty_on.to_vec(),
+                answer,
+                seen: Mutex::new(Vec::new()),
+            })
+        }
+
+        pub(crate) fn calls(&self) -> usize {
+            self.seen.lock().map(|seen| seen.len()).unwrap_or(0)
+        }
+
+        /// The prompt of the first request that carried only `id`.
+        pub(crate) fn prompts_for_single(&self, id: i64) -> String {
+            self.seen
+                .lock()
+                .ok()
+                .and_then(|seen| {
+                    seen.iter()
+                        .find(|prompt| ids_in(prompt) == vec![id])
+                        .cloned()
+                })
+                .unwrap_or_default()
+        }
+
+        /// The ids in each request, in order.
+        pub(crate) fn requests(&self) -> Vec<Vec<i64>> {
+            self.seen
+                .lock()
+                .map(|seen| seen.iter().map(|prompt| ids_in(prompt)).collect())
+                .unwrap_or_default()
+        }
+    }
+
+    fn ids_in(prompt: &str) -> Vec<i64> {
+        prompt
+            .lines()
+            .filter_map(|line| line.strip_prefix("--- id: "))
+            .filter_map(|id| id.trim().parse().ok())
+            .collect()
+    }
+
+    /// A triage answer for `ids`.
+    pub(crate) fn answer_for(ids: &[i64]) -> String {
+        let items = ids
+            .iter()
+            .map(|id| format!(r#"{{"id":{id},"interest":6,"kind":"essay","why":"fine"}}"#))
+            .collect::<Vec<_>>()
+            .join(",");
+        format!(r#"{{"articles":[{items}]}}"#)
+    }
+
+    /// A deep-assessment answer for `ids`.
+    pub(crate) fn deep_answer_for(ids: &[i64]) -> String {
+        let items = ids
+            .iter()
+            .map(|id| {
+                format!(
+                    r#"{{"id":{id},"quality":7,"fit":6,"category":"Top Stories","rationale":"fine","facets":{{"format":"analysis_essay"}}}}"#
+                )
+            })
+            .collect::<Vec<_>>()
+            .join(",");
+        format!(r#"{{"articles":[{items}]}}"#)
+    }
+
+    impl ChatBackend for FilterBackend {
+        fn complete<'a>(
+            &'a self,
+            req: ChatRequest,
+        ) -> Pin<Box<dyn Future<Output = Result<ChatCompletion, LlmError>> + Send + 'a>> {
+            Box::pin(async move {
+                let index = self.calls();
+                if let Ok(mut seen) = self.seen.lock() {
+                    seen.push(req.user.clone());
+                }
+                if self.forbidden.iter().any(|title| req.user.contains(title)) {
+                    return Err(LlmError::api(
+                        "deepseek",
+                        "400 Bad Request: {\"error\":{\"message\":\"Content Exists Risk\"}}",
+                    ));
+                }
+                let content = if self.empty_on.contains(&index) {
+                    r#"{"articles":[]}"#.to_string()
+                } else {
+                    (self.answer)(&ids_in(&req.user))
+                };
+                Ok(ChatCompletion {
+                    content,
+                    usage: TokenUsage::default(),
+                })
+            })
+        }
+    }
+
+    fn client(provider: &str, model: &str, backend: Arc<dyn ChatBackend>) -> LlmClient {
+        LlmClient::with_backend_options(
+            provider,
+            model,
+            "SYSTEM".into(),
+            None,
+            UsageMeter::with_prices(PriceTable::from(&ProviderConfig::deepseek()), 10.0),
+            backend,
+        )
+    }
+
+    fn candidates(n: i64) -> Vec<Candidate> {
+        (1..=n)
+            .map(|id| Candidate::new(article(id, &format!("Title number {id}"), 800), false))
+            .collect()
+    }
+
+    async fn run(
+        llm: &LlmClient,
+        fallback: Option<&LlmClient>,
+        candidates: &[Candidate],
+        batch_size: usize,
+    ) -> BatchOutcome<TriageItem> {
+        let runner = BatchRunner {
+            llm,
+            fallback,
+            temperature: 0.3,
+            build_prompt: &build_batch_prompt,
+            parse: &parse_triage_response,
+        };
+        let all = candidates.iter().collect::<Vec<_>>();
+        let batches = all.chunks(batch_size).map(<[&Candidate]>::to_vec).collect();
+        run_batches(&runner, batches, 4).await
+    }
+
+    fn sorted_ids(outcome: &BatchOutcome<TriageItem>) -> Vec<i64> {
+        let mut ids = outcome
+            .items
+            .iter()
+            .map(|scored| scored.item.id)
+            .collect::<Vec<_>>();
+        ids.sort_unstable();
+        ids
+    }
+
+    #[tokio::test]
+    async fn one_bad_article_in_four_costs_five_requests_and_one_rejection() {
+        let backend = FilterBackend::new(&["Title number 3"], &[]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let candidates = candidates(4);
+        let outcome = run(&llm, None, &candidates, 4).await;
+
+        // [1,2,3,4] is rejected → split. [1,2] succeeds. [3,4] is rejected →
+        // split. [3] alone is rejected → recorded. [4] succeeds. 1 + 2 + 2 = 5.
+        assert_eq!(
+            backend.requests(),
+            vec![vec![1, 2, 3, 4], vec![1, 2], vec![3, 4], vec![3], vec![4]]
+        );
+        assert_eq!(outcome.requests, 5);
+        assert_eq!(sorted_ids(&outcome), vec![1, 2, 4]);
+        assert!(outcome.items.iter().all(|s| s.model == "deepseek-v4-flash"));
+        assert_eq!(outcome.rejected, 1);
+        assert_eq!(outcome.recovered, 0);
+        assert_eq!(outcome.rejections.len(), 1);
+        let rejection = &outcome.rejections[0];
+        assert_eq!(rejection.id, 3);
+        assert_eq!(rejection.provider, "deepseek");
+        assert!(rejection.message.contains("Content Exists Risk"));
+        assert!(
+            rejection
+                .rationale()
+                .starts_with("deepseek: 400 Bad Request"),
+            "{}",
+            rejection.rationale()
+        );
+        assert!(!outcome.budget_stopped);
+    }
+
+    #[tokio::test]
+    async fn every_article_rejected_gives_every_article_a_rejection_in_2n_minus_1_requests() {
+        let titles = (1..=8)
+            .map(|id| format!("Title number {id}"))
+            .collect::<Vec<_>>();
+        let backend =
+            FilterBackend::new(&titles.iter().map(String::as_str).collect::<Vec<_>>(), &[]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let candidates = candidates(8);
+        let outcome = run(&llm, None, &candidates, 8).await;
+        assert_eq!(backend.calls(), 2 * 8 - 1);
+        assert_eq!(outcome.requests, 15);
+        assert!(outcome.items.is_empty());
+        let mut rejected = outcome
+            .rejections
+            .iter()
+            .map(|rejection| rejection.id)
+            .collect::<Vec<_>>();
+        rejected.sort_unstable();
+        assert_eq!(rejected, (1..=8).collect::<Vec<_>>());
+        assert_eq!(outcome.rejected, 8);
+    }
+
+    #[tokio::test]
+    async fn a_rejected_article_is_recovered_on_an_editor_of_another_provider() {
+        let backend = FilterBackend::new(&["Title number 2"], &[]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let editor_backend = Arc::new(MockBackend::new());
+        editor_backend.push(answer_for(&[2]), TokenUsage::default());
+        let editor = client("gemini", "gemini-3.8-flash", editor_backend.clone());
+        let candidates = candidates(2);
+        let outcome = run(&llm, Some(&editor), &candidates, 2).await;
+        assert_eq!(backend.requests(), vec![vec![1, 2], vec![1], vec![2]]);
+        assert_eq!(editor_backend.calls(), 1);
+        let prompt = editor_backend.prompts()[0].clone();
+        assert_eq!(prompt.system.as_str(), "SYSTEM", "same system prompt");
+        assert!(prompt.user.contains("--- id: 2\n"), "same article prompt");
+        assert_eq!(sorted_ids(&outcome), vec![1, 2]);
+        let recovered = outcome
+            .items
+            .iter()
+            .find(|scored| scored.item.id == 2)
+            .expect("recovered");
+        assert_eq!(recovered.model, "gemini-3.8-flash");
+        assert!(outcome.rejections.is_empty(), "no rejection row");
+        assert_eq!((outcome.rejected, outcome.recovered), (1, 1));
+
+        // The same provider name is no alternative: no fallback attempt.
+        let backend = FilterBackend::new(&["Title number 2"], &[]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let same_backend = Arc::new(MockBackend::new());
+        same_backend.push(answer_for(&[2]), TokenUsage::default());
+        let same = client("deepseek", "deepseek-v4-flash", same_backend.clone());
+        let outcome = run(&llm, Some(&same), &candidates, 2).await;
+        assert_eq!(same_backend.calls(), 0);
+        assert_eq!(outcome.rejections.len(), 1);
+        assert_eq!(outcome.recovered, 0);
+
+        // A tripped editor meter is skipped too, and a failing editor still
+        // yields a rejection with the bulk provider's message.
+        let backend = FilterBackend::new(&["Title number 2"], &[]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let tripped_backend = Arc::new(MockBackend::new());
+        let tripped = client("gemini", "gemini-3.8-flash", tripped_backend.clone());
+        tripped.meter.preload_cost(100.0);
+        let outcome = run(&llm, Some(&tripped), &candidates, 2).await;
+        assert_eq!(tripped_backend.calls(), 0);
+        assert_eq!(outcome.rejections.len(), 1);
+
+        let backend = FilterBackend::new(&["Title number 2"], &[]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let failing_backend = Arc::new(MockBackend::new());
+        failing_backend.push_llm_error(LlmError::refusal("gemini"));
+        let failing = client("gemini", "gemini-3.8-flash", failing_backend.clone());
+        let outcome = run(&llm, Some(&failing), &candidates, 2).await;
+        assert_eq!(failing_backend.calls(), 1);
+        assert_eq!(outcome.rejections.len(), 1);
+        assert_eq!(outcome.rejections[0].provider, "deepseek");
+        assert!(
+            outcome.rejections[0]
+                .message
+                .contains("Content Exists Risk")
+        );
+    }
+
+    #[tokio::test]
+    async fn zero_items_bisect_once_and_a_single_zero_is_a_rejection() {
+        // Batch of 4 parses to zero items → its two halves are sent; both fine.
+        let backend = FilterBackend::new(&[], &[0]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let candidates = candidates(4);
+        let outcome = run(&llm, None, &candidates, 4).await;
+        assert_eq!(
+            backend.requests(),
+            vec![vec![1, 2, 3, 4], vec![1, 2], vec![3, 4]]
+        );
+        assert_eq!(sorted_ids(&outcome), vec![1, 2, 3, 4]);
+        assert!(outcome.rejections.is_empty());
+
+        // A half that parses to zero items again is not split further: its
+        // articles simply stay unassessed for this run.
+        let backend = FilterBackend::new(&[], &[0, 1]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let outcome = run(&llm, None, &candidates, 4).await;
+        assert_eq!(backend.calls(), 3);
+        assert_eq!(sorted_ids(&outcome), vec![3, 4]);
+        assert!(outcome.rejections.is_empty());
+
+        // A single article that parses to zero items is a rejection.
+        let backend = FilterBackend::new(&[], &[0]);
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        let outcome = run(&llm, None, &candidates[..1], 1).await;
+        assert_eq!(backend.calls(), 1);
+        assert!(outcome.items.is_empty());
+        assert_eq!(outcome.rejections.len(), 1);
+        assert_eq!(
+            outcome.rejections[0].rationale(),
+            "deepseek: returned no assessment for the article"
+        );
+    }
+
+    #[tokio::test]
+    async fn transient_and_budget_failures_are_not_bisected() {
+        let backend = Arc::new(MockBackend::new());
+        for _ in 0..3 {
+            backend.push_llm_error(LlmError::Transient {
+                provider: "deepseek".into(),
+                message: "503".into(),
+            });
+        }
+        let llm =
+            client("deepseek", "deepseek-v4-flash", backend.clone()).with_retry(RetryPolicy {
+                max_attempts: 3,
+                base_delay: Duration::from_millis(1),
+                max_delay: Duration::from_millis(2),
+            });
+        let candidates = candidates(4);
+        // Transient errors are retried inside the client and, once exhausted,
+        // leave the batch unassessed without bisecting it.
+        let outcome = run(&llm, None, &candidates, 4).await;
+        assert_eq!(backend.calls(), 3, "three attempts, one batch");
+        assert!(outcome.items.is_empty());
+        assert!(
+            outcome.rejections.is_empty(),
+            "transient is not a rejection"
+        );
+        assert_eq!(outcome.rejected, 0);
+
+        let backend = Arc::new(MockBackend::new());
+        let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
+        llm.meter.preload_cost(100.0);
+        let outcome = run(&llm, None, &candidates, 2).await;
+        assert_eq!(backend.calls(), 0);
+        assert!(outcome.budget_stopped);
+    }
+
+    #[test]
+    fn summary_line_matches_the_documented_shape() {
+        let summary = StageSummary {
+            stage: "triage",
+            pool: 398,
+            reused: 210,
+            known_rejected: 0,
+            requested: 188,
+            batches: 8,
+            applied: 187,
+            rejected: 3,
+            recovered: 2,
+            fallback_provider: Some("gemini".into()),
+        };
+        assert_eq!(
+            summary.info_line(),
+            "triage: 398 in pool · 210 reused · 188 requested in 8 batches · 3 rejected (2 recovered on gemini)"
+        );
+        assert_eq!(summary.assessed(), 397);
+        assert_eq!(summary.rejected_total(), 1);
+        let quiet = StageSummary {
+            stage: "assess",
+            pool: 120,
+            reused: 20,
+            known_rejected: 2,
+            requested: 98,
+            batches: 13,
+            applied: 98,
+            ..StageSummary::default()
+        };
+        assert_eq!(
+            quiet.info_line(),
+            "assess: 120 in pool · 20 reused · 98 requested in 13 batches · 0 rejected · 2 known rejected"
+        );
+        assert_eq!(quiet.rejected_total(), 2);
+    }
+
+    #[test]
+    fn rejection_messages_are_short_and_single_line() {
+        let long = "x".repeat(500);
+        let cut = truncate_chars(&long, REJECTION_MESSAGE_CHARS);
+        assert_eq!(cut.chars().count(), REJECTION_MESSAGE_CHARS);
+        assert!(cut.ends_with('…'));
+        assert_eq!(truncate_chars("a\n  b", 10), "a b");
+        assert_eq!(
+            rejection_message(&LlmError::refusal("anthropic")),
+            "returned a refusal"
+        );
+        assert_eq!(
+            rejection_message(&LlmError::api("deepseek", "400: nope")),
+            "400: nope"
+        );
+    }
+}

src/curate/editorial.rs

@@ -1,34 +1,18 @@
-//! Stage C — summaries, section intros and the front page (spec §3.6).
+//! Editor-first summaries and The Brief, with per-call bulk fallback (§14).
-//!
-//! Voice: warm, literate, a little playful; never fabricates facts that are not
-//! present in the summaries.
-//!
-//! Everything here is best-effort. If the cost ceiling trips mid-way (§3.6) or a
-//! call fails, the affected article silently falls back to its own opening words
-//! and the run continues — an issue with plain excerpts is far better than no
-//! issue at all.
 
 use std::collections::BTreeMap;
 use std::fmt::Write as _;
 
-use serde::{Deserialize, Serialize};
+use futures::{StreamExt, stream};
+use serde::Deserialize;
 
-use super::llm::{LlmClient, LlmError};
+use super::llm::{LlmClient, LlmError, Llms};
-use super::{escape_html, html_to_text, text_to_paragraphs, truncate_tokens, truncate_words};
+use super::{escape_html, prompt_text, text_to_paragraphs, truncate_tokens, truncate_words};
+use crate::config::{EditorialConfig, SummaryModel};
 use crate::types::{ArticleId, Editorial, Lineup, Pick};
 
-/// Article text is truncated to roughly this many tokens per summary call (§3.6).
-pub const SUMMARY_INPUT_TOKEN_BUDGET: usize = 5000;
-/// Target length of the "From the Editor" front page, in words (§3.6).
-pub const FRONT_PAGE_WORDS: (usize, usize) = (250, 400);
-/// Words of body text used when a summary has to fall back to the excerpt.
 pub const FALLBACK_SUMMARY_WORDS: usize = 45;
 
-// ---------------------------------------------------------------------------
-// Prompts (reusable instructions here; per-call material in the user message)
-// ---------------------------------------------------------------------------
-
-/// Per-article summary instructions (§3.6 stage C).
 pub const SUMMARY_INSTRUCTIONS: &str = "\
 TASK: write the newspaper abstract for one article in today's issue.
 
@@ -57,66 +41,35 @@ excerpt.
 
 Return JSON exactly: {\"summary\": \"<two or three sentences>\"}";
 
-/// Front-page + section-intro instructions (§3.6 stage C).
+/// The Brief instructions (§14.2).
-pub const FRONT_PAGE_INSTRUCTIONS: &str = "\
+pub const BRIEF_INSTRUCTIONS: &str = r#"TASK: write "The Brief" for today's issue — the note at the top of the paper.
-TASK: write the front page of today's issue of The Daily EPUB.
 
-You are given the whole lineup: sections, headlines, sources and the abstract \
+120-200 words, one or two paragraphs. It must earn its place: if a reader skipped
-written for each article. Everything you write must come from those abstracts — \
+it, what would he miss? Name at least three of today's picks by title and say the
-you have not read the articles themselves, and inventing a fact would be worse \
+specific thing that makes each worth his time (the result, the argument, the scale,
-than saying less.
+the person). If there is a thread connecting several pieces, say it in one sentence;
+if there is not, do not invent one. If the issue is short, say why in one clause.
 
-Produce two things.
+Do not: welcome the reader, describe the weather, summarize every section, use
+"delve", "dive", "explore", "a mix of", "something for everyone", or any sentence
+that could introduce any other issue. No headings. No bullet points.
 
-1. \"from_the_editor\" — 250 to 400 words of prose addressed to the paper's one \
+Return JSON exactly: {"brief": "<the text, plain prose>"}"#;
-reader. Find the two or three threads that actually run through today's lineup \
-(a shared question, an argument between two pieces, an accidental theme) and use \
-them to guide the read: what to start with over coffee, what to save for the \
-commute, what rewards patience. Name the lead story and say why it leads. It is \
-fine — good, even — to note when a day is quiet or lopsided. Voice: warm, \
-literate, lightly playful, never breathless; a real editor writing to someone \
-whose taste he knows. No bullet lists, no headings, no emoji, 2–4 paragraphs \
-separated by a blank line.
 
-2. \"section_intros\" — for EACH section name given below, two or three \
+#[derive(Debug, Clone, PartialEq, Deserialize)]
-sentences (35–60 words) introducing what is in it today. Concrete, specific to \
+pub struct BriefResponse {
-these articles, no filler like \"a variety of interesting stories\". Use the \
-section names exactly as spelled in the lineup.
-
-Return JSON exactly:
-{\"from_the_editor\": \"<paragraphs separated by \\n\\n>\", \
-\"section_intros\": {\"<section name>\": \"<2-3 sentences>\"}}";
-
-/// The single front-page call's JSON response (§3.6).
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct FrontPageResponse {
-    /// "From the Editor", 250–400 words.
-    pub from_the_editor: String,
-    /// Section name → 2–3 sentence intro.
     #[serde(default)]
-    pub section_intros: BTreeMap<String, String>,
+    pub brief: String,
 }
 
-/// The per-article summary call's JSON response.
 #[derive(Debug, Clone, Default, Deserialize)]
 struct SummaryResponse {
     #[serde(default)]
     summary: String,
 }
 
-// ---------------------------------------------------------------------------
+fn summary_prompt(title: &str, body_html: &str, input_tokens: usize) -> String {
-// Per-article summaries
+    let body = truncate_tokens(&prompt_text(body_html), input_tokens);
-// ---------------------------------------------------------------------------
-
-/// One 2–3 sentence newspaper abstract: what it argues, why it's worth reading (§3.6).
-pub async fn summarize_article(
-    llm: &LlmClient,
-    title: &str,
-    body_html: &str,
-    temperature: f32,
-) -> Result<String, LlmError> {
-    llm.meter.check_budget()?;
-    let body = truncate_tokens(&html_to_text(body_html), SUMMARY_INPUT_TOKEN_BUDGET);
     let mut prompt = String::with_capacity(body.len() + SUMMARY_INSTRUCTIONS.len() + 256);
     prompt.push_str(SUMMARY_INSTRUCTIONS);
     let _ = write!(
@@ -129,187 +82,189 @@ pub async fn summarize_article(
             ""
         },
         if body.is_empty() {
-            "(no body text was extracted; summarize from the headline alone and say the \
+            "(no body text was extracted; summarize from the headline alone and say the full text was unavailable)"
-             full text was unavailable)"
         } else {
             &body
         }
     );
+    prompt
+}
+
+pub async fn summarize_article(
+    llm: &LlmClient,
+    title: &str,
+    body_html: &str,
+    input_tokens: usize,
+    temperature: f32,
+) -> Result<String, LlmError> {
+    let prompt = summary_prompt(title, body_html, input_tokens);
     let response: SummaryResponse = llm.complete_json(&prompt, temperature).await?;
     let summary = response.summary.trim().to_string();
     if summary.is_empty() {
-        return Err(LlmError::EmptyResponse);
+        return Err(LlmError::empty_response(llm.provider()));
     }
     Ok(summary)
 }
 
-/// Summarize every pick, returning `article_id → summary` (§3.6).
+/// `(primary, fallback)` for the summaries per `editorial.summary_model` (§14.1).
-///
+fn summary_clients(llms: &Llms, model: SummaryModel) -> (Option<&LlmClient>, Option<&LlmClient>) {
-/// Stops early and returns what it has when the cost guardrail trips (§3.6).
+    match model {
+        SummaryModel::Bulk => (llms.bulk.as_ref(), None),
+        SummaryModel::Editor => {
+            let primary = llms.editor_or_bulk();
+            let fallback = primary.and_then(|client| {
+                llms.bulk
+                    .as_ref()
+                    .filter(|bulk| bulk.provider != client.provider)
+            });
+            (primary, fallback)
+        }
+    }
+}
+
+async fn summarize_pick(
+    pick: &Pick,
+    primary: Option<&LlmClient>,
+    fallback: Option<&LlmClient>,
+    config: &EditorialConfig,
+    temperature: f32,
+) -> Option<String> {
+    let primary = primary?;
+    match summarize_article(
+        primary,
+        &pick.article.title,
+        &pick.article.content_html,
+        config.summary_input_tokens,
+        temperature,
+    )
+    .await
+    {
+        Ok(summary) => Some(summary),
+        Err(error) => {
+            let Some(fallback) = fallback else {
+                tracing::warn!(article_id = pick.article.id, %error, "summary failed; using excerpt");
+                return None;
+            };
+            tracing::warn!(article_id = pick.article.id, %error, "editor summary failed; retrying on bulk");
+            summarize_article(
+                fallback,
+                &pick.article.title,
+                &pick.article.content_html,
+                config.summary_input_tokens,
+                temperature,
+            )
+            .await
+            .map_err(|fallback_error| {
+                tracing::warn!(article_id = pick.article.id, %fallback_error, "bulk summary failed; using excerpt");
+            })
+            .ok()
+        }
+    }
+}
+
 pub async fn summarize_all(
-    llm: &LlmClient,
+    llms: &Llms,
     lineup: &Lineup,
+    config: &EditorialConfig,
     temperature: f32,
 ) -> BTreeMap<ArticleId, String> {
-    let mut out = BTreeMap::new();
+    let (primary, fallback) = summary_clients(llms, config.summary_model);
-    for (n, pick) in lineup.picks.iter().enumerate() {
+    // The summary provider's own `max_concurrent_requests` bounds the fan-out.
-        if llm.meter.budget_exceeded() {
+    let concurrency = primary
-            tracing::error!(
+        .map(|client| client.max_concurrent_requests)
-                summarized = out.len(),
+        .unwrap_or(1)
-                remaining = lineup.picks.len() - n,
+        .max(1);
-                spent_usd = llm.meter.cost_usd(),
+    stream::iter(lineup.picks.iter())
-                "COST CEILING HIT during stage C — the remaining articles fall back to \
+        .map(|pick| async move {
-                 feed excerpts as summaries"
+            let summary = summarize_pick(pick, primary, fallback, config, temperature).await;
-            );
+            (pick.article.id, summary)
-            break;
+        })
-        }
+        .buffer_unordered(concurrency)
-        match summarize_article(
+        .filter_map(|(id, summary)| async move { summary.map(|summary| (id, summary)) })
-            llm,
+        .collect()
-            &pick.article.title,
-            &pick.article.content_html,
-            temperature,
-        )
         .await
-        {
-            Ok(summary) => {
-                out.insert(pick.article.id, summary);
-            }
-            Err(LlmError::BudgetExceeded { spent, limit }) => {
-                tracing::error!(spent, limit, "COST CEILING HIT during stage C");
-                break;
-            }
-            Err(e) => {
-                tracing::warn!(
-                    article_id = pick.article.id,
-                    title = %pick.article.title,
-                    error = %e,
-                    "summary failed; falling back to the article's own opening"
-                );
-            }
-        }
-    }
-    tracing::info!(
-        summarized = out.len(),
-        picks = lineup.picks.len(),
-        "stage C summaries complete"
-    );
-    out
 }
 
-// ---------------------------------------------------------------------------
+pub fn build_brief_prompt(lineup: &Lineup, summaries: &BTreeMap<ArticleId, String>) -> String {
-// Front page
-// ---------------------------------------------------------------------------
-
-/// The single front-page + section-intro call (§3.6).
-pub async fn front_page(
-    llm: &LlmClient,
-    lineup: &Lineup,
-    summaries: &BTreeMap<ArticleId, String>,
-    temperature: f32,
-) -> Result<FrontPageResponse, LlmError> {
-    llm.meter.check_budget()?;
-    let prompt = build_front_page_prompt(lineup, summaries);
-    tracing::debug!(
-        approx_tokens = super::approx_tokens(&prompt),
-        "stage C front-page request"
-    );
-    let mut response: FrontPageResponse = llm.complete_json(&prompt, temperature).await?;
-    response.from_the_editor = response.from_the_editor.trim().to_string();
-    if response.from_the_editor.is_empty() {
-        return Err(LlmError::EmptyResponse);
-    }
-    // Keep only intros for sections that actually exist in the issue.
-    response
-        .section_intros
-        .retain(|name, text| lineup.section_order.contains(name) && !text.trim().is_empty());
-    Ok(response)
-}
-
-/// Render the front-page user prompt: the whole lineup with its abstracts (§3.6).
-pub fn build_front_page_prompt(lineup: &Lineup, summaries: &BTreeMap<ArticleId, String>) -> String {
     let mut prompt = String::with_capacity(4096);
-    prompt.push_str(FRONT_PAGE_INSTRUCTIONS);
+    prompt.push_str(BRIEF_INSTRUCTIONS);
-    let minutes: i64 = lineup
-        .picks
-        .iter()
-        .map(|p| p.article.reading_minutes())
-        .sum();
     let _ = write!(
         prompt,
-        "\n\nISSUE: {} · {} articles across {} sections · about {} minutes of reading\n\
+        "\n\nISSUE: {} · {} articles\n\nLINEUP\n",
-         SECTIONS, in order: {}\n\nLINEUP\n",
         lineup.date,
-        lineup.picks.len(),
+        lineup.picks.len()
-        lineup.section_order.len(),
-        minutes,
-        lineup.section_order.join(" | ")
     );
     for section in &lineup.section_order {
-        let _ = write!(prompt, "\n## {section}\n");
+        let _ = writeln!(prompt, "\n## {section}");
         for pick in lineup.section_picks(section) {
-            let _ = write!(prompt, "{}", render_pick(pick, summaries));
+            let quality = pick
+                .llm
+                .as_ref()
+                .map(|assessment| format!("{:.1}", assessment.quality))
+                .unwrap_or_else(|| "unassessed".into());
+            let fit = pick
+                .llm
+                .as_ref()
+                .map(|assessment| format!("{:.1}", assessment.fit))
+                .unwrap_or_else(|| "unassessed".into());
+            let summary = summaries
+                .get(&pick.article.id)
+                .cloned()
+                .unwrap_or_else(|| excerpt_summary(pick));
+            let _ = writeln!(
+                prompt,
+                "- {}\n  feed: {}\n  why: {}\n  quality: {}\n  fit: {}\n  summary: {}",
+                pick.article.title.trim(),
+                pick.article.feed_title.trim(),
+                pick.why.as_deref().unwrap_or("not supplied"),
+                quality,
+                fit,
+                summary
+            );
         }
     }
     prompt
 }
 
-fn render_pick(pick: &Pick, summaries: &BTreeMap<ArticleId, String>) -> String {
+pub async fn brief(
-    let a = &pick.article;
+    llms: &Llms,
-    let mut block = String::with_capacity(400);
+    lineup: &Lineup,
-    let _ = writeln!(
+    summaries: &BTreeMap<ArticleId, String>,
-        block,
+    temperature: f32,
-        "\n- {}{}",
+) -> Result<String, LlmError> {
-        a.title.trim(),
+    let prompt = build_brief_prompt(lineup, summaries);
-        if pick.is_lead { "  [LEAD STORY]" } else { "" }
+    let Some(primary) = llms.editor_or_bulk() else {
-    );
+        return Err(LlmError::api("editorial", "no provider configured"));
-    let _ = writeln!(
+    };
-        block,
+    let response = match primary
-        "  source: {} · {} words (~{} min){}",
+        .complete_json::<BriefResponse>(&prompt, temperature)
-        if a.feed_title.is_empty() {
+        .await
-            "unknown"
+    {
-        } else {
+        Ok(response) => response,
-            a.feed_title.trim()
+        Err(error) => {
-        },
+            let Some(fallback) = llms
-        a.word_count,
+                .bulk
-        a.reading_minutes(),
+                .as_ref()
-        social_note(pick)
+                .filter(|bulk| bulk.provider != primary.provider)
-    );
+            else {
-    let abstract_text = summaries
+                return Err(error);
-        .get(&a.id)
+            };
-        .cloned()
+            tracing::warn!(%error, "brief failed on editor; retrying on bulk");
-        .unwrap_or_else(|| excerpt_summary(pick));
+            fallback
-    let _ = writeln!(block, "  abstract: {abstract_text}");
+                .complete_json::<BriefResponse>(&prompt, temperature)
-    block
+                .await?
-}
+        }
-
+    };
-fn social_note(pick: &Pick) -> String {
+    let brief = response.brief.trim().to_string();
-    if pick.article.social.is_empty() {
+    if brief.is_empty() {
-        return String::new();
+        return Err(LlmError::empty_response(primary.provider()));
     }
-    let parts: Vec<String> = pick
+    Ok(brief)
-        .article
-        .social
-        .iter()
-        .map(|s| {
-            format!(
-                "{} {} pts/{} comments",
-                s.source.display_name(),
-                s.score,
-                s.num_comments
-            )
-        })
-        .collect();
-    format!(" · {}", parts.join(", "))
 }
 
-// ---------------------------------------------------------------------------
-// Fallbacks (§3.6, notes §6)
-// ---------------------------------------------------------------------------
-
-/// The article's own opening words, used when no LLM summary exists (§3.6).
 pub fn excerpt_summary(pick: &Pick) -> String {
     let text = truncate_words(
-        &html_to_text(&pick.article.content_html),
+        &prompt_text(&pick.article.content_html),
         FALLBACK_SUMMARY_WORDS,
     );
     if text.is_empty() {
@@ -326,17 +281,14 @@ pub fn excerpt_summary(pick: &Pick) -> String {
     }
 }
 
-/// A plain, factual front page used when the model is unavailable (§3.6, notes §6).
 pub fn fallback_front_page_html(lineup: &Lineup) -> String {
     let minutes: i64 = lineup
         .picks
         .iter()
-        .map(|p| p.article.reading_minutes())
+        .map(|pick| pick.article.reading_minutes())
         .sum();
     let mut text = format!(
-        "Today's issue collects {} articles across {} sections — about {} minutes of \
+        "Today's issue collects {} articles across {} sections — about {} minutes of reading. Editorial notes are unavailable for this issue, so the lineup speaks for itself.",
-         reading. Editorial notes are unavailable for this issue, so the lineup speaks \
-         for itself.",
         lineup.picks.len(),
         lineup.section_order.len(),
         minutes
@@ -346,29 +298,15 @@ pub fn fallback_front_page_html(lineup: &Lineup) -> String {
             text,
             "\n\nLeading today: “{}” ({}).",
             lead.article.title.trim(),
-            if lead.article.feed_title.is_empty() {
+            lead.article.feed_title.trim()
-                "source unknown"
-            } else {
-                lead.article.feed_title.trim()
-            }
-        );
-    }
-    if !lineup.section_order.is_empty() {
-        let _ = write!(
-            text,
-            "\n\nIn this issue: {}.",
-            lineup.section_order.join(", ")
         );
     }
     text_to_paragraphs(&text)
 }
 
-/// `--skip-llm` / budget-exceeded fallback: feed excerpts stand in for summaries
-/// and the front page is a plain stats line (§3.6, notes §6).
 pub fn fallback_editorial(lineup: &Lineup) -> Editorial {
     Editorial {
         front_page_html: fallback_front_page_html(lineup),
-        section_intros: BTreeMap::new(),
         summaries: lineup
             .picks
             .iter()
@@ -377,54 +315,62 @@ pub fn fallback_editorial(lineup: &Lineup) -> Editorial {
     }
 }
 
-// ---------------------------------------------------------------------------
+/// Wall-clock milliseconds of the two editorial calls, for the run report's
-// Stage driver
+/// `summaries` and `brief` stage timings (§15.4).
-// ---------------------------------------------------------------------------
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct EditorialTimings {
+    pub summaries_ms: i64,
+    pub brief_ms: i64,
+}
 
-/// Stage C end to end: summaries, then one front-page call, with excerpts filling
+pub async fn run(
-/// every gap (§3.6).
+    llms: &Llms,
-pub async fn run(llm: &LlmClient, lineup: &Lineup, temperature: f32) -> Editorial {
+    lineup: &Lineup,
+    config: &EditorialConfig,
+    temperature: f32,
+) -> Editorial {
+    run_timed(llms, lineup, config, temperature).await.0
+}
+
+/// [`run`], also reporting how long the summaries and the brief took.
+pub async fn run_timed(
+    llms: &Llms,
+    lineup: &Lineup,
+    config: &EditorialConfig,
+    temperature: f32,
+) -> (Editorial, EditorialTimings) {
     if lineup.picks.is_empty() {
-        return fallback_editorial(lineup);
+        return (fallback_editorial(lineup), EditorialTimings::default());
+    }
+    let started = std::time::Instant::now();
+    let mut summaries = summarize_all(llms, lineup, config, temperature).await;
+    for pick in &lineup.picks {
+        summaries
+            .entry(pick.article.id)
+            .or_insert_with(|| excerpt_summary(pick));
     }
-
+    let summaries_ms = started.elapsed().as_millis() as i64;
-    let mut summaries = summarize_all(llm, lineup, temperature).await;
+    let started = std::time::Instant::now();
-    let missing: Vec<&Pick> = lineup
+    let front_page_html = match brief(llms, lineup, &summaries, temperature).await {
-        .picks
+        Ok(text) => text_to_paragraphs(&text),
-        .iter()
+        Err(error) => {
-        .filter(|p| !summaries.contains_key(&p.article.id))
+            tracing::warn!(%error, "brief failed; using fallback front page");
-        .collect();
+            fallback_front_page_html(lineup)
-    if !missing.is_empty() {
-        tracing::warn!(
-            count = missing.len(),
-            "using feed excerpts as summaries for articles the model did not cover"
-        );
-        for pick in missing {
-            summaries.insert(pick.article.id, excerpt_summary(pick));
         }
-    }
+    };
-
+    let brief_ms = started.elapsed().as_millis() as i64;
-    let (front_page_html, section_intros) =
+    (
-        match front_page(llm, lineup, &summaries, temperature).await {
+        Editorial {
-            Ok(response) => (
+            front_page_html,
-                text_to_paragraphs(&response.from_the_editor),
+            summaries,
-                response.section_intros,
+        },
-            ),
+        EditorialTimings {
-            Err(e) => {
+            summaries_ms,
-                tracing::error!(error = %e,
+            brief_ms,
-                    "front-page generation failed; using the plain front page");
+        },
-                (fallback_front_page_html(lineup), BTreeMap::new())
+    )
-            }
-        };
-
-    Editorial {
-        front_page_html,
-        section_intros,
-        summaries,
-    }
 }
 
-/// Escape-and-wrap helper for callers rendering a summary straight into XHTML.
 pub fn summary_to_html(summary: &str) -> String {
     format!("<p>{}</p>", escape_html(summary.trim()))
 }
@@ -432,15 +378,15 @@ pub fn summary_to_html(summary: &str) -> String {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::config::DeepseekConfig;
+    use crate::config::ProviderConfig;
-    use crate::curate::llm::{MockBackend, UsageMeter};
+    use crate::curate::llm::{ChatBackend, MockBackend, PriceTable, UsageMeter};
     use crate::curate::prefilter::tests::article;
     use crate::types::TokenUsage;
     use std::sync::Arc;
 
-    const FRONT_PAGE_FIXTURE: &str = include_str!(concat!(
+    const BRIEF_FIXTURE: &str = include_str!(concat!(
         env!("CARGO_MANIFEST_DIR"),
-        "/tests/fixtures/deepseek_front_page.json"
+        "/tests/fixtures/claude_brief.json"
     ));
 
     fn pick(id: i64, title: &str, section: &str, is_lead: bool) -> Pick {
@@ -451,6 +397,7 @@ mod tests {
             section: section.into(),
             position: 1,
             is_lead,
+            why: Some(format!("the {title} piece you'd argue with")),
             summary: None,
             llm: None,
             discussion: None,
@@ -468,15 +415,40 @@ mod tests {
         }
     }
 
-    fn client(backend: Arc<MockBackend>, limit: f64) -> LlmClient {
+    fn mock(provider: &'static str, backend: Arc<MockBackend>, limit: f64) -> LlmClient {
-        LlmClient::with_backend(
+        let prices = if provider == "anthropic" {
-            "deepseek-v4-flash",
+            PriceTable::from(&ProviderConfig::anthropic())
+        } else {
+            PriceTable::from(&ProviderConfig::deepseek())
+        };
+        LlmClient::with_backend_options(
+            provider,
+            "model",
             "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), limit),
+            None,
-            backend,
+            UsageMeter::with_prices(prices, limit),
+            backend as Arc<dyn ChatBackend>,
         )
     }
 
+    fn bulk_only(backend: Arc<MockBackend>, limit: f64) -> Llms {
+        Llms {
+            bulk: Some(mock("deepseek", backend, limit)),
+            editor: None,
+        }
+    }
+
+    fn editor_and_bulk(editor: Arc<MockBackend>, bulk: Arc<MockBackend>) -> Llms {
+        Llms {
+            bulk: Some(mock("deepseek", bulk, 2.0)),
+            editor: Some(mock("anthropic", editor, 3.0)),
+        }
+    }
+
+    fn config() -> EditorialConfig {
+        EditorialConfig::default()
+    }
+
     #[tokio::test]
     async fn summary_prompt_carries_headline_and_truncated_body() {
         let backend = Arc::new(MockBackend::new());
@@ -484,9 +456,9 @@ mod tests {
             r#"{"summary": "A team moves 40TB of relational data off Postgres and documents every rollback."}"#,
             TokenUsage::default(),
         );
-        let llm = client(Arc::clone(&backend), 2.0);
+        let llm = mock("deepseek", Arc::clone(&backend), 2.0);
         let body = format!("<p>{}</p>", "word ".repeat(20_000));
-        let summary = summarize_article(&llm, "Migrating 40TB", &body, 0.8)
+        let summary = summarize_article(&llm, "Migrating 40TB", &body, 3_000, 0.8)
             .await
             .expect("summary");
         assert!(summary.starts_with("A team moves 40TB"));
@@ -495,60 +467,132 @@ mod tests {
         assert!(prompt.starts_with(SUMMARY_INSTRUCTIONS));
         assert!(prompt.contains("HEADLINE: Migrating 40TB"));
         assert!(prompt.contains("(truncated for length)"));
-        // ~5k tokens ≈ 20k characters of body, not the full 100k.
+        // 3k tokens ≈ 12k characters of body, not the full 100k.
-        assert!(prompt.len() < 26_000, "prompt was {} bytes", prompt.len());
+        assert!(prompt.len() < 16_000, "prompt was {} bytes", prompt.len());
     }
 
     #[tokio::test]
-    async fn front_page_parses_and_filters_unknown_sections() {
+    async fn the_brief_is_parsed_and_rendered() {
         let backend = Arc::new(MockBackend::new());
-        backend.push(FRONT_PAGE_FIXTURE, TokenUsage::default());
+        backend.push(BRIEF_FIXTURE, TokenUsage::default());
-        let llm = client(Arc::clone(&backend), 2.0);
+        let llms = bulk_only(Arc::clone(&backend), 2.0);
         let lineup = lineup();
         let summaries = BTreeMap::from([
             (1, "A migration story with numbers.".to_string()),
             (2, "Transit data, charted.".to_string()),
         ]);
 
-        let response = front_page(&llm, &lineup, &summaries, 0.8)
+        let text = brief(&llms, &lineup, &summaries, 0.8).await.expect("brief");
-            .await
+        assert!(text.split_whitespace().count() > 100);
-            .expect("front page");
+        assert!(text.contains("Migrating 40TB off Postgres"));
-        assert!(response.from_the_editor.split_whitespace().count() > 40);
-        assert_eq!(response.section_intros.len(), 2);
-        assert!(response.section_intros.contains_key("Top Stories"));
-        assert!(
-            !response.section_intros.contains_key("Niche Corner"),
-            "intros for absent sections are dropped"
-        );
 
         let prompt = &backend.prompts()[0].user;
-        assert!(prompt.starts_with(FRONT_PAGE_INSTRUCTIONS));
+        assert!(prompt.starts_with(BRIEF_INSTRUCTIONS));
         assert!(prompt.contains("## Top Stories"));
-        assert!(prompt.contains("[LEAD STORY]"));
+        assert!(prompt.contains("## Boston & Local"));
-        assert!(prompt.contains("abstract: A migration story with numbers."));
+        assert!(prompt.contains("- Migrating 40TB off Postgres"));
+        assert!(prompt.contains("why: the Migrating 40TB off Postgres piece you'd argue with"));
+        assert!(prompt.contains("summary: A migration story with numbers."));
+        assert!(prompt.contains("quality: unassessed"));
+        assert!(prompt.contains("fit: unassessed"));
         assert!(prompt.contains("2026-08-15"));
+        assert!(
+            !prompt.contains("section_intros"),
+            "section intros are gone"
+        );
     }
 
     #[tokio::test]
-    async fn full_stage_c_produces_summaries_intros_and_front_page() {
+    async fn full_stage_c_produces_summaries_and_the_brief() {
         let backend = Arc::new(MockBackend::new());
         backend.push(r#"{"summary": "First abstract."}"#, TokenUsage::default());
         backend.push(r#"{"summary": "Second abstract."}"#, TokenUsage::default());
-        backend.push(FRONT_PAGE_FIXTURE, TokenUsage::default());
+        backend.push(BRIEF_FIXTURE, TokenUsage::default());
-        let llm = client(Arc::clone(&backend), 2.0);
+        let llms = bulk_only(Arc::clone(&backend), 2.0);
 
-        let editorial = run(&llm, &lineup(), 0.8).await;
+        let editorial = run(&llms, &lineup(), &config(), 0.8).await;
-        assert_eq!(
+        assert_eq!(backend.calls(), 3, "one call per article plus the brief");
-            backend.calls(),
-            3,
-            "one call per article plus the front page"
-        );
         assert_eq!(editorial.summaries.len(), 2);
         assert_eq!(editorial.summaries[&1], "First abstract.");
         assert!(editorial.front_page_html.starts_with("<p>"));
         assert!(editorial.front_page_html.contains("</p>"));
+        assert!(
+            editorial
+                .front_page_html
+                .contains("Migrating 40TB off Postgres")
+        );
         assert!(!editorial.front_page_html.contains("<script"));
-        assert_eq!(editorial.section_intros.len(), 2);
+    }
+
+    #[tokio::test]
+    async fn summaries_run_on_the_editor_and_fall_back_per_article() {
+        let editor = Arc::new(MockBackend::new());
+        editor.push(
+            r#"{"summary": "Opus wrote this one."}"#,
+            TokenUsage::default(),
+        );
+        editor.push_llm_error(LlmError::refusal("anthropic"));
+        editor.push(BRIEF_FIXTURE, TokenUsage::default());
+        let bulk = Arc::new(MockBackend::new());
+        bulk.push(
+            r#"{"summary": "DeepSeek covered the refusal."}"#,
+            TokenUsage::default(),
+        );
+        let llms = editor_and_bulk(Arc::clone(&editor), Arc::clone(&bulk));
+
+        let editorial = run(&llms, &lineup(), &config(), 0.8).await;
+        assert_eq!(
+            editor.calls(),
+            3,
+            "two summaries and the brief on the editor"
+        );
+        assert_eq!(bulk.calls(), 1, "only the refused summary went to bulk");
+        assert_eq!(editorial.summaries[&1], "Opus wrote this one.");
+        assert_eq!(editorial.summaries[&2], "DeepSeek covered the refusal.");
+        assert_eq!(
+            editor.prompts()[1].user,
+            bulk.prompts()[0].user,
+            "the bulk client gets the identical summary prompt"
+        );
+        assert!(
+            editorial
+                .front_page_html
+                .contains("Migrating 40TB off Postgres")
+        );
+    }
+
+    #[tokio::test]
+    async fn the_brief_falls_back_to_bulk_with_the_same_prompt() {
+        let editor = Arc::new(MockBackend::new());
+        editor.push_error("500 opus is down");
+        let bulk = Arc::new(MockBackend::new());
+        bulk.push(BRIEF_FIXTURE, TokenUsage::default());
+        let llms = editor_and_bulk(Arc::clone(&editor), Arc::clone(&bulk));
+
+        let text = brief(&llms, &lineup(), &BTreeMap::new(), 0.8)
+            .await
+            .expect("bulk brief");
+        assert!(text.contains("Migrating 40TB off Postgres"));
+        assert_eq!(editor.prompts()[0].user, bulk.prompts()[0].user);
+    }
+
+    #[tokio::test]
+    async fn summary_model_bulk_skips_the_editor_for_summaries() {
+        let editor = Arc::new(MockBackend::new());
+        editor.push(BRIEF_FIXTURE, TokenUsage::default());
+        let bulk = Arc::new(MockBackend::new());
+        bulk.push(r#"{"summary": "First abstract."}"#, TokenUsage::default());
+        bulk.push(r#"{"summary": "Second abstract."}"#, TokenUsage::default());
+        let llms = editor_and_bulk(Arc::clone(&editor), Arc::clone(&bulk));
+        let config = EditorialConfig {
+            summary_model: SummaryModel::Bulk,
+            ..EditorialConfig::default()
+        };
+
+        let editorial = run(&llms, &lineup(), &config, 0.8).await;
+        assert_eq!(bulk.calls(), 2);
+        assert_eq!(editor.calls(), 1, "the brief still runs on the editor");
+        assert_eq!(editorial.summaries[&2], "Second abstract.");
     }
 
     #[tokio::test]
@@ -560,14 +604,15 @@ mod tests {
             TokenUsage {
                 input_tokens: 1_000_000,
                 cached_tokens: 0,
+                cache_write_tokens: 0,
                 output_tokens: 0,
             },
         );
-        let llm = client(Arc::clone(&backend), 0.05);
+        let llms = bulk_only(Arc::clone(&backend), 0.05);
 
-        let editorial = run(&llm, &lineup(), 0.8).await;
+        let editorial = run(&llms, &lineup(), &config(), 0.8).await;
         assert_eq!(backend.calls(), 1, "no further calls after the ceiling");
-        assert!(llm.meter.budget_exceeded());
+        assert!(llms.bulk.as_ref().expect("bulk").meter.budget_exceeded());
         assert_eq!(
             editorial.summaries.len(),
             2,
@@ -581,7 +626,6 @@ mod tests {
         );
         // The front page degraded to the plain version.
         assert!(editorial.front_page_html.contains("2 articles"));
-        assert!(editorial.section_intros.is_empty());
     }
 
     #[tokio::test]
@@ -589,28 +633,30 @@ mod tests {
         let backend = Arc::new(MockBackend::new());
         backend.push_error("400 bad request");
         backend.push(r#"{"summary": "Second abstract."}"#, TokenUsage::default());
-        backend.push_error("500 front page exploded");
+        backend.push_error("500 brief exploded");
-        let llm = client(Arc::clone(&backend), 2.0);
+        let llms = bulk_only(Arc::clone(&backend), 2.0);
 
-        let editorial = run(&llm, &lineup(), 0.8).await;
+        let editorial = run(&llms, &lineup(), &config(), 0.8).await;
         assert_eq!(editorial.summaries.len(), 2);
         assert!(editorial.summaries[&1].contains("opens with a specific"));
         assert_eq!(editorial.summaries[&2], "Second abstract.");
         assert!(editorial.front_page_html.contains("Leading today"));
     }
 
+    #[tokio::test]
+    async fn no_provider_means_the_fallback_editorial() {
+        let editorial = run(&Llms::default(), &lineup(), &config(), 0.8).await;
+        assert_eq!(editorial.summaries.len(), 2);
+        assert!(editorial.front_page_html.contains("2 articles"));
+    }
+
     #[test]
     fn fallback_editorial_covers_every_pick() {
         let lineup = lineup();
         let editorial = fallback_editorial(&lineup);
         assert_eq!(editorial.summaries.len(), lineup.picks.len());
-        assert!(editorial.section_intros.is_empty());
         assert!(editorial.front_page_html.contains("2 articles"));
-        assert!(
+        assert!(editorial.front_page_html.contains("2 sections"));
-            editorial
-                .front_page_html
-                .contains("Top Stories, Boston &amp; Local")
-        );
         assert!(editorial.front_page_html.starts_with("<p>"));
 
         // An empty lineup is still a valid editorial.

src/curate/mod.rs

@@ -1,154 +1,153 @@
-//! Curation pipeline: pre-filter → LLM scoring → selection → editorial (spec §3.5, §3.6).
+//! Personalized curation: signals → triage → admission → assessment → editor.
 //!
 //! ```text
-//! ~400 articles ─prefilter─▶ ~120 candidates ─stage A─▶ scored ─stage B─▶ lineup ─stage C─▶ editorial
+//! ~400 eligible ─triage─▶ admission (120) ─assess/rank─▶ shortlist (60) ─editor─▶ editorial
 //! ```
 //!
 //! [`Curator`] is the thin orchestration layer the `generate` pipeline calls; the
 //! interesting logic lives in the stage modules. Every stage is safe to run with
-//! `llm == None` (`--skip-llm`): the prefilter order stands in for selection and
+//! no provider at all (`--skip-llm`): the cheap-signal blend stands in for selection
-//! feed excerpts stand in for summaries (notes §6).
+//! and feed excerpts stand in for summaries (notes §6). Scoring runs on the bulk
+//! client; selection and editorial on the editor with per-call bulk fallback.
 
+pub mod admit;
+pub mod assess;
+pub mod batch;
+pub mod editor;
 pub mod editorial;
+pub mod embedding;
 pub mod llm;
 pub mod prefilter;
 pub mod profile;
-pub mod score;
+pub mod rank;
-pub mod select;
+pub mod signals;
+pub mod telemetry;
+pub mod triage;
 
-use jiff::civil::Date;
+use jiff::{Timestamp, civil::Date};
 
 use crate::config::Config;
 use crate::db::Db;
-use crate::types::{Article, Editorial, Lineup, ScoredArticle};
+use crate::types::{Candidate, Editorial, Lineup};
 
-/// Runs the three curation stages against one day's articles (§3.5, §3.6).
+/// Runs the LLM curation stages against one day's articles (§12, §13, §14).
 pub struct Curator {
     pub config: Config,
     pub db: Db,
-    pub llm: Option<llm::LlmClient>,
+    pub llms: llm::Llms,
 }
 
 impl Curator {
-    /// `llm == None` corresponds to `--skip-llm`: prefilter order is used for
+    /// An empty [`llm::Llms`] corresponds to `--skip-llm`: cheap-signal order is
-    /// selection and feed excerpts stand in for summaries (notes §6).
+    /// used for selection and feed excerpts stand in for summaries (notes §6).
-    pub fn new(config: Config, db: Db, llm: Option<llm::LlmClient>) -> Self {
+    /// With only `bulk`, every editor call runs on the bulk provider (§4.2).
-        Self { config, db, llm }
+    pub fn new(config: Config, db: Db, llms: llm::Llms) -> Self {
+        Self { config, db, llms }
     }
 
-    /// Heuristic pre-filter: 300–500 articles → `prefilter_keep` (§3.5).
+    /// Deep assessment of the admitted set on the bulk client (§12.1), reusing
+    /// cached `article_assessments` rows within `assessment_reuse_days`.
     ///
-    /// Also persists each candidate's `prefilter_score` for the day so that a
+    /// A no-op under `--skip-llm`: like triage, nothing is read or written and
-    /// re-run of the same date is idempotent (notes §12).
+    /// utility falls back to the present signals (§12.3, §17). When the bulk
-    pub async fn prefilter(
+    /// provider is down or its budget trips, cached rows are still reused and
+    /// the failed batches simply stay unassessed. A batch the provider's
+    /// content filter rejects is bisected, and a rejected single article is
+    /// retried on the editor when that is another provider (`curate::batch`).
+    pub async fn assess(
         &self,
-        articles: Vec<Article>,
+        candidates: &mut [Candidate],
-        date: Date,
+        rescore: bool,
-    ) -> anyhow::Result<Vec<ScoredArticle>> {
+        profile_version: Option<i64>,
-        let span = tracing::info_span!("prefilter", articles = articles.len());
+        assessed_at: Timestamp,
-        let _guard = span.enter();
+    ) -> anyhow::Result<batch::StageSummary> {
-
+        let Some(bulk) = self.llms.bulk.as_ref() else {
-        let ctx = prefilter::PrefilterContext::load(&self.db, date).await?;
+            tracing::info!("--skip-llm: deep assessment skipped");
-        let candidates = prefilter::run(articles, &ctx, &self.config);
+            return Ok(batch::StageSummary::default());
-        for candidate in &candidates {
-            if candidate.article.id == 0 {
-                continue; // not persisted yet (dry run over synthetic articles)
-            }
-            if let Err(e) = self
-                .db
-                .upsert_score(
-                    candidate.article.id,
-                    date,
-                    Some(candidate.prefilter_score),
-                    None,
-                )
-                .await
-            {
-                tracing::warn!(article_id = candidate.article.id, error = %e,
-                    "could not persist the prefilter score");
-            }
-        }
-        Ok(candidates)
-    }
-
-    /// Stage A: batched LLM scoring of the surviving candidates (§3.6).
-    ///
-    /// A no-op under `--skip-llm`. Scores are persisted per `(article, date)`.
-    pub async fn score(&self, candidates: &mut [ScoredArticle], date: Date) -> anyhow::Result<()> {
-        let Some(llm) = self.llm.as_ref() else {
-            tracing::info!("--skip-llm: stage A scoring skipped");
-            return Ok(());
         };
-        let span = tracing::info_span!("llm_score", candidates = candidates.len());
+        let admitted = candidates
+            .iter()
+            .filter(|candidate| candidate.stage == "admitted")
+            .count();
+        let span = tracing::info_span!("llm_assess", candidates = admitted);
         let _guard = span.enter();
-
+        assess::run(
-        let scored = score::score_all(
+            &self.db,
-            llm,
+            Some(bulk),
+            self.llms.editor.as_ref(),
+            &bulk.model,
             candidates,
-            self.config.deepseek.score_batch_size,
+            self.config.llm.deep_batch_size,
+            bulk.max_concurrent_requests,
+            self.config.curation.ranking.assessment_reuse_days,
+            rescore,
+            profile_version,
+            assessed_at,
+            self.config.llm.score_temperature,
             &self.config.curation.sections,
-            self.config.deepseek.score_temperature,
         )
-        .await?;
+        .await
-        tracing::info!(scored, total = candidates.len(), "stage A complete");
-
-        for candidate in candidates.iter() {
-            if candidate.article.id == 0 {
-                continue;
-            }
-            if let Some(llm_score) = candidate.llm.as_ref()
-                && let Err(e) = self
-                    .db
-                    .upsert_score(candidate.article.id, date, None, Some(llm_score))
-                    .await
-            {
-                tracing::warn!(article_id = candidate.article.id, error = %e,
-                    "could not persist the llm score");
-            }
-        }
-        Ok(())
     }
 
-    /// Stage B: single-call lineup selection into sections (§3.6).
+    /// The editor: one call that assembles the issue from the shortlist (§13).
-    pub async fn select(
+    pub async fn select(&self, candidates: Vec<Candidate>, date: Date) -> anyhow::Result<Lineup> {
-        &self,
-        candidates: Vec<ScoredArticle>,
-        date: Date,
-    ) -> anyhow::Result<Lineup> {
         let sections = &self.config.curation.sections;
-        let target = self.config.target_article_count;
+        let soft_target = self.config.target_article_count;
-        let Some(llm) = self.llm.as_ref() else {
+        let hard_max = self.config.curation.max_article_count;
-            tracing::info!("--skip-llm: selecting by prefilter order");
+        let span = tracing::info_span!("llm_editor", candidates = candidates.len());
-            return Ok(select::select_without_llm(
-                candidates, sections, target, date,
-            ));
-        };
-        let span = tracing::info_span!("llm_select", candidates = candidates.len());
         let _guard = span.enter();
-
+        match editor::select(
-        match select::select(llm, candidates.clone(), sections, target, date).await {
+            &self.llms,
+            candidates.clone(),
+            sections,
+            soft_target,
+            hard_max,
+            date,
+        )
+        .await
+        {
             Ok(lineup) => Ok(lineup),
-            Err(e) => {
+            Err(error) => {
-                tracing::error!(error = %e,
+                tracing::error!(%error, "editor and bulk fallback failed; selecting heuristically");
-                    "stage B selection failed; falling back to prefilter order");
+                Ok(editor::select_without_llm(
-                Ok(select::select_without_llm(
+                    candidates,
-                    candidates, sections, target, date,
+                    sections,
+                    soft_target,
+                    hard_max,
+                    date,
                 ))
             }
         }
     }
 
-    /// Stage C: per-article summaries, section intros and the front page (§3.6).
+    /// Stage C: per-article summaries and the Brief (§14).
     ///
     /// Never fails the run: a budget trip or an API error degrades to excerpts.
     pub async fn editorial(&self, lineup: &Lineup) -> anyhow::Result<Editorial> {
-        let Some(llm) = self.llm.as_ref() else {
+        Ok(self.editorial_timed(lineup).await?.0)
+    }
+
+    /// [`Self::editorial`] plus the `summaries` / `brief` stage timings (§15.4).
+    pub async fn editorial_timed(
+        &self,
+        lineup: &Lineup,
+    ) -> anyhow::Result<(Editorial, editorial::EditorialTimings)> {
+        if self.llms.editor_or_bulk().is_none() {
             tracing::info!("--skip-llm: using feed excerpts as summaries");
-            return Ok(editorial::fallback_editorial(lineup));
+            return Ok((
-        };
+                editorial::fallback_editorial(lineup),
+                editorial::EditorialTimings::default(),
+            ));
+        }
         let span = tracing::info_span!("llm_editorial", picks = lineup.picks.len());
         let _guard = span.enter();
-        Ok(editorial::run(llm, lineup, self.config.deepseek.editorial_temperature).await)
+        Ok(editorial::run_timed(
+            &self.llms,
+            lineup,
+            &self.config.editorial,
+            self.config.llm.editorial_temperature,
+        )
+        .await)
     }
 }
 
@@ -157,14 +156,18 @@ impl Curator {
 // ---------------------------------------------------------------------------
 
 /// Crude token estimate: DeepSeek averages ~4 characters per token for English
-/// prose. Only used to size prompt budgets (§3.6 stage C).
+/// prose. Only used to size prompt budgets.
 pub fn approx_tokens(text: &str) -> usize {
     text.len().div_ceil(4)
 }
 
 /// Strip markup and collapse whitespace, so article bodies can go into prompts
 /// as plain text (cheaper and less confusing for the model than raw HTML).
-pub fn html_to_text(html: &str) -> String {
+///
+/// Deliberately not [`crate::html::html_to_text`]: this one collapses runs of
+/// whitespace and never builds a DOM, because it runs over every candidate
+/// body on every run and only has to be good enough to size a prompt.
+pub fn prompt_text(html: &str) -> String {
     /// Does `tail` open the named element, i.e. `<name` or `</name`?
     fn opens(tail: &str, name: &str) -> bool {
         let bytes = tail.as_bytes();
@@ -220,7 +223,7 @@ pub fn truncate_words(text: &str, max_words: usize) -> String {
     out
 }
 
-/// Truncate to roughly `max_tokens` tokens on a word boundary (§3.6 stage C).
+/// Truncate to roughly `max_tokens` tokens on a word boundary.
 pub fn truncate_tokens(text: &str, max_tokens: usize) -> String {
     let max_chars = max_tokens.saturating_mul(4);
     if text.len() <= max_chars {
@@ -272,11 +275,11 @@ mod tests {
     fn html_becomes_readable_text() {
         let html = "<h1>Title</h1><p>First &amp; best.</p><script>alert('x')</script>\
                     <p>Second<br/>line</p><style>p{color:red}</style>";
-        assert_eq!(html_to_text(html), "Title First & best. Second line");
+        assert_eq!(prompt_text(html), "Title First & best. Second line");
-        assert_eq!(html_to_text(""), "");
+        assert_eq!(prompt_text(""), "");
-        assert_eq!(html_to_text("no markup at all"), "no markup at all");
+        assert_eq!(prompt_text("no markup at all"), "no markup at all");
         // Unicode survives byte-wise walking.
-        assert_eq!(html_to_text("<p>café — naïve</p>"), "café — naïve");
+        assert_eq!(prompt_text("<p>café — naïve</p>"), "café — naïve");
     }
 
     #[test]

src/curate/prefilter.rs

@@ -1,29 +1,12 @@
-//! Heuristic pre-filter: 300–500 articles → ~120 candidates (spec §3.5).
+//! Hygiene matchers and the text-only heuristic used by personalized ranking.
 //!
-//! Pure Rust and free: this is what keeps LLM cost flat as feed volume grows.
+//! This module no longer gates the candidate pool. Admission lives in
-//!
+//! `curate::admit`; these helpers remain here because hygiene and cheap signals
-//! The 0–100 score is a sum of bounded components so that no single signal can
+//! share them (plan §8.1, §9, §18).
-//! dominate, and every component is monotonic in its input:
-//!
-//! | component | range | source |
-//! |---|---|---|
-//! | long-form word count | 0 … +35 | §3.5 "0 pts <300 words, max at ~2500+" |
-//! | social proof | 0 … +25 | §3.4 composite, log-scaled again |
-//! | came via Scour | +8 | §3.5 (already matched a stated interest) |
-//! | came via HN frontpage | +8 | §3.5 |
-//! | carried by several feeds | 0 … +8 | §3.2 (multi-source *is* social proof) |
-//! | feed prior | −12 … +12 | §3.9 beta-smoothed upvote rate, neutral at 0.5 |
-//! | excerpt only | −20 | §3.5 (penalized, never banned — §7) |
-//! | roundup/release-notes title | −15 | §3.5 |
-//! | blocked domain | excluded | §3.5 |
 
-use std::collections::{HashMap, HashSet};
+use crate::config::CurationConfig;
+use crate::types::{Article, FeedId};
 
-use crate::config::{Config, CurationConfig};
-use crate::types::{Article, ArticleId, FeedId, FeedPrior, ScoredArticle, SourceKind};
-
-/// Title patterns that mark low-effort posts: link roundups, release notes,
-/// sponsor posts (§3.5).
 pub const PENALTY_TITLE_PATTERNS: &[&str] = &[
     "link roundup",
     "links for",
@@ -49,96 +32,12 @@ pub const PENALTY_TITLE_PATTERNS: &[&str] = &[
     "digest #",
 ];
 
-/// Word count at which the long-form bonus saturates (§3.5).
 pub const LONGFORM_SATURATION_WORDS: i64 = 2500;
-/// Below this word count the long-form bonus is zero (§3.5).
 pub const LONGFORM_FLOOR_WORDS: i64 = 300;
-/// Articles the LLM scored below this within the last week are not re-scored (§3.5).
-pub const STALE_LOW_SCORE: f64 = 3.0;
-/// Lookback for the "don't re-score churn" rule (§3.5).
-pub const STALE_LOOKBACK_DAYS: i64 = 7;
-
-/// Maximum contribution of each scoring component (§3.5).
 pub const MAX_LONGFORM_POINTS: f64 = 35.0;
-pub const MAX_SOCIAL_POINTS: f64 = 25.0;
-pub const SCOUR_BONUS: f64 = 8.0;
-pub const HN_FRONTPAGE_BONUS: f64 = 8.0;
-pub const MAX_MULTI_SOURCE_POINTS: f64 = 8.0;
-pub const MAX_FEED_PRIOR_POINTS: f64 = 12.0;
 pub const EXCERPT_ONLY_PENALTY: f64 = 20.0;
 pub const ROUNDUP_TITLE_PENALTY: f64 = 15.0;
 
-/// `composite_social_score` value that earns the full social bonus. Empirically
-/// ~6.0 is a 1,000-point HN story with 500 comments (§3.4 formula).
-const SOCIAL_SATURATION: f64 = 6.0;
-
-/// Everything the pre-filter needs beyond the articles themselves (§3.5, §3.9).
-#[derive(Debug, Clone, Default)]
-pub struct PrefilterContext {
-    /// Per-feed Bayesian upvote rate from ratings history (§3.9).
-    pub feed_priors: HashMap<FeedId, FeedPrior>,
-    /// Article ids already published in a previous issue (§3.5).
-    pub already_published: Vec<ArticleId>,
-    /// Article ids the LLM scored < [`STALE_LOW_SCORE`] recently (§3.5).
-    pub recently_rejected: Vec<ArticleId>,
-}
-
-impl PrefilterContext {
-    /// Load the history/priors context from SQLite (§3.5 dedup-vs-history, §3.9).
-    ///
-    /// `today` anchors the [`STALE_LOOKBACK_DAYS`] window.
-    pub async fn load(
-        db: &crate::db::Db,
-        today: jiff::civil::Date,
-    ) -> Result<Self, crate::db::DbError> {
-        let since = today
-            .checked_sub(jiff::Span::new().days(STALE_LOOKBACK_DAYS))
-            .unwrap_or(today);
-        let feed_priors = db
-            .feed_priors()
-            .await?
-            .into_iter()
-            .map(|p| (p.feed_id, p))
-            .collect();
-        let already_published = db.previously_published_ids().await?;
-        let recently_rejected = db.recently_low_scored_ids(STALE_LOW_SCORE, since).await?;
-        tracing::debug!(
-            priors = ?feed_priors_len(&feed_priors),
-            published = already_published.len(),
-            rejected = recently_rejected.len(),
-            "loaded prefilter context"
-        );
-        Ok(Self {
-            feed_priors,
-            already_published,
-            recently_rejected,
-        })
-    }
-
-    fn prior_for(&self, article: &Article) -> f64 {
-        // The cluster's feeds are all candidates; take the most favourable one,
-        // since a story carried by a well-rated feed is a better bet.
-        let mut best = self.feed_priors.get(&article.feed_id).map(FeedPrior::rate);
-        for source in &article.sources {
-            if let Some(p) = self.feed_priors.get(&source.feed_id) {
-                let rate = p.rate();
-                best = Some(best.map_or(rate, |b: f64| b.max(rate)));
-            }
-        }
-        best.unwrap_or(0.5)
-    }
-}
-
-fn feed_priors_len(m: &HashMap<FeedId, FeedPrior>) -> usize {
-    m.len()
-}
-
-/// True when the article's feed is in `curation.always_include_feeds` (§3.5).
-///
-/// Entries are matched either as a Miniflux feed id (any feed in the cluster) or
-/// as a case-insensitive substring of the article/site URL.
-///
-/// Auto-includes are still LLM-scored (for section + summary) but can't be dropped.
 pub fn is_auto_include(article: &Article, cfg: &CurationConfig) -> bool {
     if cfg.always_include_feeds.is_empty() {
         return false;
@@ -151,38 +50,30 @@ pub fn is_auto_include(article: &Article, cfg: &CurationConfig) -> bool {
             return false;
         }
         if let Ok(id) = needle.parse::<FeedId>()
-            && (article.feed_id == id || article.sources.iter().any(|s| s.feed_id == id))
+            && (article.feed_id == id || article.sources.iter().any(|source| source.feed_id == id))
         {
             return true;
         }
-        let needle = needle.to_lowercase();
-        // Bare host or full site URL: compare against both URLs we hold.
         let needle = needle
+            .to_lowercase()
             .trim_start_matches("https://")
             .trim_start_matches("http://")
-            .trim_end_matches('/');
+            .trim_end_matches('/')
-        !needle.is_empty() && (url.contains(needle) || canonical.contains(needle))
+            .to_string();
+        !needle.is_empty() && (url.contains(&needle) || canonical.contains(&needle))
     })
 }
 
-/// True when the article's host matches `curation.blocked_domains` (§3.5).
 pub fn is_blocked(article: &Article, cfg: &CurationConfig) -> bool {
-    if cfg.blocked_domains.is_empty() {
-        return false;
-    }
     let host = host_of(&article.canonical_url)
         .or_else(|| host_of(&article.url))
         .unwrap_or_default();
-    if host.is_empty() {
-        return false;
-    }
     cfg.blocked_domains.iter().any(|raw| {
         let blocked = raw.trim().trim_start_matches('.').to_lowercase();
         !blocked.is_empty() && (host == blocked || host.ends_with(&format!(".{blocked}")))
     })
 }
 
-/// Lowercased host of a URL, `www.` stripped.
 fn host_of(url: &str) -> Option<String> {
     let rest = url
         .split_once("://")
@@ -190,17 +81,12 @@ fn host_of(url: &str) -> Option<String> {
         .unwrap_or(url)
         .split(['/', '?', '#'])
         .next()?;
-    let host = rest.rsplit_once('@').map(|(_, h)| h).unwrap_or(rest);
+    let host = rest.rsplit_once('@').map(|(_, host)| host).unwrap_or(rest);
-    let host = host.split_once(':').map(|(h, _)| h).unwrap_or(host);
+    let host = host.split_once(':').map(|(host, _)| host).unwrap_or(host);
     let host = host.trim().to_lowercase();
-    if host.is_empty() {
+    (!host.is_empty()).then(|| host.trim_start_matches("www.").to_string())
-        None
-    } else {
-        Some(host.trim_start_matches("www.").to_string())
-    }
 }
 
-/// True when the title reads like a link roundup / release note / sponsor post (§3.5).
 pub fn looks_like_roundup(title: &str) -> bool {
     let lower = title.to_lowercase();
     PENALTY_TITLE_PATTERNS
@@ -208,139 +94,38 @@ pub fn looks_like_roundup(title: &str) -> bool {
         .any(|pattern| lower.contains(pattern))
 }
 
-/// Long-form bonus: zero below [`LONGFORM_FLOOR_WORDS`], saturating at
-/// [`LONGFORM_SATURATION_WORDS`], with a concave curve so that the jump from a
-/// 400-word note to a 1,200-word piece matters more than 2,000 → 2,500 (§3.5).
 pub fn longform_points(word_count: i64) -> f64 {
     let span = (LONGFORM_SATURATION_WORDS - LONGFORM_FLOOR_WORDS) as f64;
     let over = (word_count - LONGFORM_FLOOR_WORDS).max(0) as f64;
     MAX_LONGFORM_POINTS * (over / span).min(1.0).powf(0.65)
 }
 
-/// Social proof, log-scaled a second time so that a viral story cannot swamp the
+pub fn text_heuristic(article: &Article) -> f64 {
-/// long-form preference (§3.4, §3.5).
+    longform_points(article.word_count)
-pub fn social_points(social_score: f64) -> f64 {
+        - excerpt_only_penalty(article)
-    if social_score <= 0.0 {
+        - roundup_penalty(&article.title)
-        return 0.0;
-    }
-    MAX_SOCIAL_POINTS * (social_score / SOCIAL_SATURATION).min(1.0).sqrt()
 }
 
-/// Score one article 0–100 from word count, social proof, source signals, feed
+pub fn excerpt_only_penalty(article: &Article) -> f64 {
-/// prior, and the excerpt/roundup/blocklist penalties (§3.5).
-pub fn score_article(article: &Article, ctx: &PrefilterContext, cfg: &Config) -> f64 {
-    if is_blocked(article, &cfg.curation) {
-        return 0.0;
-    }
-    let mut score = longform_points(article.word_count);
-    score += social_points(article.social_score());
-
-    if article.came_via(SourceKind::Scour) {
-        score += SCOUR_BONUS;
-    }
-    if article.came_via(SourceKind::HnFrontpage) {
-        score += HN_FRONTPAGE_BONUS;
-    }
-
-    let extra_feeds = article.sources.len().saturating_sub(1) as f64;
-    score += (extra_feeds * 4.0).min(MAX_MULTI_SOURCE_POINTS);
-
-    // Beta-smoothed upvote rate, neutral (0.5) contributing nothing (§3.9).
-    score += (ctx.prior_for(article) - 0.5) * 2.0 * MAX_FEED_PRIOR_POINTS;
-
     if article.excerpt_only {
-        score -= EXCERPT_ONLY_PENALTY;
+        EXCERPT_ONLY_PENALTY
-    }
-    if looks_like_roundup(&article.title) {
-        score -= ROUNDUP_TITLE_PENALTY;
-    }
-
-    score.clamp(0.0, 100.0)
-}
-
-/// Apply [`score_article`] to everything, drop history duplicates, then keep the
-/// top `prefilter_keep` plus every auto-include (§3.5).
-pub fn run(articles: Vec<Article>, ctx: &PrefilterContext, cfg: &Config) -> Vec<ScoredArticle> {
-    let published: HashSet<ArticleId> = ctx.already_published.iter().copied().collect();
-    let rejected: HashSet<ArticleId> = ctx.recently_rejected.iter().copied().collect();
-
-    let total = articles.len();
-    let (mut dropped_history, mut dropped_blocked) = (0usize, 0usize);
-    let mut scored: Vec<ScoredArticle> = Vec::with_capacity(total);
-
-    for article in articles {
-        let auto_include = is_auto_include(&article, &cfg.curation);
-
-        // Never print the same story twice, not even from an always-include feed.
-        if published.contains(&article.id) {
-            dropped_history += 1;
-            continue;
-        }
-        // "Don't re-score churn" (§3.5) — but an always-include feed still gets in.
-        if !auto_include && rejected.contains(&article.id) {
-            dropped_history += 1;
-            continue;
-        }
-        if !auto_include && is_blocked(&article, &cfg.curation) {
-            dropped_blocked += 1;
-            continue;
-        }
-
-        let prefilter_score = score_article(&article, ctx, cfg);
-        let social_score = article.social_score();
-        let feed_prior = ctx.prior_for(&article);
-        scored.push(ScoredArticle {
-            article,
-            prefilter_score,
-            social_score,
-            feed_prior,
-            llm: None,
-            auto_include,
-        });
-    }
-
-    // Descending by score; ties broken by word count then id so the order is
-    // deterministic across runs (notes §12).
-    sort_by_prefilter(&mut scored);
-
-    let keep = cfg.prefilter_keep.max(cfg.target_article_count);
-    let kept: Vec<ScoredArticle> = if scored.len() <= keep {
-        scored
     } else {
-        let (head, tail) = scored.split_at(keep);
+        0.0
-        let mut kept = head.to_vec();
+    }
-        // Auto-includes below the cut are pulled back in — they can't be dropped.
-        kept.extend(tail.iter().filter(|s| s.auto_include).cloned());
-        sort_by_prefilter(&mut kept);
-        kept
-    };
-
-    tracing::info!(
-        input = total,
-        kept = kept.len(),
-        auto_includes = kept.iter().filter(|s| s.auto_include).count(),
-        dropped_history,
-        dropped_blocked,
-        "pre-filter complete"
-    );
-    kept
 }
 
-/// Deterministic ranking: score desc, then longer, then lowest id (notes §12).
+pub fn roundup_penalty(title: &str) -> f64 {
-pub fn sort_by_prefilter(scored: &mut [ScoredArticle]) {
+    if looks_like_roundup(title) {
-    scored.sort_by(|a, b| {
+        ROUNDUP_TITLE_PENALTY
-        b.prefilter_score
+    } else {
-            .partial_cmp(&a.prefilter_score)
+        0.0
-            .unwrap_or(std::cmp::Ordering::Equal)
+    }
-            .then_with(|| b.article.word_count.cmp(&a.article.word_count))
-            .then_with(|| a.article.id.cmp(&b.article.id))
-    });
 }
 
 #[cfg(test)]
 pub(crate) mod tests {
     use super::*;
-    use crate::types::{ExtractMethod, SocialRef, SocialSource, SourceRef};
+    use crate::types::{ArticleId, ExtractMethod, SocialRef, SocialSource, SourceKind, SourceRef};
     use jiff::Timestamp;
 
     pub(crate) fn ts() -> Timestamp {
@@ -349,7 +134,6 @@ pub(crate) mod tests {
             .expect("static timestamp parses")
     }
 
-    /// A plain 800-word article from feed 7 with no social proof.
     pub(crate) fn article(id: ArticleId, title: &str, word_count: i64) -> Article {
         Article {
             id,
@@ -381,285 +165,40 @@ pub(crate) mod tests {
         }
     }
 
-    pub(crate) fn with_social(mut a: Article, points: i64, comments: i64) -> Article {
+    pub(crate) fn with_social(mut article: Article, points: i64, comments: i64) -> Article {
-        a.social = vec![SocialRef {
+        article.social = vec![SocialRef {
-            article_id: a.id,
+            article_id: article.id,
             source: SocialSource::Hn,
             item_id: Some("1".into()),
             score: points,
             num_comments: comments,
-            item_url: Some("https://news.ycombinator.com/item?id=1".into()),
+            item_url: None,
             fetched_at: ts(),
         }];
-        a
+        article
-    }
-
-    pub(crate) fn via(mut a: Article, kind: SourceKind, feed_id: FeedId) -> Article {
-        a.sources.push(SourceRef {
-            entry_id: a.best_entry_id,
-            feed_id,
-            feed_title: format!("{kind:?} feed"),
-            category: None,
-            kind,
-        });
-        a
-    }
-
-    fn cfg() -> Config {
-        Config {
-            prefilter_keep: 3,
-            target_article_count: 2,
-            ..Config::default()
-        }
     }
 
     #[test]
-    fn longform_curve_is_monotonic_and_bounded() {
+    fn text_heuristic_has_only_text_terms() {
-        assert_eq!(longform_points(0), 0.0);
+        let quiet = article(1, "An essay", 1200);
-        assert_eq!(longform_points(LONGFORM_FLOOR_WORDS), 0.0);
+        let loud = with_social(quiet.clone(), 500, 200);
-        let mut prev = -1.0;
+        assert_eq!(text_heuristic(&quiet), text_heuristic(&loud));
-        for wc in [0, 100, 299, 300, 500, 900, 1500, 2200, 2500, 9000] {
+        assert!(text_heuristic(&article(2, "This Week in Rust", 1200)) < text_heuristic(&quiet));
-            let pts = longform_points(wc);
-            assert!(pts >= prev, "not monotonic at {wc}");
-            assert!(pts <= MAX_LONGFORM_POINTS);
-            prev = pts;
-        }
-        assert!((longform_points(2500) - MAX_LONGFORM_POINTS).abs() < 1e-9);
-        assert!((longform_points(50_000) - MAX_LONGFORM_POINTS).abs() < 1e-9);
     }
 
     #[test]
-    fn social_curve_is_monotonic_and_bounded() {
+    fn blocked_and_auto_include_match() {
-        let mut prev = -1.0;
+        let cfg = CurationConfig {
-        for s in [0.0, 0.5, 1.0, 2.0, 4.0, 6.0, 20.0] {
+            blocked_domains: vec!["spam.example".into()],
-            let pts = social_points(s);
+            always_include_feeds: vec!["99".into(), "tyler.blog".into()],
-            assert!(pts >= prev);
+            ..CurationConfig::default()
-            assert!(pts <= MAX_SOCIAL_POINTS);
+        };
-            prev = pts;
+        let mut blocked = article(1, "spam", 100);
-        }
+        blocked.url = "https://news.spam.example/a".into();
-        assert_eq!(social_points(0.0), 0.0);
+        blocked.canonical_url.clone_from(&blocked.url);
-        assert!((social_points(6.0) - MAX_SOCIAL_POINTS).abs() < 1e-9);
+        assert!(is_blocked(&blocked, &cfg));
-    }
+        let mut auto = article(2, "post", 100);
-
-    #[test]
-    fn score_rises_with_length_and_social_proof() {
-        let (ctx, cfg) = (PrefilterContext::default(), cfg());
-        let short = score_article(&article(1, "A thought", 200), &ctx, &cfg);
-        let medium = score_article(&article(2, "An essay", 1200), &ctx, &cfg);
-        let long = score_article(&article(3, "A treatise", 3000), &ctx, &cfg);
-        assert!(short < medium, "{short} !< {medium}");
-        assert!(medium < long, "{medium} !< {long}");
-
-        let quiet = score_article(&article(4, "An essay", 1200), &ctx, &cfg);
-        let loud = score_article(
-            &with_social(article(5, "An essay", 1200), 400, 250),
-            &ctx,
-            &cfg,
-        );
-        assert!(loud > quiet);
-        assert!(loud <= 100.0);
-    }
-
-    #[test]
-    fn source_bonuses_and_penalties_apply() {
-        let (ctx, cfg) = (PrefilterContext::default(), cfg());
-        // Long enough that the penalties do not run into the 0 floor.
-        let plain = score_article(&article(1, "Deep dive", 3000), &ctx, &cfg);
-        assert!(plain > EXCERPT_ONLY_PENALTY);
-
-        let scoured = score_article(
-            &via(article(2, "Deep dive", 3000), SourceKind::Scour, 42),
-            &ctx,
-            &cfg,
-        );
-        // Scour bonus + one extra feed in the cluster.
-        assert!(scoured > plain + SCOUR_BONUS - 0.001);
-
-        let mut excerpt = article(3, "Deep dive", 3000);
-        excerpt.excerpt_only = true;
-        assert!(
-            (score_article(&excerpt, &ctx, &cfg) - (plain - EXCERPT_ONLY_PENALTY)).abs() < 1e-9
-        );
-
-        let roundup = article(4, "This Week in Rust #612", 3000);
-        assert!(looks_like_roundup(&roundup.title));
-        assert!(
-            (score_article(&roundup, &ctx, &cfg) - (plain - ROUNDUP_TITLE_PENALTY)).abs() < 1e-9
-        );
-    }
-
-    #[test]
-    fn feed_prior_moves_the_score_both_ways() {
-        let cfg = cfg();
-        let mut liked = PrefilterContext::default();
-        liked.feed_priors.insert(
-            7,
-            FeedPrior {
-                feed_id: 7,
-                upvotes: 18,
-                downvotes: 0,
-                included: 18,
-            },
-        );
-        let mut disliked = PrefilterContext::default();
-        disliked.feed_priors.insert(
-            7,
-            FeedPrior {
-                feed_id: 7,
-                upvotes: 0,
-                downvotes: 18,
-                included: 18,
-            },
-        );
-        let a = article(1, "Deep dive", 1200);
-        let neutral = score_article(&a, &PrefilterContext::default(), &cfg);
-        assert!(score_article(&a, &liked, &cfg) > neutral);
-        assert!(score_article(&a, &disliked, &cfg) < neutral);
-    }
-
-    #[test]
-    fn blocked_domains_and_auto_includes_match_urls_and_ids() {
-        let mut cfg = cfg();
-        cfg.curation.blocked_domains = vec!["spam.example".into()];
-        cfg.curation.always_include_feeds = vec!["99".into(), "tyler.blog".into()];
-
-        let mut blocked = article(1, "Buy now", 1200);
-        blocked.canonical_url = "https://news.spam.example/post".into();
-        blocked.url.clone_from(&blocked.canonical_url);
-        assert!(is_blocked(&blocked, &cfg.curation));
-        assert_eq!(
-            score_article(&blocked, &PrefilterContext::default(), &cfg),
-            0.0
-        );
-
-        let mut by_url = article(2, "A rare post", 900);
-        by_url.url = "https://tyler.blog/2026/rare".into();
-        assert!(is_auto_include(&by_url, &cfg.curation));
-
-        let mut by_id = article(3, "Another rare post", 900);
-        by_id.feed_id = 99;
-        assert!(is_auto_include(&by_id, &cfg.curation));
-
-        assert!(!is_auto_include(&article(4, "Normal", 900), &cfg.curation));
-    }
-
-    #[test]
-    fn keeps_top_n_plus_auto_includes_and_drops_history() {
-        let mut cfg = cfg();
-        cfg.prefilter_keep = 2;
-        cfg.curation.always_include_feeds = vec!["99".into()];
-
-        let mut auto = article(5, "A short personal note", 120);
         auto.feed_id = 99;
-
+        assert!(is_auto_include(&auto, &cfg));
-        let articles = vec![
-            article(1, "Long treatise", 4000),
-            article(2, "Medium essay", 1500),
-            article(3, "Shorter piece", 700),
-            article(4, "Already printed", 5000),
-            auto,
-            article(6, "Rejected yesterday", 3000),
-        ];
-        let ctx = PrefilterContext {
-            already_published: vec![4],
-            recently_rejected: vec![6],
-            ..PrefilterContext::default()
-        };
-
-        let kept = run(articles, &ctx, &cfg);
-        let ids: Vec<ArticleId> = kept.iter().map(|s| s.article.id).collect();
-        assert!(!ids.contains(&4), "previously published must be dropped");
-        assert!(!ids.contains(&6), "recently rejected must be dropped");
-        assert!(ids.contains(&5), "auto-include survives below the cut");
-        assert!(ids.contains(&1) && ids.contains(&2));
-        assert!(!ids.contains(&3), "cut at prefilter_keep");
-        assert_eq!(kept.len(), 3); // 2 kept + 1 auto-include
-
-        // Sorted by score, descending.
-        for pair in kept.windows(2) {
-            assert!(pair[0].prefilter_score >= pair[1].prefilter_score);
-        }
-        assert!(
-            kept.iter()
-                .find(|s| s.article.id == 5)
-                .is_some_and(|s| s.auto_include)
-        );
-    }
-
-    #[test]
-    fn auto_include_survives_the_recently_rejected_list_but_not_republication() {
-        let mut cfg = cfg();
-        cfg.curation.always_include_feeds = vec!["99".into()];
-        let mut a = article(1, "Personal note", 200);
-        a.feed_id = 99;
-        let mut b = article(2, "Personal note two", 200);
-        b.feed_id = 99;
-
-        let ctx = PrefilterContext {
-            recently_rejected: vec![1],
-            already_published: vec![2],
-            ..PrefilterContext::default()
-        };
-        let kept = run(vec![a, b], &ctx, &cfg);
-        let ids: Vec<ArticleId> = kept.iter().map(|s| s.article.id).collect();
-        assert_eq!(ids, vec![1]);
-    }
-
-    #[tokio::test]
-    async fn context_loads_history_from_sqlite() {
-        let dir = tempfile::tempdir().expect("tempdir");
-        let db = crate::db::Db::open_and_migrate(&dir.path().join("t.db"))
-            .await
-            .expect("db");
-        let date: jiff::civil::Date = "2026-08-15".parse().expect("date");
-
-        db.upsert_feed_prior(&FeedPrior {
-            feed_id: 7,
-            upvotes: 4,
-            downvotes: 1,
-            included: 5,
-        })
-        .await
-        .expect("prior");
-        sqlx::query(
-            "INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
-                 (42, 'https://example.com/42', 'Printed', '2026-08-14T00:00:00Z'),
-                 (43, 'https://example.com/43', 'Rejected', '2026-08-14T00:00:00Z'),
-                 (44, 'https://example.com/44', 'Ancient', '2020-01-01T00:00:00Z')",
-        )
-        .execute(db.pool())
-        .await
-        .expect("articles");
-        db.upsert_issue(
-            "2026-08-14".parse().expect("date"),
-            1,
-            ts(),
-            None,
-            None,
-            None,
-            None,
-            None,
-        )
-        .await
-        .expect("issue");
-        sqlx::query(
-            "INSERT INTO issue_articles (issue_date, article_id, section, position, is_lead)
-             VALUES ('2026-08-14', 42, 'Top Stories', 1, 0)",
-        )
-        .execute(db.pool())
-        .await
-        .expect("issue article");
-        sqlx::query(
-            "INSERT INTO scores (article_id, run_date, llm_score) VALUES (43, '2026-08-14', 1.5),
-                                                                        (44, '2020-01-01', 1.0)",
-        )
-        .execute(db.pool())
-        .await
-        .expect("scores");
-
-        let ctx = PrefilterContext::load(&db, date).await.expect("context");
-        assert_eq!(ctx.already_published, vec![42]);
-        assert_eq!(ctx.recently_rejected, vec![43], "old rejects age out");
-        assert!((ctx.feed_priors[&7].rate() - 5.0 / 7.0).abs() < 1e-12);
     }
 }

src/curate/profile/mod.rs

@@ -0,0 +1,684 @@
+//! Reader-profile and system-prompt construction (personalized curation v2 §8).
+//!
+//! Every run rebuilds one byte-stable prompt from the hand-maintained profile,
+//! standing interests, stored weekly adjustments, and current explicit verdicts.
+
+use std::collections::BTreeSet;
+use std::fmt::Write as _;
+use std::path::Path;
+
+use anyhow::Context as _;
+use jiff::Timestamp;
+use serde::{Deserialize, Serialize};
+
+use super::llm::LlmClient;
+use crate::db::{Db, KV_PROFILE_VERSION, KV_TASTE_PROFILE};
+use crate::types::{Facets, RatedArticle, TasteProfile};
+
+pub const REBUILD_INTERVAL_DAYS: i64 = 7;
+/// The verdict block and the weekly rebuild are bounded by count
+/// (`verdicts_in_prompt`, [`MAX_RATINGS_IN_REBUILD`]), not by age (§8.3, §8.4),
+/// so their `current_ratings` lookback is effectively unbounded.
+const RATINGS_LOOKBACK_DAYS: i64 = 36_500;
+pub const KV_LEARNED_ADJUSTMENTS: &str = "taste_profile_learned";
+const MAX_RATINGS_IN_REBUILD: usize = 200;
+
+pub const NO_LEARNED_ADJUSTMENTS: &str = "No reader ratings have been collected yet. Judge purely on the stated preferences and interests above.";
+
+/// The only reader-profile prose that remains in code (§8.2).
+const EDITOR_IN_CHIEF_FRAMING: &str = "You are the editor-in-chief of *The Daily EPUB*, a personal morning newspaper assembled every day for exactly one reader. Everything you are asked to do — score, select, place, summarize, introduce — serves his taste, not a general audience's. When a judgement call is close, re-read this profile and decide the way he would.";
+
+// ---------------------------------------------------------------------------
+// Interest and profile-file parsing
+// ---------------------------------------------------------------------------
+
+pub fn parse_interests(opml_path: &Path) -> anyhow::Result<Vec<String>> {
+    let raw = std::fs::read_to_string(opml_path)
+        .with_context(|| format!("reading the interests OPML at {}", opml_path.display()))?;
+    let interests = parse_interests_str(&raw);
+    if interests.is_empty() {
+        anyhow::bail!(
+            "no <outline text=\"…\"> interests found in {}",
+            opml_path.display()
+        );
+    }
+    tracing::debug!(count = interests.len(), "parsed scour interests");
+    Ok(interests)
+}
+
+pub fn parse_interests_str(raw: &str) -> Vec<String> {
+    let mut seen = BTreeSet::new();
+    let mut out = Vec::new();
+    for chunk in raw.split("text=\"").skip(1) {
+        let Some((value, _)) = chunk.split_once('"') else {
+            continue;
+        };
+        let name = xml_unescape(value).trim().to_string();
+        if !name.is_empty() && seen.insert(name.to_lowercase()) {
+            out.push(name);
+        }
+    }
+    out
+}
+
+fn xml_unescape(s: &str) -> String {
+    if !s.contains('&') {
+        return s.to_string();
+    }
+    s.replace("&lt;", "<")
+        .replace("&gt;", ">")
+        .replace("&quot;", "\"")
+        .replace("&apos;", "'")
+        .replace("&#39;", "'")
+        .replace("&amp;", "&")
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ProfileFile {
+    /// Original Markdown with every `## Interests` section removed.
+    pub body: String,
+    pub interests: Vec<String>,
+}
+
+/// Remove any `## Interests` section and parse its non-empty lines as interests.
+/// A leading `- ` is stripped; all other profile bytes pass through unchanged.
+pub fn parse_profile_str(raw: &str) -> ProfileFile {
+    let mut body = String::with_capacity(raw.len());
+    let mut interests = Vec::new();
+    let mut in_interests = false;
+
+    for line in raw.split_inclusive('\n') {
+        let heading = line.trim_end_matches(['\r', '\n']).trim();
+        if heading.eq_ignore_ascii_case("## Interests") {
+            in_interests = true;
+            continue;
+        }
+        if in_interests && heading.starts_with("## ") {
+            in_interests = false;
+        }
+        if in_interests {
+            let interest = heading.strip_prefix("- ").unwrap_or(heading).trim();
+            if !interest.is_empty()
+                && !interest.eq_ignore_ascii_case(
+                    "(optional: one per line; merged with data/scour-interests.opml)",
+                )
+            {
+                interests.push(interest.to_string());
+            }
+        } else {
+            body.push_str(line);
+        }
+    }
+    ProfileFile { body, interests }
+}
+
+pub fn load_profile(path: &Path) -> anyhow::Result<ProfileFile> {
+    match std::fs::read_to_string(path) {
+        Ok(raw) => Ok(parse_profile_str(&raw)),
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
+            tracing::warn!(path = %path.display(), "profile file is missing; using OPML interests only");
+            Ok(ProfileFile {
+                body: String::new(),
+                interests: Vec::new(),
+            })
+        }
+        Err(error) => {
+            Err(error).with_context(|| format!("reading the reader profile at {}", path.display()))
+        }
+    }
+}
+
+/// Load the exact standing-interest union used in the system prompt.
+pub fn load_standing_interests(
+    opml_path: &Path,
+    profile_path: &Path,
+) -> anyhow::Result<Vec<String>> {
+    let opml = parse_interests(opml_path)?;
+    let profile = load_profile(profile_path)?;
+    Ok(union_interests(opml, profile.interests))
+}
+
+fn union_interests(opml: Vec<String>, profile: Vec<String>) -> Vec<String> {
+    let mut seen = BTreeSet::new();
+    let mut out = Vec::new();
+    for interest in opml.into_iter().chain(profile) {
+        let interest = interest.trim();
+        if !interest.is_empty() && seen.insert(interest.to_lowercase()) {
+            out.push(interest.to_string());
+        }
+    }
+    out
+}
+
+pub mod themes;
+pub use themes::group_into_themes;
+
+// ---------------------------------------------------------------------------
+// Prompt assembly
+// ---------------------------------------------------------------------------
+
+fn verdict_label(label: &str) -> &str {
+    match label {
+        "loved" => "LOVED",
+        "good" => "GOOD",
+        "not_for_me" => "NOT FOR ME",
+        other => other,
+    }
+}
+
+fn one_line(text: &str) -> String {
+    text.split_whitespace().collect::<Vec<_>>().join(" ")
+}
+
+/// Assemble sections in the exact cache-friendly order required by §8.4.
+pub fn build(
+    profile_body: &str,
+    interests: &[String],
+    learned_adjustments: &str,
+    ratings: &[RatedArticle],
+    verdict_limit: usize,
+) -> String {
+    let mut doc = String::with_capacity(16 * 1024);
+    doc.push_str(EDITOR_IN_CHIEF_FRAMING);
+    doc.push_str("\n\n");
+
+    if !profile_body.is_empty() {
+        doc.push_str(profile_body);
+        if !profile_body.ends_with('\n') {
+            doc.push('\n');
+        }
+        doc.push('\n');
+    }
+
+    doc.push_str("## Standing interests\n\n");
+    doc.push_str("These are his subscribed interest topics, grouped. They raise the floor for a match, but never cap the paper: an outstanding article on none of these still belongs.\n\n");
+    for (theme, members) in group_into_themes(interests) {
+        let _ = writeln!(doc, "- **{}**: {}", theme, members.join(", "));
+    }
+
+    doc.push_str("\n## Learned adjustments (rebuilt weekly from ratings)\n\n");
+    let learned = learned_adjustments.trim();
+    doc.push_str(if learned.is_empty() {
+        NO_LEARNED_ADJUSTMENTS
+    } else {
+        learned
+    });
+
+    doc.push_str("\n\n## Recent verdicts\n\n");
+    for rating in ratings.iter().take(verdict_limit) {
+        let summary = rating
+            .summary
+            .as_deref()
+            .map(one_line)
+            .filter(|summary| !summary.is_empty())
+            .unwrap_or_else(|| "no summary available".to_string());
+        let feed = if rating.feed_title.trim().is_empty() {
+            "unknown"
+        } else {
+            rating.feed_title.trim()
+        };
+        let _ = writeln!(
+            doc,
+            "{} | {} | {} | {}",
+            verdict_label(&rating.label),
+            one_line(&rating.title),
+            one_line(feed),
+            summary
+        );
+    }
+    doc
+}
+
+// ---------------------------------------------------------------------------
+// Persistence and per-run loading
+// ---------------------------------------------------------------------------
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct ProfileVersion {
+    version: i64,
+    built_at: String,
+}
+
+async fn stored_version(db: &Db) -> anyhow::Result<Option<(i64, Timestamp)>> {
+    let Some(raw) = db.kv_get(KV_PROFILE_VERSION).await? else {
+        return Ok(None);
+    };
+    match serde_json::from_str::<ProfileVersion>(&raw) {
+        Ok(version) => {
+            let built_at = version.built_at.parse::<Timestamp>().unwrap_or_else(|_| {
+                tracing::warn!(value = %version.built_at, "unparseable profile build time");
+                Timestamp::UNIX_EPOCH
+            });
+            Ok(Some((version.version, built_at)))
+        }
+        Err(error) => {
+            tracing::warn!(%error, "unparseable kv[profile_version]; treating as absent");
+            Ok(None)
+        }
+    }
+}
+
+async fn store_version(db: &Db, version: i64, built_at: Timestamp) -> anyhow::Result<()> {
+    let json = serde_json::to_string(&ProfileVersion {
+        version,
+        built_at: built_at.to_string(),
+    })?;
+    db.kv_set(KV_PROFILE_VERSION, &json).await?;
+    Ok(())
+}
+
+async fn prompt_inputs(
+    db: &Db,
+    opml_path: &Path,
+    profile_path: &Path,
+) -> anyhow::Result<(ProfileFile, Vec<String>, Vec<RatedArticle>, String)> {
+    let opml = parse_interests(opml_path)?;
+    let profile = load_profile(profile_path)?;
+    let interests = union_interests(opml, profile.interests.clone());
+    let ratings = db.current_ratings(RATINGS_LOOKBACK_DAYS).await?;
+    let learned = db.kv_get(KV_LEARNED_ADJUSTMENTS).await?.unwrap_or_default();
+    Ok((profile, interests, ratings, learned))
+}
+
+/// Rebuild the complete system prompt from its live inputs on every run.
+pub async fn load_or_build(
+    db: &Db,
+    opml_path: &Path,
+    profile_path: &Path,
+    verdict_limit: usize,
+) -> anyhow::Result<TasteProfile> {
+    let (profile_file, interests, ratings, learned) =
+        prompt_inputs(db, opml_path, profile_path).await?;
+    let (version, built_at) = match stored_version(db).await? {
+        Some(stored) => stored,
+        None => {
+            let built_at = Timestamp::now();
+            store_version(db, 1, built_at).await?;
+            (1, built_at)
+        }
+    };
+    let profile = TasteProfile {
+        text: build(
+            &profile_file.body,
+            &interests,
+            &learned,
+            &ratings,
+            verdict_limit,
+        ),
+        version,
+        built_at,
+        verdicts: ratings.len().min(verdict_limit),
+    };
+    db.kv_set(KV_TASTE_PROFILE, &profile.text).await?;
+    tracing::debug!(
+        version,
+        interests = interests.len(),
+        verdicts = ratings.len().min(verdict_limit),
+        chars = profile.text.len(),
+        "rebuilt the taste profile prompt"
+    );
+    Ok(profile)
+}
+
+pub async fn is_stale(db: &Db) -> anyhow::Result<bool> {
+    let Some((_, built_at)) = stored_version(db).await? else {
+        return Ok(true);
+    };
+    let age_days = (Timestamp::now().as_second() - built_at.as_second()) / 86_400;
+    Ok(age_days >= REBUILD_INTERVAL_DAYS)
+}
+
+pub async fn weekly_rebuild_if_due(
+    db: &Db,
+    llm: &LlmClient,
+    opml_path: &Path,
+    profile_path: &Path,
+    verdict_limit: usize,
+) -> anyhow::Result<Option<TasteProfile>> {
+    if !is_stale(db).await? {
+        return Ok(None);
+    }
+    if db.current_ratings(RATINGS_LOOKBACK_DAYS).await?.is_empty() {
+        tracing::debug!("profile is stale but there are no ratings to learn from");
+        return Ok(None);
+    }
+    tracing::info!("taste profile is over a week old; rebuilding learned adjustments");
+    Ok(Some(
+        rebuild(db, llm, opml_path, profile_path, verdict_limit).await?,
+    ))
+}
+
+// ---------------------------------------------------------------------------
+// Weekly learned-adjustments rebuild
+// ---------------------------------------------------------------------------
+
+pub const LEARNED_ADJUSTMENTS_PROMPT: &str = r#"TASK: rewrite the "Learned adjustments" section of the reader profile in your system prompt, using only the rating history below.
+
+Each line is an explicit verdict with the article title, feed, summary, any deep-assessment facets, and the operator's note.
+
+Look for patterns, not one-offs. Treat the stated preferences as a strong prior, not a rule. When repeated, recent behaviour clearly conflicts with an older stated preference, say so. Do not override a stated preference on one or two ratings.
+
+Write 120–200 words as 4–8 bullet points, each one imperative and usable while scoring. One bullet is required and must begin "Diversity check:": name any subject or format that is starting to dominate the loved list and should not crowd out the rest of the paper. Do not mention specific article titles, rating counts, or this instruction. If the history is too thin to support any pattern, say so in one sentence instead of inventing one, while still including the Diversity check bullet.
+
+Return JSON exactly: {"learned_adjustments": "<the bullet points, as markdown>"}
+
+RATING HISTORY (newest first):
+"#;
+
+#[derive(Debug, Clone, Deserialize)]
+struct LearnedAdjustmentsResponse {
+    #[serde(default)]
+    learned_adjustments: String,
+}
+
+fn facets_line(facets: Option<&Facets>) -> Option<String> {
+    let facets = facets?;
+    let values = [
+        facets.format.as_deref(),
+        facets.depth.as_deref(),
+        facets.evidence.as_deref(),
+        facets.technicality.as_deref(),
+        facets.topic_group.as_deref(),
+    ]
+    .into_iter()
+    .flatten()
+    .filter(|value| !value.trim().is_empty())
+    .collect::<Vec<_>>();
+    (!values.is_empty()).then(|| values.join("/"))
+}
+
+pub fn build_rebuild_prompt(ratings: &[RatedArticle]) -> String {
+    let mut prompt = String::from(LEARNED_ADJUSTMENTS_PROMPT);
+    for rating in ratings.iter().take(MAX_RATINGS_IN_REBUILD) {
+        let mut parts = vec![
+            verdict_label(&rating.label).to_string(),
+            one_line(&rating.title),
+            if rating.feed_title.trim().is_empty() {
+                "unknown".to_string()
+            } else {
+                one_line(&rating.feed_title)
+            },
+        ];
+        if let Some(summary) = rating
+            .summary
+            .as_deref()
+            .map(one_line)
+            .filter(|s| !s.is_empty())
+        {
+            parts.push(summary);
+        }
+        if let Some(facets) = facets_line(rating.facets.as_ref()) {
+            parts.push(format!("facets: {facets}"));
+        }
+        if let Some(note) = rating
+            .note
+            .as_deref()
+            .map(one_line)
+            .filter(|s| !s.is_empty())
+        {
+            parts.push(format!("note: {note}"));
+        }
+        let _ = writeln!(prompt, "{}", parts.join(" | "));
+    }
+    prompt
+}
+
+pub async fn rebuild(
+    db: &Db,
+    llm: &LlmClient,
+    opml_path: &Path,
+    profile_path: &Path,
+    verdict_limit: usize,
+) -> anyhow::Result<TasteProfile> {
+    let ratings = db.current_ratings(RATINGS_LOOKBACK_DAYS).await?;
+    let previous = db.kv_get(KV_LEARNED_ADJUSTMENTS).await?.unwrap_or_default();
+    let learned = if ratings.is_empty() {
+        tracing::info!("no ratings available; keeping the existing adjustments");
+        previous
+    } else {
+        let prompt = build_rebuild_prompt(&ratings);
+        match llm
+            .complete_json::<LearnedAdjustmentsResponse>(&prompt, 0.4)
+            .await
+        {
+            Ok(response) if !response.learned_adjustments.trim().is_empty() => {
+                response.learned_adjustments.trim().to_string()
+            }
+            Ok(_) => {
+                tracing::warn!("the model returned empty adjustments; keeping the previous ones");
+                previous
+            }
+            Err(error) => {
+                tracing::warn!(%error, "learned-adjustments rewrite failed; keeping the previous ones");
+                previous
+            }
+        }
+    };
+
+    db.kv_set(KV_LEARNED_ADJUSTMENTS, &learned).await?;
+    let next_version = stored_version(db)
+        .await?
+        .map_or(1, |(version, _)| version + 1);
+    let built_at = Timestamp::now();
+    store_version(db, next_version, built_at).await?;
+
+    let opml = parse_interests(opml_path)?;
+    let profile_file = load_profile(profile_path)?;
+    let interests = union_interests(opml, profile_file.interests.clone());
+    let current = db.current_ratings(RATINGS_LOOKBACK_DAYS).await?;
+    let profile = TasteProfile {
+        text: build(
+            &profile_file.body,
+            &interests,
+            &learned,
+            &current,
+            verdict_limit,
+        ),
+        version: next_version,
+        built_at,
+        verdicts: current.len().min(verdict_limit),
+    };
+    db.kv_set(KV_TASTE_PROFILE, &profile.text).await?;
+    tracing::info!(
+        version = next_version,
+        chars = profile.text.len(),
+        "stored a new taste profile"
+    );
+    Ok(profile)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const OPML_PATH: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/data/scour-interests.opml");
+    const PROFILE_PATH: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/data/profile.md");
+
+    #[test]
+    fn profile_interests_are_removed_and_union_case_insensitively() {
+        let parsed = parse_profile_str(
+            "# P\n\n## Interests\n- Rust\nBoston Tech\n- rust\n\n## Notes\nKeep this.\n",
+        );
+        assert_eq!(parsed.body, "# P\n\n## Notes\nKeep this.\n");
+        assert_eq!(parsed.interests, ["Rust", "Boston Tech", "rust"]);
+        let union = union_interests(vec!["rust".into(), "E-Ink".into()], parsed.interests);
+        assert_eq!(union, ["rust", "E-Ink", "Boston Tech"]);
+    }
+
+    #[test]
+    fn prompt_sections_are_ordered_and_verdicts_have_required_labels() {
+        let rating = RatedArticle {
+            article_id: 1,
+            issue_date: None,
+            title: "A title".into(),
+            feed_title: "A feed".into(),
+            summary: Some("A summary\nwith whitespace.".into()),
+            facets: None,
+            note: None,
+            value: -1.0,
+            label: "not_for_me".into(),
+            event_at: "2026-08-15T12:00:00Z".parse().unwrap(),
+        };
+        let prompt = build(
+            "# Reader profile\n\nProfile prose.",
+            &["Rust".into()],
+            "- Adjust.",
+            &[rating],
+            60,
+        );
+        let framing = prompt.find("editor-in-chief").unwrap();
+        let profile = prompt.find("# Reader profile").unwrap();
+        let interests = prompt.find("## Standing interests").unwrap();
+        let learned = prompt.find("## Learned adjustments").unwrap();
+        let verdicts = prompt.find("## Recent verdicts").unwrap();
+        assert!(
+            framing < profile && profile < interests && interests < learned && learned < verdicts
+        );
+        assert!(prompt.contains("NOT FOR ME | A title | A feed | A summary with whitespace."));
+    }
+
+    #[test]
+    fn shipped_profile_and_opml_parse() {
+        let profile = load_profile(Path::new(PROFILE_PATH)).unwrap();
+        assert!(profile.body.contains("## Who he is"));
+        assert!(!profile.body.contains("## Interests"));
+        assert!(profile.interests.is_empty());
+        let interests = parse_interests(Path::new(OPML_PATH)).unwrap();
+        assert!(interests.iter().any(|interest| interest == "Rust"));
+    }
+
+    #[test]
+    fn rebuild_prompt_carries_summary_facets_note_and_diversity_instruction() {
+        let rating = RatedArticle {
+            article_id: 1,
+            issue_date: Some("2026-08-15".parse().unwrap()),
+            title: "Postgres failover".into(),
+            feed_title: "Engineering Notes".into(),
+            summary: Some("A detailed incident report.".into()),
+            facets: Some(Facets {
+                format: Some("first_hand_account".into()),
+                depth: Some("deep".into()),
+                evidence: Some("first_hand".into()),
+                technicality: Some("advanced".into()),
+                topic_group: Some("software_engineering".into()),
+                ..Facets::default()
+            }),
+            note: Some("Great operational detail".into()),
+            value: 1.0,
+            label: "loved".into(),
+            event_at: "2026-08-15T12:00:00Z".parse().unwrap(),
+        };
+        let prompt = build_rebuild_prompt(&[rating]);
+        assert!(prompt.contains("strong prior, not a rule"));
+        assert!(prompt.contains("Diversity check:"));
+        assert!(prompt.contains(
+            "LOVED | Postgres failover | Engineering Notes | A detailed incident report."
+        ));
+        assert!(
+            prompt.contains(
+                "facets: first_hand_account/deep/first_hand/advanced/software_engineering"
+            )
+        );
+        assert!(prompt.contains("note: Great operational detail"));
+    }
+
+    #[tokio::test]
+    async fn per_run_profile_reload_changes_prompt_without_bumping_version() {
+        let dir = tempfile::tempdir().unwrap();
+        let db = Db::open_and_migrate(&dir.path().join("profile.db"))
+            .await
+            .unwrap();
+        let opml = dir.path().join("interests.opml");
+        let profile_path = dir.path().join("profile.md");
+        std::fs::write(&opml, r#"<outline text="Rust"/>"#).unwrap();
+        std::fs::write(
+            &profile_path,
+            "# Reader profile\n\nOriginal prose.\n\n## Interests\n- Custom Topic\n",
+        )
+        .unwrap();
+
+        let first = load_or_build(&db, &opml, &profile_path, 60).await.unwrap();
+        assert_eq!(first.version, 1);
+        assert!(first.text.contains("Original prose."));
+        assert!(first.text.contains("Custom Topic"));
+        assert!(!first.text.contains("## Interests"));
+
+        std::fs::write(
+            &profile_path,
+            "# Reader profile\n\nChanged prose.\n\n## Interests\n- Another Topic\n",
+        )
+        .unwrap();
+        let second = load_or_build(&db, &opml, &profile_path, 60).await.unwrap();
+        assert_eq!(second.version, first.version);
+        assert_eq!(second.built_at, first.built_at);
+        assert!(second.text.contains("Changed prose."));
+        assert!(second.text.contains("Another Topic"));
+        assert!(!second.text.contains("Original prose."));
+
+        let missing = load_profile(&dir.path().join("missing.md")).unwrap();
+        assert!(missing.body.is_empty() && missing.interests.is_empty());
+    }
+
+    #[tokio::test]
+    async fn weekly_rebuild_uses_mock_backend_and_bumps_profile_version() {
+        use std::sync::Arc;
+
+        use super::super::llm::{MockBackend, UsageMeter};
+        use crate::config::ProviderConfig;
+        use crate::types::{RatingEvent, TokenUsage};
+
+        let dir = tempfile::tempdir().unwrap();
+        let db = Db::open_and_migrate(&dir.path().join("profile.db"))
+            .await
+            .unwrap();
+        let opml = dir.path().join("interests.opml");
+        let profile_path = dir.path().join("profile.md");
+        std::fs::write(&opml, r#"<outline text="Rust"/>"#).unwrap();
+        std::fs::write(&profile_path, "# Reader profile\n\nLikes depth.\n").unwrap();
+        let initial = load_or_build(&db, &opml, &profile_path, 60).await.unwrap();
+        assert_eq!(initial.version, 1);
+
+        sqlx::query(
+            "INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
+             (1, 'https://example.com/deep', 'A deep report', '2026-08-15T00:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .unwrap();
+        db.append_rating_event(&RatingEvent {
+            id: 0,
+            article_id: 1,
+            issue_date: None,
+            kind: "explicit".into(),
+            source: "cli".into(),
+            label: "loved".into(),
+            value: 1.0,
+            note: Some("specific evidence".into()),
+            event_at: Timestamp::now(),
+        })
+        .await
+        .unwrap();
+
+        let backend = Arc::new(MockBackend::new());
+        backend.push(
+            r#"{"learned_adjustments":"- Rank first-hand reports higher.\n- Diversity check: keep formats balanced."}"#,
+            TokenUsage::default(),
+        );
+        let llm = LlmClient::with_backend(
+            "deepseek-v4-flash",
+            initial.text,
+            UsageMeter::for_provider(&ProviderConfig::deepseek()),
+            backend.clone(),
+        );
+        let rebuilt = rebuild(&db, &llm, &opml, &profile_path, 60).await.unwrap();
+        assert_eq!(rebuilt.version, 2);
+        assert!(rebuilt.text.contains("Rank first-hand reports higher."));
+        assert!(
+            rebuilt
+                .text
+                .contains("Diversity check: keep formats balanced.")
+        );
+        assert_eq!(backend.calls(), 1);
+        assert!(backend.prompts()[0].user.contains("LOVED | A deep report"));
+    }
+}

src/curate/profile/themes.rs

@@ -0,0 +1,354 @@
+//! Grouping the reader's ~220 Scour interests into readable themes (§3.6a).
+//!
+//! A lookup table and one function over it. It lives in its own file because the
+//! table is long and almost never changes, while the profile logic around it
+//! does — and a 260-line const in the middle of that logic is a wall to scroll
+//! past, not something to read.
+
+/// Theme name → lowercase keywords, in priority order. The first theme whose
+/// keyword appears in the interest name wins, so the table is ordered from the
+/// most specific bucket to the most general.
+const THEMES: &[(&str, &[&str])] = &[
+    (
+        "Systems & languages",
+        &[
+            "rust",
+            "zig",
+            "lua",
+            "assembly",
+            "compiler",
+            "systems programming",
+            "concurrency",
+            "async",
+            "memory",
+            "simd",
+            "zero-copy",
+            "parser",
+            "fuzz",
+            "static analysis",
+            "functional programming",
+            "data structures",
+            "algorithm",
+            "performance",
+            "profil",
+            "python",
+            "typescript",
+            "node.js",
+            "wasm",
+            "webassembly",
+            "reactive programming",
+            "lsp",
+        ],
+    ),
+    (
+        "Databases & data",
+        &[
+            "database",
+            "sql",
+            "postgres",
+            "query",
+            "vector databases",
+            "bloom",
+            "compression",
+            "data engineering",
+            "knowledge graph",
+            "message queue",
+            "distributed systems",
+            "crdt",
+            "microservices",
+            "system design",
+        ],
+    ),
+    (
+        "AI & machine learning",
+        &[
+            "ai",
+            "llm",
+            "machine learning",
+            "nlp",
+            "rag",
+            "prompt",
+            "agent",
+            "claude",
+            "eval",
+        ],
+    ),
+    (
+        "Web, infra & devtools",
+        &[
+            "web",
+            "css",
+            "htmx",
+            "svelte",
+            "react",
+            "pwa",
+            "api",
+            "developer",
+            "devops",
+            "docker",
+            "git",
+            "observability",
+            "cloud",
+            "cdn",
+            "edge",
+            "networking",
+            "mesh",
+            "ipfs",
+            "activitypub",
+            "atproto",
+            "search engines",
+            "tauri",
+            "design systems",
+            "cybersecurity",
+            "cryptography",
+            "monitoring",
+            "spatial computing",
+            "webrtc",
+            "webgpu",
+            "webgl",
+            "webcodecs",
+            "android",
+            "mobile",
+            "code generation",
+        ],
+    ),
+    (
+        "Self-hosting, RSS & the indie web",
+        &[
+            "self-host",
+            "self host",
+            "homelab",
+            "rss",
+            "feed reader",
+            "indie web",
+            "personal website",
+            "personal wiki",
+            "digital garden",
+            "static site",
+            "static sites",
+            "personal archiving",
+            "offline-first",
+            "privacy",
+            "open source",
+            "licensing",
+            "side projects",
+            "digital nomad",
+            "remote living",
+            "off-grid",
+            "quantified self",
+            "cloudflare workers",
+        ],
+    ),
+    (
+        "E-ink, terminals & hardware",
+        &[
+            "e-ink",
+            "eink",
+            "writerdeck",
+            "tmux",
+            "neovim",
+            "terminal",
+            "text editor",
+            "editor",
+            "window manager",
+            "keyboard",
+            "hardware",
+            "electronics",
+            "calibre",
+            "ghostty",
+            "retro computing",
+            "low-tech",
+            "manufacturing",
+            "typography",
+            "linux",
+        ],
+    ),
+    (
+        "Games & interactive fiction",
+        &[
+            "game",
+            "gaming",
+            "minecraft",
+            "mud",
+            "interactive fiction",
+            "inform7",
+            "bevy",
+            "sim racing",
+            "modding",
+        ],
+    ),
+    (
+        "Creative coding & digital art",
+        &[
+            "generative",
+            "creative coding",
+            "creative automation",
+            "glitch",
+            "procedural",
+            "digital art",
+            "gaussian splatting",
+            "cellular automata",
+            "sonification",
+            "code visualization",
+            "photo",
+            "photography",
+            "agent-based",
+        ],
+    ),
+    (
+        "Writing, books & PKM",
+        &[
+            "writing",
+            "write",
+            "reading",
+            "book",
+            "essay",
+            "poetry",
+            "literature",
+            "note-taking",
+            "journaling",
+            "pkm",
+            "knowledge management",
+            "obsidian",
+            "markdown",
+            "commonplace",
+            "longform",
+            "long-form",
+            "fiction",
+            "worldbuilding",
+            "sci-fi",
+            "science fiction",
+            "speculative",
+            "podcast",
+            "narrative",
+            "choice architecture",
+        ],
+    ),
+    (
+        "Science, space & nature",
+        &[
+            "space",
+            "aerospace",
+            "aviation",
+            "science",
+            "neuroscience",
+            "bioinformatics",
+            "nature",
+            "cognitive",
+            "social science",
+        ],
+    ),
+    (
+        "History, policy & culture",
+        &[
+            "history",
+            "policy",
+            "culture",
+            "lgbt",
+            "queer",
+            "startups",
+            "apple",
+            "boston",
+            "criticism",
+            "internet history",
+            "computing history",
+        ],
+    ),
+    (
+        "Outdoors, coffee & everyday life",
+        &[
+            "hiking",
+            "camping",
+            "running",
+            "trail",
+            "coffee",
+            "cats",
+            "films",
+            "music",
+            "engineering",
+        ],
+    ),
+];
+
+/// Fallback bucket for interests no keyword claims.
+const OTHER_THEME: &str = "Other standing interests";
+
+/// Group ~220 interests into readable themes for the prompt (§3.6).
+///
+/// Deterministic: theme order follows [`THEMES`], members are sorted
+/// case-insensitively, and empty themes are omitted.
+pub fn group_into_themes(interests: &[String]) -> Vec<(String, Vec<String>)> {
+    let mut buckets: Vec<Vec<String>> = vec![Vec::new(); THEMES.len() + 1];
+    for interest in interests {
+        let lower = interest.to_lowercase();
+        let idx = THEMES
+            .iter()
+            .position(|(_, keywords)| keywords.iter().any(|k| lower.contains(k)))
+            .unwrap_or(THEMES.len());
+        buckets[idx].push(interest.clone());
+    }
+    let mut out = Vec::new();
+    for (idx, mut members) in buckets.into_iter().enumerate() {
+        if members.is_empty() {
+            continue;
+        }
+        members.sort_by_key(|m| (m.to_lowercase(), m.clone()));
+        let name = THEMES.get(idx).map(|(n, _)| *n).unwrap_or(OTHER_THEME);
+        out.push((name.to_string(), members));
+    }
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn interests() -> Vec<String> {
+        [
+            "Rust",
+            "Zig",
+            "Postgres",
+            "SQLite",
+            "Kubernetes",
+            "E-ink",
+            "Keyboards",
+            "Retrocomputing",
+            "Bicycles",
+            "Trail running",
+            "Coffee",
+            "Bread baking",
+            "Cartography",
+            "Typography",
+            "Ambient music",
+            "Science fiction",
+            "Local politics",
+            "Boston",
+            "Machine learning",
+            "Self-hosting",
+        ]
+        .iter()
+        .map(|s| s.to_string())
+        .collect()
+    }
+
+    #[test]
+    fn theming_is_deterministic_and_total() {
+        let list = interests();
+        let a = group_into_themes(&list);
+        let b = group_into_themes(&list);
+        assert_eq!(a, b);
+        let grouped: usize = a.iter().map(|(_, m)| m.len()).sum();
+        assert_eq!(
+            grouped,
+            list.len(),
+            "every interest lands in exactly one theme"
+        );
+        assert!(a.len() >= 6, "expected several themes, got {}", a.len());
+        // Members are sorted within a theme.
+        for (_, members) in &a {
+            let mut sorted = members.clone();
+            sorted.sort_by_key(|m| (m.to_lowercase(), m.clone()));
+            assert_eq!(members, &sorted);
+        }
+    }
+}

src/curate/rank.rs

@@ -0,0 +1,586 @@
+//! Utility normalization, leader clustering and diversified shortlisting (§12.2–§12.5).
+
+use std::collections::{HashMap, HashSet};
+
+use crate::config::{RankingConfig, UtilityWeights};
+use crate::curate::embedding::dot;
+use crate::curate::signals;
+use crate::types::{ArticleId, Candidate};
+
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RankSummary {
+    pub shortlisted: usize,
+    pub clusters: usize,
+}
+
+/// Re-normalize cheap signals over the deep set and calculate utility on 0–100.
+pub fn calculate_utility(candidates: &mut [Candidate], configured: &UtilityWeights) {
+    let indices = (0..candidates.len()).collect::<Vec<_>>();
+    calculate_utility_for(candidates, &indices, configured);
+}
+
+fn calculate_utility_for(
+    candidates: &mut [Candidate],
+    indices: &[usize],
+    configured: &UtilityWeights,
+) {
+    let mut normalized = indices
+        .iter()
+        .map(|index| candidates[*index].signals.clone())
+        .collect::<Vec<_>>();
+    for signals in &mut normalized {
+        signals.norm.clear();
+        signals.weights.clear();
+    }
+    let mut signal_refs = normalized.iter_mut().collect::<Vec<_>>();
+    signals::normalize(&mut signal_refs);
+    for (index, signals) in indices.iter().zip(normalized) {
+        candidates[*index].signals.norm = signals.norm;
+        candidates[*index].signals.weights.clear();
+    }
+
+    for index in indices {
+        let candidate = &mut candidates[*index];
+        if let Some(triage) = &candidate.assessment.triage {
+            candidate
+                .signals
+                .norm
+                .insert("triage".into(), (triage.interest / 10.0).clamp(0.0, 1.0));
+        }
+        if let Some(deep) = &candidate.assessment.deep {
+            candidate
+                .signals
+                .norm
+                .insert("quality".into(), (deep.quality / 10.0).clamp(0.0, 1.0));
+            candidate
+                .signals
+                .norm
+                .insert("fit".into(), (deep.fit / 10.0).clamp(0.0, 1.0));
+        }
+        let weighted = [
+            ("quality", configured.quality, 1.0),
+            ("fit", configured.fit, 1.0),
+            ("knn", configured.knn, candidate.signals.knn_gate),
+            ("interest", configured.interest, 1.0),
+            ("feed", configured.feed, candidate.signals.feed_gate),
+            ("triage", configured.triage, 1.0),
+            ("social", configured.social, 1.0),
+            ("heuristic", configured.heuristic, 1.0),
+        ]
+        .into_iter()
+        .filter_map(|(name, weight, gate)| {
+            let value = candidate.signals.norm.get(name).copied()?;
+            let effective = weight * gate;
+            (effective > 0.0).then_some((name, effective, value))
+        })
+        .collect::<Vec<_>>();
+        let total = weighted.iter().map(|(_, weight, _)| weight).sum::<f64>();
+        if total <= 0.0 {
+            candidate.utility = None;
+            continue;
+        }
+        candidate.signals.weights = weighted
+            .iter()
+            .map(|(name, weight, _)| ((*name).to_string(), weight / total))
+            .collect();
+        candidate.utility = Some(
+            weighted
+                .iter()
+                .map(|(_, weight, value)| weight / total * value)
+                .sum::<f64>()
+                * 100.0,
+        );
+    }
+}
+
+fn ranked_indices(candidates: &[Candidate], indices: &[usize]) -> Vec<usize> {
+    let mut sorted = indices.to_vec();
+    sorted.sort_by(|left, right| {
+        candidates[*right]
+            .utility
+            .unwrap_or(f64::NEG_INFINITY)
+            .total_cmp(&candidates[*left].utility.unwrap_or(f64::NEG_INFINITY))
+            .then_with(|| {
+                candidates[*left]
+                    .article
+                    .id
+                    .cmp(&candidates[*right].article.id)
+            })
+    });
+    sorted
+}
+
+#[derive(Debug)]
+struct Cluster {
+    id: i64,
+    leader: usize,
+}
+
+/// Rank the admitted deep set and leave only the diversified shortlist at
+/// `stage = shortlisted`. All deep-set articles receive ranks and clusters.
+pub fn shortlist(
+    candidates: &mut [Candidate],
+    embeddings: &HashMap<ArticleId, Vec<f32>>,
+    ranking: &RankingConfig,
+) -> RankSummary {
+    let deep = candidates
+        .iter()
+        .enumerate()
+        .filter(|(_, candidate)| matches!(candidate.stage.as_str(), "admitted" | "assessed"))
+        .map(|(index, _)| index)
+        .collect::<Vec<_>>();
+    calculate_utility_for(candidates, &deep, &ranking.weights.utility);
+    let sorted = ranked_indices(candidates, &deep);
+    for (rank, index) in sorted.iter().enumerate() {
+        candidates[*index].rank_utility = Some(rank as i64 + 1);
+        candidates[*index].cluster = None;
+        candidates[*index].cluster_rank = None;
+    }
+
+    let mut clusters = Vec::<Cluster>::new();
+    let mut members: HashMap<i64, Vec<usize>> = HashMap::new();
+    for index in &sorted {
+        let assigned = embeddings
+            .get(&candidates[*index].article.id)
+            .and_then(|vector| {
+                clusters.iter().find_map(|cluster| {
+                    let leader_id = candidates[cluster.leader].article.id;
+                    let leader = embeddings.get(&leader_id)?;
+                    dot(vector, leader)
+                        .ok()
+                        .filter(|cosine| *cosine >= ranking.diversity.cluster_threshold)
+                        .map(|_| cluster.id)
+                })
+            });
+        let cluster_id = assigned.unwrap_or_else(|| {
+            let id = clusters.len() as i64 + 1;
+            clusters.push(Cluster { id, leader: *index });
+            id
+        });
+        let cluster_members = members.entry(cluster_id).or_default();
+        cluster_members.push(*index);
+        candidates[*index].cluster = Some(cluster_id);
+        candidates[*index].cluster_rank = Some(cluster_members.len() as i64);
+    }
+
+    let protected = sorted
+        .iter()
+        .take(ranking.diversity.utility_protected)
+        .copied()
+        .collect::<HashSet<_>>();
+    let mut admitted = HashSet::new();
+    let mut admitted_per_cluster = HashMap::<i64, usize>::new();
+    let admit = |index: usize, admitted: &mut HashSet<usize>, counts: &mut HashMap<i64, usize>| {
+        if admitted.insert(index)
+            && let Some(cluster) = candidates[index].cluster
+        {
+            *counts.entry(cluster).or_default() += 1;
+        }
+    };
+
+    for index in &sorted {
+        if protected.contains(index) || candidates[*index].auto_include {
+            admit(*index, &mut admitted, &mut admitted_per_cluster);
+        }
+    }
+    for index in sorted
+        .iter()
+        .filter(|index| candidates[**index].exploration)
+        .take(3)
+    {
+        if admitted.len() >= ranking.shortlist_keep {
+            break;
+        }
+        admit(*index, &mut admitted, &mut admitted_per_cluster);
+    }
+
+    let target = ranking.shortlist_keep.max(admitted.len());
+    let mut suppressed_at_base_cap = HashSet::new();
+    admit_under_cap(
+        candidates,
+        &sorted,
+        target,
+        ranking.diversity.per_cluster_cap,
+        &mut admitted,
+        &mut admitted_per_cluster,
+        Some(&mut suppressed_at_base_cap),
+    );
+    if admitted.len() < target {
+        admit_under_cap(
+            candidates,
+            &sorted,
+            target,
+            3,
+            &mut admitted,
+            &mut admitted_per_cluster,
+            None,
+        );
+    }
+    if admitted.len() < target {
+        for index in &sorted {
+            if admitted.len() >= target {
+                break;
+            }
+            admit(*index, &mut admitted, &mut admitted_per_cluster);
+        }
+    }
+
+    for index in deep {
+        if admitted.contains(&index) {
+            candidates[index].stage = "shortlisted".into();
+            candidates[index].excluded_reason = None;
+        } else {
+            // The stage stays where the article stopped (`admitted` when the
+            // deep assessment never happened, else `assessed`).
+            candidates[index].excluded_reason = Some(
+                if suppressed_at_base_cap.contains(&index) {
+                    "cluster_suppressed"
+                } else {
+                    "shortlist_cap"
+                }
+                .into(),
+            );
+        }
+    }
+    RankSummary {
+        shortlisted: admitted.len(),
+        clusters: clusters.len(),
+    }
+}
+
+#[allow(clippy::too_many_arguments)]
+fn admit_under_cap(
+    candidates: &[Candidate],
+    sorted: &[usize],
+    target: usize,
+    cap: usize,
+    admitted: &mut HashSet<usize>,
+    admitted_per_cluster: &mut HashMap<i64, usize>,
+    mut suppressed: Option<&mut HashSet<usize>>,
+) {
+    for index in sorted {
+        if admitted.len() >= target || admitted.contains(index) {
+            continue;
+        }
+        let Some(cluster) = candidates[*index].cluster else {
+            continue;
+        };
+        if admitted_per_cluster.get(&cluster).copied().unwrap_or(0) >= cap {
+            if let Some(suppressed) = suppressed.as_deref_mut() {
+                suppressed.insert(*index);
+            }
+            continue;
+        }
+        admitted.insert(*index);
+        *admitted_per_cluster.entry(cluster).or_default() += 1;
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::config::DiversityConfig;
+    use crate::curate::prefilter::tests::article;
+    use crate::curate::signals::Signals;
+    use crate::types::{Deep, Facets};
+
+    fn candidate(id: i64, utility_hint: f64) -> Candidate {
+        let mut candidate = Candidate::new(article(id, &format!("article {id}"), 800), false);
+        candidate.stage = "assessed".into();
+        candidate.signals = Signals {
+            heuristic: Some(utility_hint),
+            ..Signals::default()
+        };
+        candidate.assessment.deep = Some(Deep {
+            quality: utility_hint,
+            fit: utility_hint,
+            category: Some("Top Stories".into()),
+            rationale: "specific".into(),
+            paywalled_guess: false,
+            facets: Facets::default(),
+            model: "mock".into(),
+            prompt_version: 1,
+            assessed_at: "2026-09-02T00:00:00Z".parse().expect("timestamp"),
+        });
+        candidate
+    }
+
+    fn vector(angle: f32) -> Vec<f32> {
+        vec![angle.cos(), angle.sin()]
+    }
+
+    fn ranking(keep: usize, protected: usize, cap: usize, threshold: f64) -> RankingConfig {
+        RankingConfig {
+            shortlist_keep: keep,
+            diversity: DiversityConfig {
+                cluster_threshold: threshold,
+                per_cluster_cap: cap,
+                utility_protected: protected,
+            },
+            ..RankingConfig::default()
+        }
+    }
+
+    #[test]
+    fn utility_renormalizes_present_signals_and_gates_learned_ones() {
+        let mut a = candidate(1, 8.0);
+        let mut b = candidate(2, 4.0);
+        a.signals.interest = Some(1.0);
+        b.signals.interest = None;
+        a.signals.knn = Some(0.9);
+        a.signals.knn_gate = 0.5;
+        let mut values = vec![a, b];
+        calculate_utility(&mut values, &UtilityWeights::default());
+        let [a, b] = values.as_slice() else {
+            panic!("two values")
+        };
+        for candidate in [&a, &b] {
+            assert!((candidate.signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
+            assert!(candidate.utility.is_some());
+        }
+        assert!(!b.signals.weights.contains_key("interest"));
+        assert!(
+            (a.signals.weights["knn"] / a.signals.weights["quality"] - (0.15 * 0.5) / 0.40).abs()
+                < 1e-9
+        );
+    }
+
+    #[test]
+    fn duplicates_cluster_and_the_third_is_suppressed() {
+        let ranking = ranking(3, 0, 2, 0.85);
+        let mut candidates = vec![
+            candidate(1, 9.0),
+            candidate(2, 8.0),
+            candidate(3, 7.0),
+            candidate(4, 6.0),
+        ];
+        let embeddings = HashMap::from([
+            (1, vector(0.0)),
+            (2, vector(0.1)),
+            (3, vector(0.2)),
+            (4, vector(2.0)),
+        ]);
+        let summary = shortlist(&mut candidates, &embeddings, &ranking);
+        assert_eq!(summary.shortlisted, 3);
+        assert_eq!(candidates[0].cluster, candidates[1].cluster);
+        assert_eq!(candidates[1].cluster, candidates[2].cluster);
+        assert_eq!(
+            candidates[2].excluded_reason.as_deref(),
+            Some("cluster_suppressed")
+        );
+    }
+
+    #[test]
+    fn protected_items_survive_and_count_toward_the_cap() {
+        let ranking = ranking(3, 2, 1, 0.85);
+        let mut candidates = vec![
+            candidate(1, 9.0),
+            candidate(2, 8.0),
+            candidate(3, 7.0),
+            candidate(4, 6.0),
+        ];
+        let embeddings = HashMap::from([
+            (1, vector(0.0)),
+            (2, vector(0.05)),
+            (3, vector(0.1)),
+            (4, vector(2.0)),
+        ]);
+        shortlist(&mut candidates, &embeddings, &ranking);
+        assert_eq!(candidates[0].stage, "shortlisted");
+        assert_eq!(candidates[1].stage, "shortlisted");
+        assert_ne!(candidates[2].stage, "shortlisted");
+    }
+
+    #[test]
+    fn bridge_case_uses_leaders_not_transitive_components() {
+        let ranking = ranking(3, 0, 2, 0.80);
+        let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
+        // A at 0°, B at 72°, C at 36°: A~C and B~C, but A not~B.
+        let embeddings =
+            HashMap::from([(1, vector(0.0)), (2, vector(1.2566)), (3, vector(0.6283))]);
+        let summary = shortlist(&mut candidates, &embeddings, &ranking);
+        assert_eq!(summary.clusters, 2);
+        assert_eq!(candidates[0].cluster, candidates[2].cluster);
+        assert_ne!(candidates[0].cluster, candidates[1].cluster);
+    }
+
+    #[test]
+    fn missing_embeddings_are_singletons_and_never_suppressed() {
+        let ranking = ranking(3, 0, 1, 0.85);
+        let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
+        shortlist(&mut candidates, &HashMap::new(), &ranking);
+        assert!(
+            candidates
+                .iter()
+                .all(|candidate| candidate.stage == "shortlisted")
+        );
+        assert_eq!(
+            candidates
+                .iter()
+                .filter_map(|candidate| candidate.cluster)
+                .collect::<HashSet<_>>()
+                .len(),
+            3
+        );
+    }
+
+    #[test]
+    fn caps_relax_to_three_then_uncapped_when_the_shortlist_is_short() {
+        // Six near-duplicates, keep 5: cap 2 admits two, cap 3 admits a third,
+        // and the uncapped pass fills the remaining two slots in utility order.
+        let ranking = ranking(5, 0, 2, 0.85);
+        let mut candidates = (1..=6)
+            .map(|id| candidate(id, 10.0 - id as f64))
+            .collect::<Vec<_>>();
+        let embeddings = (1..=6)
+            .map(|id| (id, vector(0.01 * id as f32)))
+            .collect::<HashMap<_, _>>();
+        let summary = shortlist(&mut candidates, &embeddings, &ranking);
+        assert_eq!(summary.clusters, 1);
+        assert_eq!(summary.shortlisted, 5);
+        let shortlisted = candidates
+            .iter()
+            .filter(|candidate| candidate.stage == "shortlisted")
+            .map(|candidate| candidate.article.id)
+            .collect::<Vec<_>>();
+        assert_eq!(shortlisted, vec![1, 2, 3, 4, 5], "filled in utility order");
+        assert_eq!(candidates[5].stage, "assessed");
+        assert_eq!(
+            candidates[5].excluded_reason.as_deref(),
+            Some("cluster_suppressed")
+        );
+        assert_eq!(
+            candidates
+                .iter()
+                .map(|c| c.rank_utility)
+                .collect::<Vec<_>>(),
+            (1..=6).map(Some).collect::<Vec<_>>()
+        );
+        assert_eq!(
+            candidates
+                .iter()
+                .map(|c| c.cluster_rank)
+                .collect::<Vec<_>>(),
+            (1..=6).map(Some).collect::<Vec<_>>()
+        );
+    }
+
+    #[test]
+    fn shortlist_cap_is_the_reason_beyond_the_keep() {
+        let ranking = ranking(2, 0, 2, 0.85);
+        let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
+        shortlist(&mut candidates, &HashMap::new(), &ranking);
+        assert_eq!(candidates[0].stage, "shortlisted");
+        assert_eq!(candidates[1].stage, "shortlisted");
+        assert_eq!(candidates[2].stage, "assessed");
+        assert_eq!(
+            candidates[2].excluded_reason.as_deref(),
+            Some("shortlist_cap")
+        );
+    }
+
+    #[test]
+    fn exploration_picks_get_up_to_three_reserved_slots() {
+        // Keep 4 with the four best by utility being ordinary articles: three
+        // exploration picks are still reserved seats, the fourth is not.
+        let ranking = ranking(4, 0, 2, 0.85);
+        let mut candidates = (1..=8)
+            .map(|id| candidate(id, 10.0 - id as f64))
+            .collect::<Vec<_>>();
+        for candidate in candidates.iter_mut().skip(4) {
+            candidate.exploration = true;
+        }
+        let summary = shortlist(&mut candidates, &HashMap::new(), &ranking);
+        assert_eq!(summary.shortlisted, 4);
+        let shortlisted = candidates
+            .iter()
+            .filter(|candidate| candidate.stage == "shortlisted")
+            .map(|candidate| candidate.article.id)
+            .collect::<Vec<_>>();
+        assert_eq!(shortlisted, vec![1, 5, 6, 7]);
+        assert_eq!(
+            candidates[7].excluded_reason.as_deref(),
+            Some("shortlist_cap")
+        );
+    }
+
+    #[test]
+    fn auto_includes_are_admitted_regardless_and_count_toward_their_cluster() {
+        let ranking = ranking(2, 0, 1, 0.85);
+        let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 1.0)];
+        candidates[2].auto_include = true;
+        let embeddings = HashMap::from([(1, vector(0.0)), (2, vector(2.0)), (3, vector(0.05))]);
+        let summary = shortlist(&mut candidates, &embeddings, &ranking);
+        assert_eq!(summary.shortlisted, 2);
+        assert_eq!(candidates[2].stage, "shortlisted", "auto-include survives");
+        // The auto-include filled its cluster's single seat, so the stronger
+        // near-duplicate is suppressed and the unrelated article gets the slot.
+        assert_eq!(candidates[0].cluster, candidates[2].cluster);
+        assert_eq!(candidates[0].stage, "assessed");
+        assert_eq!(
+            candidates[0].excluded_reason.as_deref(),
+            Some("cluster_suppressed")
+        );
+        assert_eq!(candidates[1].stage, "shortlisted");
+    }
+
+    #[test]
+    fn unassessed_articles_rank_on_present_signals_and_keep_their_stage() {
+        // DeepSeek down: no quality/fit anywhere, utility comes from what is present.
+        let ranking = ranking(2, 0, 2, 0.85);
+        let mut candidates = vec![candidate(1, 3.0), candidate(2, 6.0), candidate(3, 9.0)];
+        for candidate in &mut candidates {
+            candidate.assessment.deep = None;
+            candidate.stage = "admitted".into();
+        }
+        candidates[0].assessment.triage = Some(crate::types::Triage {
+            interest: 9.0,
+            kind: "essay".into(),
+            why: "promising".into(),
+            model: "mock".into(),
+            prompt_version: 1,
+            assessed_at: "2026-09-02T00:00:00Z".parse().expect("timestamp"),
+        });
+        let summary = shortlist(&mut candidates, &HashMap::new(), &ranking);
+        assert_eq!(summary.shortlisted, 2);
+        for candidate in &candidates {
+            assert!(candidate.utility.is_some(), "scored on present signals");
+            assert!(!candidate.signals.weights.contains_key("quality"));
+            assert!(!candidate.signals.weights.contains_key("fit"));
+            assert!((candidate.signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
+        }
+        // Triage (0.05) outweighs heuristic (0.02): the triaged article with
+        // the weakest heuristic overtakes the middle one.
+        assert_eq!(candidates[2].rank_utility, Some(1));
+        assert_eq!(candidates[0].rank_utility, Some(2));
+        assert_eq!(candidates[1].rank_utility, Some(3));
+        assert_eq!(
+            candidates[1].stage, "admitted",
+            "never assessed, so not `assessed`"
+        );
+        assert_eq!(
+            candidates[1].excluded_reason.as_deref(),
+            Some("shortlist_cap")
+        );
+    }
+
+    #[test]
+    fn percentiles_are_taken_over_the_deep_set_only() {
+        // The eligible-but-not-admitted article has the strongest heuristic;
+        // it must not shift the deep set's percentiles or receive a utility.
+        let ranking = ranking(10, 0, 2, 0.85);
+        let mut candidates = vec![candidate(1, 5.0), candidate(2, 5.0), candidate(3, 5.0)];
+        candidates[2].stage = "triaged".into();
+        candidates[2].excluded_reason = Some("not_admitted".into());
+        candidates[2].signals.heuristic = Some(99.0);
+        candidates[1].signals.heuristic = Some(5.0);
+        shortlist(&mut candidates, &HashMap::new(), &ranking);
+        assert_eq!(
+            candidates[0].signals.norm["heuristic"], 0.5,
+            "ties share a percentile"
+        );
+        assert_eq!(candidates[1].signals.norm["heuristic"], 0.5);
+        assert!(candidates[2].utility.is_none());
+        assert!(candidates[2].rank_utility.is_none());
+    }
+}

src/curate/score.rs

@@ -1,593 +0,0 @@
-//! Stage A — batched LLM scoring (spec §3.6).
-//!
-//! Batches of `deepseek.score_batch_size` articles per request. Per article we
-//! send title, source feed, author, word count, social stats, sources list and a
-//! ~200-word excerpt; the model returns one JSON object per article.
-//!
-//! Parsing is deliberately forgiving: one malformed item must not cost us the
-//! other eleven, and a failed batch must not fail the run.
-
-use std::collections::HashMap;
-use std::fmt::Write as _;
-
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-
-use super::llm::{LlmClient, LlmError, strip_code_fence};
-use super::{html_to_text, truncate_words};
-use crate::types::{ArticleId, LlmScore, ScoredArticle, SourceKind};
-
-/// Words of article text sent per candidate in stage A (§3.6).
-pub const EXCERPT_WORDS: usize = 200;
-
-/// One element of the stage-A JSON response (§3.6).
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct ScoreItem {
-    pub id: ArticleId,
-    /// 0–10.
-    pub score: f64,
-    pub category: String,
-    /// ≤ 20 words.
-    #[serde(default)]
-    pub rationale: String,
-    #[serde(default)]
-    pub is_paywalled_guess: bool,
-}
-
-impl From<ScoreItem> for LlmScore {
-    fn from(i: ScoreItem) -> Self {
-        LlmScore {
-            score: i.score,
-            category: i.category,
-            rationale: i.rationale,
-            is_paywalled_guess: i.is_paywalled_guess,
-        }
-    }
-}
-
-/// Envelope the model is asked to return (`{"articles": [...]}`).
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct ScoreResponse {
-    #[serde(default)]
-    pub articles: Vec<ScoreItem>,
-}
-
-/// The invariant instruction block for stage A. Everything article-specific goes
-/// in the per-batch tail so this prefix stays cacheable (§3.6).
-pub const SCORE_INSTRUCTIONS: &str = "\
-TASK: score a batch of candidate articles for today's issue of The Daily EPUB.
-
-Judge each article against the reader profile in your system prompt — not against \
-a general audience, and not against what is objectively newsworthy.
-
-Return one object per input article with these fields:
-  \"id\"                  integer, copied exactly from the input
-  \"score\"               number 0-10, the rubric below
-  \"category\"            one short label from the palette below
-  \"rationale\"           at most 20 words, concrete, no hedging, no restating the title
-  \"is_paywalled_guess\"  true when the text looks truncated, teaser-like or paywalled
-
-SCORING RUBRIC — calibrate hard; a normal day averages about 4, and a 9 should \
-appear a couple of times a week, not a couple of times a day:
-  9-10  Exceptional. Original reporting, a deep technical dive, or an essay he \
-will still be thinking about next week. Evident effort and a real point of view.
-  7-8   Strong. A well-made long-form piece squarely in his interests, or an \
-outstanding piece outside them.
-  5-6   Worth a slot on a thin day. Solid, useful, a little thin or a little \
-familiar.
-  3-4   Marginal. Competent news-of-the-day, short posts, incremental updates, \
-good writing about an over-covered story.
-  1-2   Weak. Announcements, changelogs and release notes, link roundups, \
-listicles, rewrites of a story available at the source, thin AI-industry churn.
-  0     Unusable. Press releases, sponsored content, engagement bait, spam, \
-pure crypto promotion, or an entry with no readable body.
-
-CALIBRATION NOTES
-- Length alone is not quality; padding scores worse than a tight short piece. But \
-between two equally good pieces, prefer the one with more substance.
-- Social proof is evidence, not a verdict: hundreds of HN points mean a critical \
-audience read it; a quiet post from a good blog can still outrank it.
-- \"came via scour\" means the story already matched one of his standing \
-interests. \"came via hn_frontpage\" means it cleared HN's front page.
-- Boston/New England local stories and ultra-niche community news get a genuine \
-lift — this paper wants them.
-- Wire-service world/US news should score low here: the World Briefing section \
-covers that separately.
-- Excerpt-only or paywalled text is a real cost to the reader; score it lower \
-unless the piece is clearly excellent.
-
-Return JSON exactly in this shape, with one entry per input article and nothing \
-else:
-{\"articles\": [{\"id\": 123, \"score\": 7.5, \"category\": \"Tech & Engineering\", \
-\"rationale\": \"first-hand account of migrating 40TB off Postgres\", \
-\"is_paywalled_guess\": false}]}";
-
-/// Render the user prompt for one batch (§3.6).
-pub fn build_batch_prompt(batch: &[ScoredArticle], sections: &[String]) -> String {
-    let mut prompt = String::with_capacity(4096 + batch.len() * 1500);
-    prompt.push_str(SCORE_INSTRUCTIONS);
-    let _ = write!(
-        prompt,
-        "\n\nCATEGORY PALETTE (use one of these exact strings): {}\n\nARTICLES ({} in this batch)\n",
-        sections.join(" | "),
-        batch.len()
-    );
-    for candidate in batch {
-        prompt.push('\n');
-        prompt.push_str(&render_candidate(candidate));
-    }
-    prompt
-}
-
-/// One article's block in the stage-A prompt (§3.6).
-fn render_candidate(candidate: &ScoredArticle) -> String {
-    let a = &candidate.article;
-    let mut block = String::with_capacity(1500);
-    let _ = writeln!(block, "--- id: {}", a.id);
-    let _ = writeln!(block, "title: {}", a.title.trim());
-    let _ = writeln!(
-        block,
-        "feed: {}{}",
-        if a.feed_title.is_empty() {
-            "unknown"
-        } else {
-            a.feed_title.trim()
-        },
-        a.category
-            .as_deref()
-            .filter(|c| !c.is_empty())
-            .map(|c| format!(" (category: {c})"))
-            .unwrap_or_default()
-    );
-    if let Some(author) = a.author.as_deref().filter(|s| !s.trim().is_empty()) {
-        let _ = writeln!(block, "author: {}", author.trim());
-    }
-    let _ = writeln!(
-        block,
-        "length: {} words (~{} min read){}",
-        a.word_count,
-        a.reading_minutes(),
-        if a.excerpt_only {
-            " [EXCERPT ONLY — full text unavailable]"
-        } else {
-            ""
-        }
-    );
-    let _ = writeln!(block, "social: {}", social_line(candidate));
-    let _ = writeln!(block, "came via: {}", sources_line(candidate));
-    let excerpt = truncate_words(&html_to_text(&a.content_html), EXCERPT_WORDS);
-    let _ = writeln!(
-        block,
-        "excerpt: {}",
-        if excerpt.is_empty() {
-            "(no body text extracted)"
-        } else {
-            &excerpt
-        }
-    );
-    block
-}
-
-fn social_line(candidate: &ScoredArticle) -> String {
-    if candidate.article.social.is_empty() {
-        return "none found".into();
-    }
-    let mut parts: Vec<String> = candidate
-        .article
-        .social
-        .iter()
-        .map(|s| {
-            format!(
-                "{} {} points / {} comments",
-                s.source.display_name(),
-                s.score,
-                s.num_comments
-            )
-        })
-        .collect();
-    parts.push(format!("composite {:.2}", candidate.social_score));
-    parts.join("; ")
-}
-
-fn sources_line(candidate: &ScoredArticle) -> String {
-    let mut kinds: Vec<&str> = candidate
-        .article
-        .sources
-        .iter()
-        .map(|s| match s.kind {
-            SourceKind::Scour => "scour",
-            SourceKind::HnFrontpage => "hn_frontpage",
-            SourceKind::Lobsters => "lobsters",
-            SourceKind::Reddit => "reddit",
-            SourceKind::Feed => "feed",
-        })
-        .collect();
-    kinds.sort_unstable();
-    kinds.dedup();
-    if candidate.auto_include {
-        kinds.push("always-include feed (cannot be dropped)");
-    }
-    if kinds.is_empty() {
-        "feed".into()
-    } else {
-        kinds.join(", ")
-    }
-}
-
-// ---------------------------------------------------------------------------
-// Response parsing (§3.6: tolerate anything the model does to us)
-// ---------------------------------------------------------------------------
-
-/// Keys the model might wrap the array in, in preference order.
-const ARRAY_KEYS: &[&str] = &["articles", "scores", "results", "items", "data"];
-
-/// Parse a stage-A response leniently: missing optional fields default, scores
-/// are clamped to 0–10, and malformed items are skipped with a warning (§3.6).
-pub fn parse_score_response(raw: &str) -> Vec<ScoreItem> {
-    let cleaned = strip_code_fence(raw);
-    let value: Value = match serde_json::from_str(cleaned) {
-        Ok(v) => v,
-        Err(e) => {
-            tracing::warn!(error = %e, "stage A response was not JSON at all");
-            return Vec::new();
-        }
-    };
-
-    let array = match &value {
-        Value::Array(items) => Some(items),
-        Value::Object(map) => ARRAY_KEYS
-            .iter()
-            .find_map(|k| map.get(*k).and_then(Value::as_array))
-            // Some models return {"1234": {...}} or a single bare object.
-            .or_else(|| map.values().find_map(Value::as_array)),
-        _ => None,
-    };
-    let Some(array) = array else {
-        tracing::warn!("stage A response contained no array of scores");
-        return Vec::new();
-    };
-
-    let mut out = Vec::with_capacity(array.len());
-    let mut skipped = 0usize;
-    for item in array {
-        match parse_item(item) {
-            Some(parsed) => out.push(parsed),
-            None => {
-                skipped += 1;
-                tracing::warn!(item = %truncate_debug(item), "skipping malformed stage A item");
-            }
-        }
-    }
-    if skipped > 0 {
-        tracing::warn!(skipped, kept = out.len(), "stage A items were dropped");
-    }
-    out
-}
-
-fn parse_item(item: &Value) -> Option<ScoreItem> {
-    let obj = item.as_object()?;
-    let id = obj.get("id").and_then(as_i64_lenient)?;
-    let score = obj
-        .get("score")
-        .and_then(as_f64_lenient)
-        .or_else(|| obj.get("rating").and_then(as_f64_lenient))?;
-    Some(ScoreItem {
-        id,
-        score: score.clamp(0.0, 10.0),
-        category: obj
-            .get("category")
-            .and_then(Value::as_str)
-            .unwrap_or_default()
-            .trim()
-            .to_string(),
-        rationale: obj
-            .get("rationale")
-            .or_else(|| obj.get("reason"))
-            .and_then(Value::as_str)
-            .unwrap_or_default()
-            .trim()
-            .to_string(),
-        is_paywalled_guess: obj
-            .get("is_paywalled_guess")
-            .or_else(|| obj.get("paywalled"))
-            .and_then(as_bool_lenient)
-            .unwrap_or(false),
-    })
-}
-
-fn as_i64_lenient(v: &Value) -> Option<i64> {
-    v.as_i64()
-        .or_else(|| v.as_f64().map(|f| f as i64))
-        .or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
-}
-
-fn as_f64_lenient(v: &Value) -> Option<f64> {
-    v.as_f64()
-        .or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
-        .filter(|f| f.is_finite())
-}
-
-fn as_bool_lenient(v: &Value) -> Option<bool> {
-    v.as_bool().or_else(|| match v.as_str()?.trim() {
-        "true" | "yes" => Some(true),
-        "false" | "no" => Some(false),
-        _ => None,
-    })
-}
-
-fn truncate_debug(v: &Value) -> String {
-    v.to_string().chars().take(160).collect()
-}
-
-// ---------------------------------------------------------------------------
-// Stage driver
-// ---------------------------------------------------------------------------
-
-/// Score every candidate, filling in [`ScoredArticle::llm`] (§3.6).
-///
-/// Batches that fail are logged and left unscored rather than aborting the run.
-/// Returns how many candidates came back with a score.
-pub async fn score_all(
-    llm: &LlmClient,
-    candidates: &mut [ScoredArticle],
-    batch_size: usize,
-    sections: &[String],
-    temperature: f32,
-) -> Result<usize, LlmError> {
-    if candidates.is_empty() {
-        return Ok(0);
-    }
-    let batch_size = batch_size.max(1);
-    let batches = candidates.len().div_ceil(batch_size);
-    let mut scores: HashMap<ArticleId, LlmScore> = HashMap::with_capacity(candidates.len());
-
-    for (n, batch) in candidates.chunks(batch_size).enumerate() {
-        if let Err(e) = llm.meter.check_budget() {
-            tracing::error!(
-                error = %e,
-                batch = n + 1,
-                of = batches,
-                unscored = candidates.len() - scores.len(),
-                "COST CEILING HIT during stage A scoring — remaining batches skipped; \
-                 the lineup will fall back to heuristic ranking for them"
-            );
-            break;
-        }
-        let prompt = build_batch_prompt(batch, sections);
-        tracing::debug!(
-            batch = n + 1,
-            of = batches,
-            articles = batch.len(),
-            approx_tokens = super::approx_tokens(&prompt),
-            "stage A request"
-        );
-        match llm.complete(&prompt, temperature, true).await {
-            Ok(raw) => {
-                let items = parse_score_response(&raw);
-                if items.is_empty() {
-                    tracing::warn!(
-                        batch = n + 1,
-                        of = batches,
-                        "stage A batch returned no scores"
-                    );
-                }
-                for item in items {
-                    scores.insert(item.id, item.into());
-                }
-            }
-            Err(e) => {
-                tracing::warn!(batch = n + 1, of = batches, error = %e,
-                    "stage A batch failed; its articles stay unscored");
-            }
-        }
-    }
-
-    let mut applied = 0usize;
-    for candidate in candidates.iter_mut() {
-        if let Some(score) = scores.remove(&candidate.article.id) {
-            candidate.llm = Some(score);
-            applied += 1;
-        }
-    }
-    if !scores.is_empty() {
-        tracing::warn!(
-            unknown_ids = scores.len(),
-            "stage A returned scores for ids that were not in the batch"
-        );
-    }
-    Ok(applied)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use crate::config::DeepseekConfig;
-    use crate::curate::llm::{MockBackend, UsageMeter};
-    use crate::curate::prefilter::tests::{article, via, with_social};
-    use crate::types::TokenUsage;
-    use std::sync::Arc;
-
-    const BATCH_FIXTURE: &str = include_str!(concat!(
-        env!("CARGO_MANIFEST_DIR"),
-        "/tests/fixtures/deepseek_score_batch.json"
-    ));
-    const MESSY_FIXTURE: &str = include_str!(concat!(
-        env!("CARGO_MANIFEST_DIR"),
-        "/tests/fixtures/deepseek_score_batch_messy.json"
-    ));
-
-    fn sections() -> Vec<String> {
-        crate::config::CurationConfig::default().sections
-    }
-
-    fn candidate(id: i64, title: &str, words: i64) -> ScoredArticle {
-        ScoredArticle {
-            article: article(id, title, words),
-            prefilter_score: 50.0,
-            social_score: 0.0,
-            feed_prior: 0.5,
-            llm: None,
-            auto_include: false,
-        }
-    }
-
-    #[test]
-    fn batch_prompt_carries_every_documented_signal() {
-        let mut c = candidate(12, "Migrating 40TB off Postgres", 3200);
-        c.article = via(
-            with_social(c.article, 342, 210),
-            SourceKind::HnFrontpage,
-            9001,
-        );
-        c.social_score = c.article.social_score();
-        c.auto_include = true;
-        let prompt = build_batch_prompt(&[c], &sections());
-
-        assert!(prompt.starts_with(SCORE_INSTRUCTIONS));
-        assert!(prompt.contains("--- id: 12"));
-        assert!(prompt.contains("title: Migrating 40TB off Postgres"));
-        assert!(prompt.contains("feed: Some Blog (category: Tech)"));
-        assert!(prompt.contains("author: A. Writer"));
-        assert!(prompt.contains("length: 3200 words"));
-        assert!(prompt.contains("HN 342 points / 210 comments"));
-        assert!(prompt.contains("hn_frontpage"));
-        assert!(prompt.contains("always-include feed"));
-        assert!(prompt.contains("excerpt: word word"));
-        assert!(prompt.contains("Tech & Engineering"));
-        // The excerpt is capped.
-        let excerpt_line = prompt
-            .lines()
-            .find(|l| l.starts_with("excerpt:"))
-            .expect("excerpt line");
-        assert!(excerpt_line.split_whitespace().count() <= EXCERPT_WORDS + 2);
-    }
-
-    #[test]
-    fn parses_a_realistic_deepseek_batch() {
-        let items = parse_score_response(BATCH_FIXTURE);
-        assert_eq!(items.len(), 4);
-        assert_eq!(items[0].id, 101);
-        assert!((items[0].score - 8.5).abs() < 1e-9);
-        assert_eq!(items[0].category, "Tech & Engineering");
-        assert!(items[0].rationale.split_whitespace().count() <= 20);
-        assert!(!items[0].is_paywalled_guess);
-        assert!(items[3].is_paywalled_guess);
-        let score: LlmScore = items[0].clone().into();
-        assert_eq!(score.category, "Tech & Engineering");
-    }
-
-    #[test]
-    fn parsing_survives_everything_a_model_might_do() {
-        let items = parse_score_response(MESSY_FIXTURE);
-        let ids: Vec<ArticleId> = items.iter().map(|i| i.id).collect();
-        // 201 fine; 202 string score clamped; 203 missing rationale/category;
-        // 204 out-of-range clamped; the two malformed entries are dropped.
-        assert_eq!(ids, vec![201, 202, 203, 204]);
-        assert!((items[1].score - 6.0).abs() < 1e-9);
-        assert_eq!(items[2].rationale, "");
-        assert_eq!(items[2].category, "");
-        assert!(
-            (items[3].score - 10.0).abs() < 1e-9,
-            "clamped to the 0-10 range"
-        );
-        assert!(items.iter().all(|i| (0.0..=10.0).contains(&i.score)));
-    }
-
-    #[test]
-    fn parsing_tolerates_fences_arrays_and_junk() {
-        assert_eq!(
-            parse_score_response("```json\n{\"articles\":[{\"id\":1,\"score\":5}]}\n```").len(),
-            1
-        );
-        assert_eq!(parse_score_response("[{\"id\": 2, \"score\": 3}]").len(), 1);
-        assert_eq!(
-            parse_score_response("{\"results\":[{\"id\":3,\"score\":\"4.5\"}]}")[0].score,
-            4.5
-        );
-        assert!(parse_score_response("I'm sorry, I can't do that").is_empty());
-        assert!(parse_score_response("{\"articles\": {}}").is_empty());
-    }
-
-    fn client(backend: Arc<MockBackend>, limit_usd: f64) -> LlmClient {
-        LlmClient::with_backend(
-            "deepseek-v4-flash",
-            "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), limit_usd),
-            backend,
-        )
-    }
-
-    #[tokio::test]
-    async fn scores_are_applied_batch_by_batch() {
-        let backend = Arc::new(MockBackend::new());
-        backend.push(
-            r#"{"articles":[{"id":1,"score":8,"category":"Tech & Engineering","rationale":"good"},
-                            {"id":2,"score":2,"category":"Niche Corner","rationale":"thin"}]}"#,
-            TokenUsage::default(),
-        );
-        backend.push(
-            r#"{"articles":[{"id":3,"score":6.5,"category":"Culture & Essays","rationale":"solid"}]}"#,
-            TokenUsage::default(),
-        );
-        let llm = client(Arc::clone(&backend), 2.0);
-
-        let mut candidates = vec![
-            candidate(1, "One", 1000),
-            candidate(2, "Two", 1000),
-            candidate(3, "Three", 1000),
-        ];
-        let scored = score_all(&llm, &mut candidates, 2, &sections(), 0.3)
-            .await
-            .expect("scoring");
-        assert_eq!(scored, 3);
-        assert_eq!(backend.calls(), 2, "batched by score_batch_size");
-        assert_eq!(candidates[0].llm.as_ref().map(|l| l.score), Some(8.0));
-        assert_eq!(candidates[2].llm.as_ref().map(|l| l.score), Some(6.5));
-        // combined_score now reflects the LLM verdict.
-        assert!(candidates[0].combined_score() > candidates[1].combined_score());
-    }
-
-    #[tokio::test]
-    async fn a_failed_batch_does_not_sink_the_run() {
-        let backend = Arc::new(MockBackend::new());
-        backend.push_error("500 upstream exploded");
-        backend.push(
-            r#"{"articles":[{"id":2,"score":7,"category":"Top Stories","rationale":"ok"}]}"#,
-            TokenUsage::default(),
-        );
-        let llm = client(Arc::clone(&backend), 2.0);
-        let mut candidates = vec![candidate(1, "One", 900), candidate(2, "Two", 900)];
-        let scored = score_all(&llm, &mut candidates, 1, &sections(), 0.3)
-            .await
-            .expect("scoring must not abort");
-        assert_eq!(scored, 1);
-        assert!(candidates[0].llm.is_none());
-        assert!(candidates[1].llm.is_some());
-    }
-
-    #[tokio::test]
-    async fn scoring_stops_when_the_budget_is_gone() {
-        let backend = Arc::new(MockBackend::new());
-        // First batch alone blows a $0.05 ceiling ($0.14 per 1M input tokens).
-        backend.push(
-            r#"{"articles":[{"id":1,"score":9,"category":"Top Stories","rationale":"great"}]}"#,
-            TokenUsage {
-                input_tokens: 1_000_000,
-                cached_tokens: 0,
-                output_tokens: 0,
-            },
-        );
-        backend.push(
-            r#"{"articles":[{"id":2,"score":9,"category":"Top Stories","rationale":"great"}]}"#,
-            TokenUsage::default(),
-        );
-        let llm = client(Arc::clone(&backend), 0.05);
-        let mut candidates = vec![candidate(1, "One", 900), candidate(2, "Two", 900)];
-        let scored = score_all(&llm, &mut candidates, 1, &sections(), 0.3)
-            .await
-            .expect("scoring");
-        assert_eq!(scored, 1, "only the first batch ran");
-        assert_eq!(backend.calls(), 1);
-        assert!(llm.meter.budget_exceeded());
-    }
-}

src/curate/select.rs

@@ -1,1000 +0,0 @@
-//! Stage B — lineup selection (spec §3.6).
-//!
-//! One call: send the top ~40 candidates by [`ScoredArticle::combined_score`] with
-//! their rationales; the model returns the final 15–25 picks, each with a section
-//! from the configured palette, an ordering, and exactly one `lead_story`.
-//!
-//! The model's answer is treated as a proposal, never as gospel: sections are
-//! validated against the palette, the lead is forced to be unique, auto-include
-//! feeds are re-inserted if they were dropped, and the size is clamped to
-//! `target_article_count ± 5`.
-
-use std::collections::{BTreeMap, HashMap, HashSet};
-use std::fmt::Write as _;
-
-use jiff::civil::Date;
-use serde::{Deserialize, Serialize};
-
-use super::llm::{LlmClient, LlmError, strip_code_fence};
-use super::{html_to_text, truncate_words};
-use crate::types::{ArticleId, Lineup, Pick, ScoredArticle, SourceKind, WORLD_BRIEFING_SECTION};
-
-/// How many candidates are offered to stage B (§3.6).
-pub const SHORTLIST_SIZE: usize = 40;
-/// How far the final count may drift from `target_article_count` (§3.6: 15–25
-/// around a default target of 20).
-pub const TARGET_TOLERANCE: usize = 5;
-/// Words of lead-in text shown per candidate in the stage-B prompt.
-const BLURB_WORDS: usize = 45;
-
-/// One element of the stage-B JSON response (§3.6).
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct SelectionItem {
-    pub id: ArticleId,
-    /// Must be one of `curation.sections`.
-    pub section: String,
-    pub position: i64,
-    #[serde(default)]
-    pub lead_story: bool,
-}
-
-/// Envelope the model is asked to return.
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct SelectionResponse {
-    #[serde(default)]
-    pub picks: Vec<SelectionItem>,
-}
-
-/// The invariant instruction block for stage B (§3.6).
-pub const SELECT_INSTRUCTIONS: &str = "\
-TASK: assemble today's issue of The Daily EPUB from the shortlist below.
-
-You are choosing what one specific reader — the profile in your system prompt — \
-will actually read on an e-ink screen over breakfast. Build a paper, not a \
-ranking: it should have a shape, a range of subjects, and a clear front page.
-
-RULES
-1. Pick articles by id from the shortlist only. Never invent an id.
-2. Give every pick a section from the palette below, spelled exactly as given.
-3. Number picks within each section from 1 upward, best first.
-4. Flag exactly one pick as \"lead_story\": true — the day's strongest, most \
-substantial piece. It must sit in the first section you use.
-5. Any candidate marked \"always-include\" MUST appear; place it in \"From the \
-Blogroll\" unless it clearly belongs elsewhere.
-6. Do not select two articles that tell the same story; keep the better one.
-
-EDITORIAL JUDGEMENT
-- Favour depth over coverage: a slim issue of excellent pieces beats a full one \
-padded with filler. Drop anything you would not defend.
-- Mix the day up. Several long technical dives in a row is a bad breakfast; \
-alternate register and subject across sections.
-- Keep the local and ultra-niche picks — a Boston story and a small-scene story \
-are worth more here than a third AI-industry item.
-- Score is evidence, not an instruction: overrule it when the paper reads better \
-for it, and say so through your placement.
-- Leave a section out entirely rather than padding it; empty sections are dropped.
-
-Return JSON exactly in this shape and nothing else:
-{\"picks\": [{\"id\": 123, \"section\": \"Top Stories\", \"position\": 1, \
-\"lead_story\": true}]}";
-
-/// Render the stage-B user prompt (§3.6).
-pub fn build_prompt(shortlist: &[ScoredArticle], sections: &[String], target: usize) -> String {
-    let (min, max) = size_bounds(target);
-    let mut prompt = String::with_capacity(2048 + shortlist.len() * 400);
-    prompt.push_str(SELECT_INSTRUCTIONS);
-    let _ = write!(
-        prompt,
-        "\n\nSECTION PALETTE (exact strings, use only these): {}\n\
-         Reserved and unavailable: \"{WORLD_BRIEFING_SECTION}\" is compiled separately.\n\n\
-         SIZE: choose {target} articles; never fewer than {min} and never more than {max}.\n\n\
-         SHORTLIST ({} candidates, best-ranked first)\n",
-        sections.join(" | "),
-        shortlist.len()
-    );
-    for candidate in shortlist {
-        prompt.push('\n');
-        prompt.push_str(&render_candidate(candidate));
-    }
-    prompt
-}
-
-fn render_candidate(candidate: &ScoredArticle) -> String {
-    let a = &candidate.article;
-    let mut block = String::with_capacity(400);
-    let _ = writeln!(block, "--- id: {}", a.id);
-    let _ = writeln!(block, "title: {}", a.title.trim());
-    let _ = writeln!(
-        block,
-        "feed: {} · {} words (~{} min){}",
-        if a.feed_title.is_empty() {
-            "unknown"
-        } else {
-            a.feed_title.trim()
-        },
-        a.word_count,
-        a.reading_minutes(),
-        if a.excerpt_only {
-            " · EXCERPT ONLY"
-        } else {
-            ""
-        }
-    );
-    match candidate.llm.as_ref() {
-        Some(llm) => {
-            let _ = writeln!(
-                block,
-                "score: {:.1} ({}) — {}",
-                llm.score,
-                if llm.category.is_empty() {
-                    "uncategorized"
-                } else {
-                    llm.category.as_str()
-                },
-                llm.rationale.trim()
-            );
-        }
-        None => {
-            let _ = writeln!(
-                block,
-                "score: unscored (heuristic rank {:.0}/100)",
-                candidate.prefilter_score
-            );
-        }
-    }
-    let _ = writeln!(
-        block,
-        "signals: social {:.2}; feed prior {:.2}; via {}{}",
-        candidate.social_score,
-        candidate.feed_prior,
-        source_kinds(candidate),
-        if candidate.auto_include {
-            "; ALWAYS-INCLUDE"
-        } else {
-            ""
-        }
-    );
-    let blurb = truncate_words(&html_to_text(&a.content_html), BLURB_WORDS);
-    if !blurb.is_empty() {
-        let _ = writeln!(block, "opening: {blurb}");
-    }
-    block
-}
-
-fn source_kinds(candidate: &ScoredArticle) -> String {
-    let mut kinds: Vec<&str> = candidate
-        .article
-        .sources
-        .iter()
-        .map(|s| match s.kind {
-            SourceKind::Scour => "scour",
-            SourceKind::HnFrontpage => "hn_frontpage",
-            SourceKind::Lobsters => "lobsters",
-            SourceKind::Reddit => "reddit",
-            SourceKind::Feed => "feed",
-        })
-        .collect();
-    kinds.sort_unstable();
-    kinds.dedup();
-    if kinds.is_empty() {
-        "feed".into()
-    } else {
-        kinds.join("+")
-    }
-}
-
-/// `target ± TARGET_TOLERANCE`, floored at one article (§3.6).
-pub fn size_bounds(target: usize) -> (usize, usize) {
-    let target = target.max(1);
-    (
-        target.saturating_sub(TARGET_TOLERANCE).max(1),
-        target + TARGET_TOLERANCE,
-    )
-}
-
-// ---------------------------------------------------------------------------
-// Section validation (§3.6: the model may only use the configured palette)
-// ---------------------------------------------------------------------------
-
-/// The section unrecognized labels fall back to.
-pub fn default_section(sections: &[String]) -> String {
-    sections
-        .iter()
-        .find(|s| s.as_str() == "Top Stories")
-        .or_else(|| sections.first())
-        .cloned()
-        .unwrap_or_else(|| "Top Stories".to_string())
-}
-
-/// Map whatever the model said onto the configured palette (§3.6).
-///
-/// Exact match → case-insensitive match → best word-overlap match → default.
-pub fn resolve_section(raw: &str, sections: &[String]) -> String {
-    let candidate = raw.trim();
-    if candidate.is_empty() || candidate.eq_ignore_ascii_case(WORLD_BRIEFING_SECTION) {
-        return default_section(sections);
-    }
-    if let Some(exact) = sections.iter().find(|s| s.as_str() == candidate) {
-        return exact.clone();
-    }
-    if let Some(ci) = sections.iter().find(|s| s.eq_ignore_ascii_case(candidate)) {
-        return ci.clone();
-    }
-    let wanted = words_of(candidate);
-    let best = sections
-        .iter()
-        .map(|s| (s, words_of(s).intersection(&wanted).count()))
-        .filter(|(_, overlap)| *overlap > 0)
-        .max_by_key(|(_, overlap)| *overlap);
-    match best {
-        Some((section, _)) => {
-            tracing::debug!(raw = candidate, mapped = %section, "mapped an off-palette section");
-            section.clone()
-        }
-        None => {
-            tracing::warn!(raw = candidate, "unknown section; using the default");
-            default_section(sections)
-        }
-    }
-}
-
-fn words_of(s: &str) -> HashSet<String> {
-    s.split(|c: char| !c.is_alphanumeric())
-        .filter(|w| w.len() > 2 && !w.eq_ignore_ascii_case("and"))
-        .map(str::to_lowercase)
-        .collect()
-}
-
-/// Section guess from feed metadata, used by `--skip-llm` and by top-ups (§3.6).
-pub fn heuristic_section(candidate: &ScoredArticle, sections: &[String]) -> String {
-    if candidate.auto_include {
-        return resolve_section("From the Blogroll", sections);
-    }
-    let haystack = format!(
-        "{} {} {}",
-        candidate.article.category.as_deref().unwrap_or_default(),
-        candidate.article.feed_title,
-        candidate.article.title
-    )
-    .to_lowercase();
-
-    const RULES: &[(&str, &[&str])] = &[
-        (
-            "Boston & Local",
-            &[
-                "boston",
-                "massachusetts",
-                "cambridge",
-                "mbta",
-                "new england",
-            ],
-        ),
-        (
-            "AI & Machine Learning",
-            &[
-                " ai ",
-                "ai:",
-                "llm",
-                "machine learning",
-                "neural",
-                "openai",
-                "anthropic",
-                "gpt",
-                "diffusion",
-            ],
-        ),
-        (
-            "Science & Space",
-            &[
-                "science",
-                "space",
-                "nasa",
-                "astronom",
-                "physics",
-                "biology",
-                "climate",
-                "aerospace",
-            ],
-        ),
-        (
-            "Culture & Essays",
-            &[
-                "essay", "culture", "book", "fiction", "poetry", "film", "music", "art", "review",
-            ],
-        ),
-        (
-            "Niche Corner",
-            &[
-                "hobby",
-                "retro",
-                "keyboard",
-                "board game",
-                "coffee",
-                "e-ink",
-                "eink",
-            ],
-        ),
-        (
-            "Tech & Engineering",
-            &[
-                "tech",
-                "programming",
-                "engineering",
-                "software",
-                "developer",
-                "rust",
-                "linux",
-                "database",
-                "systems",
-                "web",
-            ],
-        ),
-    ];
-    for (section, needles) in RULES {
-        if needles.iter().any(|n| haystack.contains(n))
-            && let Some(found) = sections.iter().find(|s| s.as_str() == *section)
-        {
-            return found.clone();
-        }
-    }
-    default_section(sections)
-}
-
-// ---------------------------------------------------------------------------
-// Response parsing
-// ---------------------------------------------------------------------------
-
-/// Keys the model might wrap the array in.
-const ARRAY_KEYS: &[&str] = &["picks", "lineup", "articles", "selection", "items"];
-
-/// Lenient parse of the stage-B response (§3.6).
-pub fn parse_selection_response(raw: &str) -> Vec<SelectionItem> {
-    let cleaned = strip_code_fence(raw);
-    let value: serde_json::Value = match serde_json::from_str(cleaned) {
-        Ok(v) => v,
-        Err(e) => {
-            tracing::warn!(error = %e, "stage B response was not JSON");
-            return Vec::new();
-        }
-    };
-    let array = match &value {
-        serde_json::Value::Array(items) => Some(items),
-        serde_json::Value::Object(map) => ARRAY_KEYS
-            .iter()
-            .find_map(|k| map.get(*k).and_then(serde_json::Value::as_array))
-            .or_else(|| map.values().find_map(serde_json::Value::as_array)),
-        _ => None,
-    };
-    let Some(array) = array else {
-        tracing::warn!("stage B response contained no array of picks");
-        return Vec::new();
-    };
-
-    let mut out = Vec::with_capacity(array.len());
-    for (idx, item) in array.iter().enumerate() {
-        let Some(obj) = item.as_object() else {
-            tracing::warn!("skipping a non-object stage B pick");
-            continue;
-        };
-        let Some(id) = obj.get("id").and_then(|v| {
-            v.as_i64()
-                .or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
-        }) else {
-            tracing::warn!("skipping a stage B pick without an id");
-            continue;
-        };
-        out.push(SelectionItem {
-            id,
-            section: obj
-                .get("section")
-                .and_then(serde_json::Value::as_str)
-                .unwrap_or_default()
-                .trim()
-                .to_string(),
-            position: obj
-                .get("position")
-                .and_then(|v| {
-                    v.as_i64()
-                        .or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
-                })
-                .unwrap_or(idx as i64 + 1),
-            lead_story: obj
-                .get("lead_story")
-                .or_else(|| obj.get("is_lead"))
-                .and_then(|v| {
-                    v.as_bool()
-                        .or_else(|| v.as_str().map(|s| s.eq_ignore_ascii_case("true")))
-                })
-                .unwrap_or(false),
-        });
-    }
-    out
-}
-
-// ---------------------------------------------------------------------------
-// Stage driver
-// ---------------------------------------------------------------------------
-
-/// Ask the model for the day's lineup, validating that every section is from the
-/// configured palette and exactly one pick is the lead (§3.6).
-pub async fn select(
-    llm: &LlmClient,
-    candidates: Vec<ScoredArticle>,
-    sections: &[String],
-    target: usize,
-    date: Date,
-) -> Result<Lineup, LlmError> {
-    if candidates.is_empty() {
-        tracing::warn!("stage B had no candidates");
-        return Ok(Lineup {
-            date,
-            picks: Vec::new(),
-            section_order: Vec::new(),
-        });
-    }
-    if let Err(e) = llm.meter.check_budget() {
-        tracing::error!(error = %e,
-            "COST CEILING HIT before stage B selection — falling back to heuristic ranking");
-        return Ok(select_without_llm(candidates, sections, target, date));
-    }
-
-    let shortlist = shortlist(&candidates, target);
-    let prompt = build_prompt(&shortlist, sections, target);
-    tracing::debug!(
-        shortlist = shortlist.len(),
-        approx_tokens = super::approx_tokens(&prompt),
-        "stage B request"
-    );
-
-    let raw = llm.complete(&prompt, llm_temperature(), true).await?;
-    let items = parse_selection_response(&raw);
-    if items.is_empty() {
-        tracing::error!("stage B returned no usable picks; falling back to heuristic ranking");
-        return Ok(select_without_llm(candidates, sections, target, date));
-    }
-
-    let by_id: HashMap<ArticleId, &ScoredArticle> =
-        candidates.iter().map(|c| (c.article.id, c)).collect();
-    let mut chosen: Vec<(SelectionItem, ScoredArticle)> = Vec::with_capacity(items.len());
-    let mut seen: HashSet<ArticleId> = HashSet::new();
-    for item in items {
-        if !seen.insert(item.id) {
-            tracing::warn!(id = item.id, "stage B picked the same article twice");
-            continue;
-        }
-        match by_id.get(&item.id) {
-            Some(candidate) => chosen.push((item, (*candidate).clone())),
-            None => tracing::warn!(id = item.id, "stage B invented an id that was not offered"),
-        }
-    }
-
-    // Auto-include feeds can never be dropped (§3.5).
-    for candidate in &candidates {
-        if candidate.auto_include && seen.insert(candidate.article.id) {
-            tracing::info!(
-                id = candidate.article.id,
-                title = %candidate.article.title,
-                "re-inserting an always-include article the model dropped"
-            );
-            chosen.push((
-                SelectionItem {
-                    id: candidate.article.id,
-                    section: "From the Blogroll".into(),
-                    position: i64::MAX,
-                    lead_story: false,
-                },
-                candidate.clone(),
-            ));
-        }
-    }
-
-    let lineup = assemble(chosen, &candidates, sections, target, date);
-    tracing::info!(
-        picks = lineup.picks.len(),
-        sections = lineup.section_order.len(),
-        lead = lineup.lead().map(|p| p.article.id),
-        "stage B lineup ready"
-    );
-    Ok(lineup)
-}
-
-/// Stage B runs at the scoring temperature: this is a judgement call, not prose.
-fn llm_temperature() -> f32 {
-    0.4
-}
-
-/// Top [`SHORTLIST_SIZE`] candidates by combined score, always including the
-/// auto-includes (§3.6).
-fn shortlist(candidates: &[ScoredArticle], target: usize) -> Vec<ScoredArticle> {
-    let mut ranked: Vec<ScoredArticle> = candidates.to_vec();
-    sort_by_combined(&mut ranked);
-    let keep = SHORTLIST_SIZE.max(target * 2);
-    if ranked.len() <= keep {
-        return ranked;
-    }
-    let (head, tail) = ranked.split_at(keep);
-    let mut out = head.to_vec();
-    out.extend(tail.iter().filter(|c| c.auto_include).cloned());
-    out
-}
-
-fn sort_by_combined(candidates: &mut [ScoredArticle]) {
-    candidates.sort_by(|a, b| {
-        b.combined_score()
-            .partial_cmp(&a.combined_score())
-            .unwrap_or(std::cmp::Ordering::Equal)
-            .then_with(|| a.article.id.cmp(&b.article.id))
-    });
-}
-
-/// Turn validated picks into a [`Lineup`]: clamp the size, force a single lead,
-/// order the sections and renumber positions (§3.6).
-fn assemble(
-    mut chosen: Vec<(SelectionItem, ScoredArticle)>,
-    all: &[ScoredArticle],
-    sections: &[String],
-    target: usize,
-    date: Date,
-) -> Lineup {
-    let (min, max) = size_bounds(target);
-
-    // Too many: drop the weakest non-auto-include picks.
-    if chosen.len() > max {
-        chosen.sort_by(|a, b| {
-            b.1.auto_include.cmp(&a.1.auto_include).then_with(|| {
-                b.1.combined_score()
-                    .partial_cmp(&a.1.combined_score())
-                    .unwrap_or(std::cmp::Ordering::Equal)
-            })
-        });
-        let dropped = chosen.len() - max;
-        chosen.truncate(max);
-        tracing::info!(dropped, max, "trimmed the lineup to the size ceiling");
-    }
-
-    // Too few: top up from the best unpicked candidates.
-    if chosen.len() < min {
-        let taken: HashSet<ArticleId> = chosen.iter().map(|(i, _)| i.id).collect();
-        let mut rest: Vec<ScoredArticle> = all
-            .iter()
-            .filter(|c| !taken.contains(&c.article.id))
-            .cloned()
-            .collect();
-        sort_by_combined(&mut rest);
-        let wanted = min - chosen.len();
-        let added = rest.len().min(wanted);
-        for candidate in rest.into_iter().take(wanted) {
-            let section = heuristic_section(&candidate, sections);
-            chosen.push((
-                SelectionItem {
-                    id: candidate.article.id,
-                    section,
-                    position: i64::MAX,
-                    lead_story: false,
-                },
-                candidate,
-            ));
-        }
-        tracing::info!(added, min, "topped the lineup up to the size floor");
-    }
-
-    // Normalize sections and pick the section order.
-    for (item, _) in chosen.iter_mut() {
-        item.section = resolve_section(&item.section, sections);
-    }
-    let used: HashSet<&str> = chosen.iter().map(|(i, _)| i.section.as_str()).collect();
-    let mut section_order: Vec<String> = sections
-        .iter()
-        .filter(|s| used.contains(s.as_str()))
-        .cloned()
-        .collect();
-    for (item, _) in &chosen {
-        if !section_order.contains(&item.section) {
-            section_order.push(item.section.clone());
-        }
-    }
-    let section_rank: HashMap<&str, usize> = section_order
-        .iter()
-        .enumerate()
-        .map(|(i, s)| (s.as_str(), i))
-        .collect();
-
-    // Order: section, then the model's position, then quality, then id.
-    chosen.sort_by(|a, b| {
-        section_rank
-            .get(a.0.section.as_str())
-            .cmp(&section_rank.get(b.0.section.as_str()))
-            .then_with(|| a.0.position.cmp(&b.0.position))
-            .then_with(|| {
-                b.1.combined_score()
-                    .partial_cmp(&a.1.combined_score())
-                    .unwrap_or(std::cmp::Ordering::Equal)
-            })
-            .then_with(|| a.1.article.id.cmp(&b.1.article.id))
-    });
-
-    // Exactly one lead, and it must live in the first section (§3.6 rule 4).
-    let lead_id = chosen
-        .iter()
-        .find(|(item, _)| item.lead_story)
-        .filter(|(item, _)| section_rank.get(item.section.as_str()) == Some(&0))
-        .or_else(|| chosen.first())
-        .map(|(item, _)| item.id);
-
-    let mut per_section: BTreeMap<String, i64> = BTreeMap::new();
-    let picks = chosen
-        .into_iter()
-        .map(|(item, candidate)| {
-            let position = per_section
-                .entry(item.section.clone())
-                .and_modify(|n| *n += 1)
-                .or_insert(1);
-            Pick {
-                section: item.section,
-                position: *position,
-                is_lead: Some(item.id) == lead_id,
-                summary: None,
-                llm: candidate.llm.clone(),
-                discussion: None,
-                article: candidate.article,
-            }
-        })
-        .collect();
-
-    Lineup {
-        date,
-        picks,
-        section_order,
-    }
-}
-
-/// `--skip-llm` fallback: take the top `target` by prefilter score and bucket them
-/// into sections by feed category (notes §6).
-pub fn select_without_llm(
-    candidates: Vec<ScoredArticle>,
-    sections: &[String],
-    target: usize,
-    date: Date,
-) -> Lineup {
-    let mut ranked = candidates.clone();
-    super::prefilter::sort_by_prefilter(&mut ranked);
-
-    let mut chosen: Vec<(SelectionItem, ScoredArticle)> = Vec::new();
-    let mut seen: HashSet<ArticleId> = HashSet::new();
-    for candidate in ranked.into_iter() {
-        let auto = candidate.auto_include;
-        if chosen.len() >= target && !auto {
-            continue;
-        }
-        if !seen.insert(candidate.article.id) {
-            continue;
-        }
-        let section = heuristic_section(&candidate, sections);
-        chosen.push((
-            SelectionItem {
-                id: candidate.article.id,
-                section,
-                position: chosen.len() as i64 + 1,
-                lead_story: false,
-            },
-            candidate,
-        ));
-    }
-    let lineup = assemble(chosen, &candidates, sections, target, date);
-    tracing::info!(
-        picks = lineup.picks.len(),
-        sections = lineup.section_order.len(),
-        "skip-llm lineup ready"
-    );
-    lineup
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use crate::config::{CurationConfig, DeepseekConfig};
-    use crate::curate::llm::{MockBackend, UsageMeter};
-    use crate::curate::prefilter::tests::article;
-    use crate::types::{LlmScore, TokenUsage};
-    use std::sync::Arc;
-
-    const LINEUP_FIXTURE: &str = include_str!(concat!(
-        env!("CARGO_MANIFEST_DIR"),
-        "/tests/fixtures/deepseek_lineup.json"
-    ));
-
-    fn sections() -> Vec<String> {
-        CurationConfig::default().sections
-    }
-
-    fn date() -> Date {
-        "2026-08-15".parse().expect("date")
-    }
-
-    fn candidate(id: i64, title: &str, words: i64, score: f64) -> ScoredArticle {
-        ScoredArticle {
-            article: article(id, title, words),
-            prefilter_score: 40.0 + score,
-            social_score: 1.0,
-            feed_prior: 0.5,
-            llm: Some(LlmScore {
-                score,
-                category: "Tech & Engineering".into(),
-                rationale: "solid".into(),
-                is_paywalled_guess: false,
-            }),
-            auto_include: false,
-        }
-    }
-
-    fn candidates(n: i64) -> Vec<ScoredArticle> {
-        (1..=n)
-            .map(|i| {
-                candidate(
-                    i,
-                    &format!("Article {i}"),
-                    500 + i * 10,
-                    (10.0 - i as f64 * 0.1).max(0.0),
-                )
-            })
-            .collect()
-    }
-
-    #[test]
-    fn section_resolution_maps_onto_the_palette() {
-        let s = sections();
-        assert_eq!(resolve_section("Top Stories", &s), "Top Stories");
-        assert_eq!(resolve_section("  top stories ", &s), "Top Stories");
-        assert_eq!(
-            resolve_section("Technology & Engineering", &s),
-            "Tech & Engineering"
-        );
-        assert_eq!(resolve_section("Science", &s), "Science & Space");
-        assert_eq!(resolve_section("Sports", &s), "Top Stories");
-        assert_eq!(resolve_section("", &s), "Top Stories");
-        // The reserved section is never allowed through (§3.6).
-        assert_eq!(resolve_section(WORLD_BRIEFING_SECTION, &s), "Top Stories");
-        // A palette without "Top Stories" falls back to its first entry.
-        let tiny = vec!["Niche Corner".to_string()];
-        assert_eq!(resolve_section("Whatever", &tiny), "Niche Corner");
-    }
-
-    #[test]
-    fn heuristic_sections_follow_feed_metadata() {
-        let s = sections();
-        let mut c = candidate(1, "MBTA slow zones, charted", 900, 6.0);
-        c.article.category = Some("News".into());
-        assert_eq!(heuristic_section(&c, &s), "Boston & Local");
-
-        let mut ai = candidate(2, "A new LLM benchmark", 900, 6.0);
-        ai.article.category = Some("Machine Learning".into());
-        assert_eq!(heuristic_section(&ai, &s), "AI & Machine Learning");
-
-        let mut blog = candidate(3, "Notes from my week", 900, 6.0);
-        blog.auto_include = true;
-        assert_eq!(heuristic_section(&blog, &s), "From the Blogroll");
-
-        let mut plain = candidate(4, "Untitled musing", 900, 6.0);
-        plain.article.category = None;
-        plain.article.feed_title = "A Journal".into();
-        assert_eq!(heuristic_section(&plain, &s), "Top Stories");
-    }
-
-    #[test]
-    fn parses_a_realistic_lineup_response() {
-        let items = parse_selection_response(LINEUP_FIXTURE);
-        assert_eq!(items.len(), 6);
-        assert_eq!(items[0].id, 101);
-        assert!(items[0].lead_story);
-        assert_eq!(items[0].section, "Top Stories");
-        assert_eq!(items.iter().filter(|i| i.lead_story).count(), 1);
-        // Junk entries in the fixture are dropped, not fatal.
-        assert!(items.iter().all(|i| i.id != 0));
-    }
-
-    #[test]
-    fn size_bounds_follow_the_spec() {
-        assert_eq!(size_bounds(20), (15, 25));
-        assert_eq!(size_bounds(6), (1, 11));
-        assert_eq!(size_bounds(0), (1, 6));
-    }
-
-    #[tokio::test]
-    async fn selection_builds_a_valid_lineup() {
-        let backend = Arc::new(MockBackend::new());
-        backend.push(LINEUP_FIXTURE, TokenUsage::default());
-        let llm = LlmClient::with_backend(
-            "deepseek-v4-flash",
-            "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), 2.0),
-            backend.clone(),
-        );
-
-        // ids 101..=112 so the fixture's picks resolve.
-        let pool: Vec<ScoredArticle> = (101..=112)
-            .map(|i| candidate(i, &format!("Article {i}"), 800, 7.0))
-            .collect();
-        let lineup = select(&llm, pool, &sections(), 6, date())
-            .await
-            .expect("selection");
-
-        assert_eq!(lineup.date, date());
-        assert_eq!(lineup.picks.len(), 6);
-        assert_eq!(lineup.picks.iter().filter(|p| p.is_lead).count(), 1);
-        assert_eq!(lineup.lead().map(|p| p.article.id), Some(101));
-        // Every section is from the palette and non-empty.
-        for section in &lineup.section_order {
-            assert!(sections().contains(section), "{section} is off-palette");
-            assert!(!lineup.section_picks(section).is_empty());
-        }
-        // Positions restart at 1 inside each section and ascend.
-        for section in &lineup.section_order {
-            let positions: Vec<i64> = lineup
-                .section_picks(section)
-                .iter()
-                .map(|p| p.position)
-                .collect();
-            assert_eq!(
-                positions,
-                (1..=positions.len() as i64).collect::<Vec<_>>(),
-                "{section} positions"
-            );
-        }
-        // The lead sits in the first section used.
-        assert_eq!(
-            lineup.lead().map(|p| p.section.clone()),
-            lineup.section_order.first().cloned()
-        );
-        // The prompt carried the shortlist and the palette.
-        let prompt = &backend.prompts()[0].user;
-        assert!(prompt.starts_with(SELECT_INSTRUCTIONS));
-        assert!(prompt.contains("--- id: 101"));
-        assert!(prompt.contains("never fewer than 1 and never more than 11"));
-    }
-
-    #[tokio::test]
-    async fn hallucinated_ids_and_missing_leads_are_repaired() {
-        let backend = Arc::new(MockBackend::new());
-        backend.push(
-            r#"{"picks":[{"id":9999,"section":"Top Stories","position":1,"lead_story":true},
-                         {"id":1,"section":"Sportsball","position":2},
-                         {"id":2,"section":"Niche Corner","position":1}]}"#,
-            TokenUsage::default(),
-        );
-        let llm = LlmClient::with_backend(
-            "deepseek-v4-flash",
-            "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), 2.0),
-            backend,
-        );
-        let lineup = select(&llm, candidates(6), &sections(), 2, date())
-            .await
-            .expect("selection");
-        assert!(lineup.picks.iter().all(|p| p.article.id != 9999));
-        assert_eq!(lineup.picks.iter().filter(|p| p.is_lead).count(), 1);
-        for pick in &lineup.picks {
-            assert!(sections().contains(&pick.section));
-        }
-    }
-
-    #[tokio::test]
-    async fn oversized_and_undersized_answers_are_clamped() {
-        // Undersized: the model returns one pick but the floor is 5.
-        let backend = Arc::new(MockBackend::new());
-        backend.push(
-            r#"{"picks":[{"id":1,"section":"Top Stories","position":1,"lead_story":true}]}"#,
-            TokenUsage::default(),
-        );
-        let llm = LlmClient::with_backend(
-            "deepseek-v4-flash",
-            "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), 2.0),
-            backend,
-        );
-        let lineup = select(&llm, candidates(30), &sections(), 10, date())
-            .await
-            .expect("selection");
-        assert!(lineup.picks.len() >= 5, "{}", lineup.picks.len());
-
-        // Oversized: 30 picks against a target of 6 (ceiling 11).
-        let picks: Vec<String> = (1..=30)
-            .map(|i| format!(r#"{{"id":{i},"section":"Top Stories","position":{i}}}"#))
-            .collect();
-        let backend = Arc::new(MockBackend::new());
-        backend.push(
-            format!(r#"{{"picks":[{}]}}"#, picks.join(",")),
-            TokenUsage::default(),
-        );
-        let llm = LlmClient::with_backend(
-            "deepseek-v4-flash",
-            "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), 2.0),
-            backend,
-        );
-        let lineup = select(&llm, candidates(30), &sections(), 6, date())
-            .await
-            .expect("selection");
-        assert_eq!(lineup.picks.len(), 11);
-    }
-
-    #[tokio::test]
-    async fn always_include_articles_are_reinserted() {
-        let backend = Arc::new(MockBackend::new());
-        backend.push(
-            r#"{"picks":[{"id":1,"section":"Top Stories","position":1,"lead_story":true}]}"#,
-            TokenUsage::default(),
-        );
-        let llm = LlmClient::with_backend(
-            "deepseek-v4-flash",
-            "SYSTEM".into(),
-            UsageMeter::new(&DeepseekConfig::default(), 2.0),
-            backend,
-        );
-        let mut pool = candidates(3);
-        pool[2].auto_include = true;
-        let lineup = select(&llm, pool, &sections(), 1, date())
-            .await
-            .expect("selection");
-        let ids: Vec<ArticleId> = lineup.picks.iter().map(|p| p.article.id).collect();
-        assert!(ids.contains(&3), "auto-include must survive: {ids:?}");
-        assert_eq!(
-            lineup
-                .picks
-                .iter()
-                .find(|p| p.article.id == 3)
-                .map(|p| p.section.as_str()),
-            Some("From the Blogroll")
-        );
-    }
-
-    #[tokio::test]
-    async fn a_tripped_budget_falls_back_without_calling_the_model() {
-        let backend = Arc::new(MockBackend::new());
-        let meter = UsageMeter::new(&DeepseekConfig::default(), 0.001);
-        meter.record(TokenUsage {
-            input_tokens: 1_000_000,
-            cached_tokens: 0,
-            output_tokens: 0,
-        });
-        let llm =
-            LlmClient::with_backend("deepseek-v4-flash", "SYSTEM".into(), meter, backend.clone());
-        let lineup = select(&llm, candidates(20), &sections(), 6, date())
-            .await
-            .expect("fallback");
-        assert_eq!(backend.calls(), 0);
-        assert_eq!(lineup.picks.len(), 6);
-    }
-
-    #[test]
-    fn skip_llm_lineup_uses_prefilter_order() {
-        let mut pool = candidates(10);
-        pool.iter_mut().for_each(|c| c.llm = None);
-        pool[7].prefilter_score = 99.0; // id 8 is the strongest heuristically
-        pool[9].auto_include = true; // id 10 is a personal blog
-        pool[9].prefilter_score = 1.0;
-
-        let lineup = select_without_llm(pool, &sections(), 4, date());
-        assert_eq!(lineup.picks.len(), 5, "4 picks + the auto-include");
-        assert_eq!(lineup.lead().map(|p| p.article.id), Some(8));
-        assert_eq!(lineup.picks.iter().filter(|p| p.is_lead).count(), 1);
-        assert!(
-            lineup
-                .picks
-                .iter()
-                .any(|p| p.article.id == 10 && p.section == "From the Blogroll")
-        );
-        for pick in &lineup.picks {
-            assert!(sections().contains(&pick.section));
-            assert!(pick.summary.is_none());
-        }
-        assert!(!lineup.section_order.is_empty());
-    }
-
-    #[test]
-    fn empty_input_yields_an_empty_lineup() {
-        let lineup = select_without_llm(Vec::new(), &sections(), 20, date());
-        assert!(lineup.picks.is_empty());
-        assert!(lineup.section_order.is_empty());
-        assert!(lineup.lead().is_none());
-    }
-}

src/curate/signals.rs

@@ -0,0 +1,981 @@
+//! Cheap per-article ranking signals, the mid-rank percentile normalizer and the
+//! preliminary blend (plan §9, §12.2, §12.4).
+//!
+//! Every signal is an `Option<f64>`: `None` means *absent*, which is never a
+//! numeric zero. Absent signals are left out of the percentile computation and
+//! of the blend, whose remaining weights are renormalized (§12.2, §12.4).
+
+use std::collections::{BTreeMap, HashMap, HashSet};
+
+use jiff::Timestamp;
+use serde::{Deserialize, Serialize};
+
+use crate::config::{PreliminaryWeights, RankingConfig, VoyageConfig};
+use crate::curate::embedding::{dot, load_article_embeddings};
+use crate::curate::prefilter;
+use crate::db::Db;
+use crate::types::{Article, ArticleId, FeedId, SourceKind};
+
+/// Below this many embedded eligible articles the z-score is too noisy, so the
+/// interest signal falls back to the raw top-1 cosine (§9.1).
+pub const INTEREST_ZSCORE_MIN_ARTICLES: usize = 30;
+/// Standard-deviation floor for the per-interest z-score (§9.1).
+const ZSCORE_STD_FLOOR: f64 = 1e-3;
+/// How many interests and rated neighbours `signals_json` records (§7.5).
+const RECORDED_TOP: usize = 3;
+
+/// The signal names that go through the percentile normalizer, in the order
+/// they are rendered (§12.2). LLM scores (`triage`, `quality`, `fit`) are
+/// absolute and arrive in steps 4–5.
+pub const PERCENTILE_SIGNALS: [&str; 5] = ["interest", "knn", "feed", "social", "heuristic"];
+
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct TopInterest {
+    pub name: String,
+    pub z: f64,
+    pub cos: f64,
+}
+
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct Neighbour {
+    pub article_id: ArticleId,
+    pub label: String,
+    pub cos: f64,
+    pub title: String,
+}
+
+/// Every cheap signal for one article, plus what the normalizer and the blend
+/// derived from them (§9, §12.2, §12.4).
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct Signals {
+    pub interest: Option<f64>,
+    /// Raw top-1 cosine behind `interest`, recorded for `explain` (§7.5).
+    pub interest_top1_cos: Option<f64>,
+    pub knn: Option<f64>,
+    pub feed: Option<f64>,
+    pub social: Option<f64>,
+    pub heuristic: Option<f64>,
+    /// Mid-rank percentiles of the present signals (§12.2).
+    #[serde(default)]
+    pub norm: BTreeMap<String, f64>,
+    /// Effective preliminary weights after gating and renormalization (§12.4).
+    #[serde(default)]
+    pub weights: BTreeMap<String, f64>,
+    #[serde(default)]
+    pub top_interests: Vec<TopInterest>,
+    #[serde(default)]
+    pub neighbours: Vec<Neighbour>,
+    #[serde(default)]
+    pub notes: Vec<String>,
+    /// Preliminary blend on a 0–100 scale; `None` when nothing is present.
+    pub preliminary: Option<f64>,
+    /// Gate ramps applied to the learned signals' weights (§9.2, §9.3).
+    #[serde(skip)]
+    pub knn_gate: f64,
+    #[serde(skip)]
+    pub feed_gate: f64,
+}
+
+impl Signals {
+    /// The raw value of a named signal, `None` when absent or unknown.
+    pub fn raw(&self, name: &str) -> Option<f64> {
+        match name {
+            "interest" => self.interest,
+            "interest_top1_cos" => self.interest_top1_cos,
+            "knn" => self.knn,
+            "feed" => self.feed,
+            "social" => self.social,
+            "heuristic" => self.heuristic,
+            _ => None,
+        }
+    }
+
+    pub fn present(&self, name: &str) -> bool {
+        self.raw(name).is_some()
+    }
+
+    /// The signals every eligible article gets without embeddings or ratings.
+    pub fn baseline(article: &Article) -> Self {
+        Self {
+            social: (!article.social.is_empty()).then(|| article.social_score()),
+            heuristic: Some(prefilter::text_heuristic(article)),
+            ..Self::default()
+        }
+    }
+}
+
+/// What the run log and the report say about the learned signals (§9.2, §15.4).
+#[derive(Debug, Clone, Copy, Default, PartialEq)]
+pub struct PreferenceSummary {
+    pub rated_with_embeddings: usize,
+    pub attributable_feed_ratings: usize,
+    pub knn_gate: f64,
+    pub feed_gate: f64,
+}
+
+/// One rated article with an embedding: the unit of the preference state (§9.2).
+#[derive(Debug, Clone, PartialEq)]
+pub struct RatedExample {
+    pub article_id: ArticleId,
+    pub label: String,
+    pub title: String,
+    /// The vote's value (`loved` 1.0, `good` 0.35, `not_for_me` −1.0).
+    pub value: f64,
+    /// `0.5 ^ (age_days / half_life_days)` at the time of the run.
+    pub decay: f64,
+    pub embedding: Vec<f32>,
+    /// Distinct direct feeds that carried the rated article (§9.3).
+    pub feeds: Vec<FeedId>,
+}
+
+impl RatedExample {
+    /// `weight_i = value_i × decay_i` (§9.2).
+    pub fn weight(&self) -> f64 {
+        self.value * self.decay
+    }
+}
+
+#[derive(Debug, Clone, Copy, Default, PartialEq)]
+struct FeedRate {
+    up: f64,
+    down: f64,
+}
+
+impl FeedRate {
+    /// Beta-smoothed rate `(up + 1) / (up + down + 2)` (§9.3).
+    fn rate(self) -> f64 {
+        (self.up + 1.0) / (self.up + self.down + 2.0)
+    }
+}
+
+/// Rated-neighbour and feed-affinity state, built once per run (§9.2, §9.3).
+#[derive(Debug, Clone, Default)]
+pub struct PreferenceState {
+    pub examples: Vec<RatedExample>,
+    feed_rates: HashMap<FeedId, FeedRate>,
+    pub attributable_feed_ratings: usize,
+    pub knn_gate: f64,
+    pub feed_gate: f64,
+}
+
+impl PreferenceState {
+    /// Build the state from already-loaded examples (pure; tests use this).
+    pub fn build(examples: Vec<RatedExample>, ranking: &RankingConfig) -> Self {
+        let (feed_rates, attributable_feed_ratings) = feed_rates(&examples);
+        Self {
+            knn_gate: gate(examples.len(), ranking.knn_floor, ranking.knn_full),
+            feed_gate: gate(
+                attributable_feed_ratings,
+                ranking.feed_floor,
+                ranking.feed_full,
+            ),
+            examples,
+            feed_rates,
+            attributable_feed_ratings,
+        }
+    }
+
+    /// Load `db::current_ratings(rating_lookback_days)` joined to
+    /// `article_embeddings`; ratings without an embedding are skipped (§9.2).
+    pub async fn load(
+        db: &Db,
+        voyage: &VoyageConfig,
+        ranking: &RankingConfig,
+        now: Timestamp,
+    ) -> anyhow::Result<Self> {
+        let ratings = db.current_ratings(ranking.rating_lookback_days).await?;
+        let ids = ratings
+            .iter()
+            .map(|rating| rating.article_id)
+            .collect::<Vec<_>>();
+        let embeddings = load_article_embeddings(db, voyage, &ids).await?;
+        let mut examples = Vec::new();
+        for rating in ratings {
+            let Some(embedding) = embeddings.get(&rating.article_id).cloned() else {
+                continue;
+            };
+            let feeds = db
+                .get_article(rating.article_id)
+                .await?
+                .as_ref()
+                .map(direct_feeds)
+                .unwrap_or_default();
+            let age_days = (now.as_second() - rating.event_at.as_second()).max(0) as f64 / 86_400.0;
+            examples.push(RatedExample {
+                article_id: rating.article_id,
+                label: rating.label,
+                title: rating.title,
+                value: rating.value,
+                decay: decay(age_days, ranking.rating_half_life_days),
+                embedding,
+                feeds,
+            });
+        }
+        Ok(Self::build(examples, ranking))
+    }
+
+    pub fn summary(&self) -> PreferenceSummary {
+        PreferenceSummary {
+            rated_with_embeddings: self.examples.len(),
+            attributable_feed_ratings: self.attributable_feed_ratings,
+            knn_gate: self.knn_gate,
+            feed_gate: self.feed_gate,
+        }
+    }
+
+    /// The once-per-run log line of §9.2.
+    pub fn log(&self, ranking: &RankingConfig) {
+        let feed_detail = if self.feed_gate > 0.0 {
+            format!("(n={})", self.attributable_feed_ratings)
+        } else {
+            format!(
+                "(n={} < {})",
+                self.attributable_feed_ratings, ranking.feed_floor
+            )
+        };
+        tracing::info!(
+            rated_with_embeddings = self.examples.len(),
+            knn_gate = self.knn_gate,
+            feed_gate = self.feed_gate,
+            "preference: {} rated articles with embeddings → knn gate {:.2}; feed gate {:.1} {}",
+            self.examples.len(),
+            self.knn_gate,
+            self.feed_gate,
+            feed_detail
+        );
+    }
+
+    /// Signed rated-neighbour preference and the three nearest rated articles
+    /// (§9.2). Absent when the gate is closed or there are no examples.
+    pub fn knn(&self, candidate: &[f32], ranking: &RankingConfig) -> (Option<f64>, Vec<Neighbour>) {
+        if self.knn_gate <= 0.0 || self.examples.is_empty() {
+            return (None, Vec::new());
+        }
+        let mut scored = self
+            .examples
+            .iter()
+            .filter_map(|example| {
+                dot(candidate, &example.embedding)
+                    .ok()
+                    .map(|s| (s, example))
+            })
+            .collect::<Vec<_>>();
+        scored.sort_by(|left, right| right.0.total_cmp(&left.0));
+
+        let side = |positive: bool| -> Option<f64> {
+            let chosen = scored
+                .iter()
+                .filter(|(_, example)| (example.weight() > 0.0) == positive)
+                .take(ranking.neighbour_k.max(1))
+                .collect::<Vec<_>>();
+            let denominator = chosen
+                .iter()
+                .map(|(_, example)| example.weight().abs())
+                .sum::<f64>();
+            (denominator > 0.0).then(|| {
+                chosen
+                    .iter()
+                    .map(|(similarity, example)| example.weight().abs() * similarity)
+                    .sum::<f64>()
+                    / denominator
+            })
+        };
+        let positive = side(true);
+        let negative = side(false);
+        let knn = (positive.is_some() || negative.is_some()).then(|| {
+            positive.unwrap_or(0.0) - ranking.negative_coefficient * negative.unwrap_or(0.0)
+        });
+        let neighbours = scored
+            .iter()
+            .take(RECORDED_TOP)
+            .map(|(cos, example)| Neighbour {
+                article_id: example.article_id,
+                label: example.label.clone(),
+                cos: *cos,
+                title: example.title.clone(),
+            })
+            .collect();
+        (knn, neighbours)
+    }
+
+    /// Mean Beta-smoothed rate over the article's rated direct feeds (§9.3).
+    pub fn feed(&self, article: &Article) -> Option<f64> {
+        if self.feed_gate <= 0.0 {
+            return None;
+        }
+        let rates = direct_feeds(article)
+            .into_iter()
+            .filter_map(|feed| self.feed_rates.get(&feed))
+            .map(|rate| rate.rate())
+            .collect::<Vec<_>>();
+        (!rates.is_empty()).then(|| rates.iter().sum::<f64>() / rates.len() as f64)
+    }
+
+    /// Per-feed `(up, down)` credit, exposed for tests of §9.3.
+    pub fn feed_credit(&self, feed: FeedId) -> Option<(f64, f64)> {
+        self.feed_rates.get(&feed).map(|rate| (rate.up, rate.down))
+    }
+}
+
+/// `0.5 ^ (age_days / half_life_days)` (§9.2).
+pub fn decay(age_days: f64, half_life_days: f64) -> f64 {
+    if half_life_days <= 0.0 {
+        return 1.0;
+    }
+    0.5f64.powf(age_days.max(0.0) / half_life_days)
+}
+
+/// `clamp((n − floor) / (full − floor), 0, 1)` (§9.2).
+pub fn gate(n: usize, floor: usize, full: usize) -> f64 {
+    if n <= floor {
+        0.0
+    } else if n >= full || full <= floor {
+        1.0
+    } else {
+        (n - floor) as f64 / (full - floor) as f64
+    }
+}
+
+/// Distinct `SourceKind::Feed` feeds that carried the article; the best entry's
+/// feed when there are none (§9.3).
+pub fn direct_feeds(article: &Article) -> Vec<FeedId> {
+    let mut feeds = article
+        .sources
+        .iter()
+        .filter(|source| source.kind == SourceKind::Feed)
+        .map(|source| source.feed_id)
+        .collect::<HashSet<_>>()
+        .into_iter()
+        .collect::<Vec<_>>();
+    if feeds.is_empty() && article.feed_id != 0 {
+        feeds.push(article.feed_id);
+    }
+    feeds.sort_unstable();
+    feeds
+}
+
+fn feed_rates(examples: &[RatedExample]) -> (HashMap<FeedId, FeedRate>, usize) {
+    let mut rates: HashMap<FeedId, FeedRate> = HashMap::new();
+    let mut attributable = 0;
+    for example in examples {
+        if example.feeds.is_empty() {
+            continue;
+        }
+        attributable += 1;
+        let credit = example.weight() / example.feeds.len() as f64;
+        for feed in &example.feeds {
+            let rate = rates.entry(*feed).or_default();
+            rate.up += credit.max(0.0);
+            rate.down += (-credit).max(0.0);
+        }
+    }
+    (rates, attributable)
+}
+
+/// The interest match of §9.1 for one article.
+#[derive(Debug, Clone, Default, PartialEq)]
+pub struct InterestMatch {
+    pub score: f64,
+    pub top1_cos: f64,
+    pub top_interests: Vec<TopInterest>,
+}
+
+/// Z-scored standing-interest match for every embedded article (§9.1).
+///
+/// Below [`INTEREST_ZSCORE_MIN_ARTICLES`] embedded articles the score is the raw
+/// top-1 cosine instead, and that is logged.
+pub fn interest_matches(
+    articles: &HashMap<ArticleId, Vec<f32>>,
+    interests: &HashMap<String, Vec<f32>>,
+) -> HashMap<ArticleId, InterestMatch> {
+    if articles.is_empty() || interests.is_empty() {
+        return HashMap::new();
+    }
+    let fallback = articles.len() < INTEREST_ZSCORE_MIN_ARTICLES;
+    if fallback {
+        tracing::info!(
+            embedded = articles.len(),
+            "fewer than {INTEREST_ZSCORE_MIN_ARTICLES} embedded articles; interest uses the raw top-1 cosine"
+        );
+    }
+
+    let mut matches: HashMap<ArticleId, Vec<TopInterest>> = HashMap::new();
+    for (name, interest) in interests {
+        let similarities = articles
+            .iter()
+            .filter_map(|(article_id, article)| {
+                dot(interest, article).ok().map(|cos| (*article_id, cos))
+            })
+            .collect::<Vec<_>>();
+        if similarities.is_empty() {
+            continue;
+        }
+        let n = similarities.len() as f64;
+        let mean = similarities.iter().map(|(_, cos)| cos).sum::<f64>() / n;
+        let variance = similarities
+            .iter()
+            .map(|(_, cos)| (cos - mean).powi(2))
+            .sum::<f64>()
+            / n;
+        let std = variance.sqrt().max(ZSCORE_STD_FLOOR);
+        for (article_id, cos) in similarities {
+            matches.entry(article_id).or_default().push(TopInterest {
+                name: name.clone(),
+                z: (cos - mean) / std,
+                cos,
+            });
+        }
+    }
+
+    matches
+        .into_iter()
+        .map(|(article_id, mut all)| {
+            all.sort_by(|left, right| {
+                right
+                    .z
+                    .total_cmp(&left.z)
+                    .then_with(|| left.name.cmp(&right.name))
+            });
+            let top1_cos = all
+                .iter()
+                .map(|item| item.cos)
+                .fold(f64::NEG_INFINITY, f64::max);
+            all.truncate(RECORDED_TOP);
+            let score = if fallback {
+                top1_cos
+            } else {
+                let top_mean = all.iter().map(|item| item.z).sum::<f64>() / all.len() as f64;
+                0.7 * all[0].z + 0.3 * top_mean
+            };
+            (
+                article_id,
+                InterestMatch {
+                    score,
+                    top1_cos,
+                    top_interests: all,
+                },
+            )
+        })
+        .collect()
+}
+
+/// Every cheap signal for the eligible set, normalized and blended (§9, §12.2,
+/// §12.4). Pure: the preference state is already loaded.
+pub fn compute(
+    articles: &[Article],
+    article_embeddings: &HashMap<ArticleId, Vec<f32>>,
+    interest_embeddings: &HashMap<String, Vec<f32>>,
+    preference: &PreferenceState,
+    ranking: &RankingConfig,
+) -> HashMap<ArticleId, Signals> {
+    let interests = interest_matches(article_embeddings, interest_embeddings);
+    let mut all = articles
+        .iter()
+        .map(|article| {
+            let mut signals = Signals::baseline(article);
+            signals.knn_gate = preference.knn_gate;
+            signals.feed_gate = preference.feed_gate;
+            if let Some(matched) = interests.get(&article.id) {
+                signals.interest = Some(matched.score);
+                signals.interest_top1_cos = Some(matched.top1_cos);
+                signals.top_interests = matched.top_interests.clone();
+            }
+            if let Some(embedding) = article_embeddings.get(&article.id) {
+                let (knn, neighbours) = preference.knn(embedding, ranking);
+                signals.knn = knn;
+                signals.neighbours = neighbours;
+            }
+            signals.feed = preference.feed(article);
+            if preference.knn_gate > 0.0 {
+                signals.notes.push(format!(
+                    "knn gate {:.2} (n={} rated with embeddings)",
+                    preference.knn_gate,
+                    preference.examples.len()
+                ));
+            }
+            signals
+        })
+        .collect::<Vec<_>>();
+
+    normalize(&mut all.iter_mut().collect::<Vec<_>>());
+    for signals in &mut all {
+        preliminary_blend(signals, &ranking.weights.preliminary);
+    }
+    articles.iter().map(|article| article.id).zip(all).collect()
+}
+
+/// [`compute`] with the preference state loaded from the database.
+pub async fn compute_all(
+    db: &Db,
+    articles: &[Article],
+    article_embeddings: &HashMap<ArticleId, Vec<f32>>,
+    interest_embeddings: &HashMap<String, Vec<f32>>,
+    voyage: &VoyageConfig,
+    ranking: &RankingConfig,
+    now: Timestamp,
+) -> anyhow::Result<(HashMap<ArticleId, Signals>, PreferenceSummary)> {
+    let preference = PreferenceState::load(db, voyage, ranking, now).await?;
+    preference.log(ranking);
+    Ok((
+        compute(
+            articles,
+            article_embeddings,
+            interest_embeddings,
+            &preference,
+            ranking,
+        ),
+        preference.summary(),
+    ))
+}
+
+/// Mid-rank percentiles over the present values of each signal (§12.2).
+///
+/// `p(x) = (count_below + (count_equal + 1) / 2) / n_present`; fewer than two
+/// present values or all-equal values give 0.5. Article id never breaks ties.
+pub fn normalize(signals: &mut [&mut Signals]) {
+    for name in PERCENTILE_SIGNALS {
+        let mut values = signals
+            .iter()
+            .filter_map(|signal| signal.raw(name))
+            .collect::<Vec<_>>();
+        if values.is_empty() {
+            continue;
+        }
+        values.sort_by(f64::total_cmp);
+        let n = values.len() as f64;
+        let constant = values.len() < 2 || values.first() == values.last();
+        for signal in signals.iter_mut() {
+            let Some(value) = signal.raw(name) else {
+                continue;
+            };
+            let percentile = if constant {
+                0.5
+            } else {
+                let below = values.partition_point(|other| *other < value);
+                let equal = values.partition_point(|other| *other <= value) - below;
+                (below as f64 + (equal as f64 + 1.0) / 2.0) / n
+            };
+            signal.norm.insert(name.to_string(), percentile);
+        }
+    }
+}
+
+/// The preliminary blend of §12.4 on a 0–100 scale: present-and-active
+/// signals only, learned weights multiplied by their gate, renormalized to 1.
+pub fn preliminary_blend(signals: &mut Signals, configured: &PreliminaryWeights) -> Option<f64> {
+    let candidates = [
+        ("interest", configured.interest, 1.0),
+        ("knn", configured.knn, signals.knn_gate),
+        ("heuristic", configured.heuristic, 1.0),
+        ("feed", configured.feed, signals.feed_gate),
+        ("social", configured.social, 1.0),
+    ];
+    let active = candidates
+        .into_iter()
+        .filter_map(|(name, weight, gate)| {
+            let norm = *signals.norm.get(name)?;
+            let effective = weight * gate;
+            (effective > 0.0).then_some((name, effective, norm))
+        })
+        .collect::<Vec<_>>();
+    let total = active.iter().map(|(_, weight, _)| weight).sum::<f64>();
+    if total <= 0.0 {
+        signals.weights.clear();
+        signals.preliminary = None;
+        return None;
+    }
+    signals.weights = active
+        .iter()
+        .map(|(name, weight, _)| ((*name).to_string(), weight / total))
+        .collect();
+    let blend = active
+        .iter()
+        .map(|(_, weight, norm)| weight / total * norm)
+        .sum::<f64>()
+        * 100.0;
+    signals.preliminary = Some(blend);
+    Some(blend)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::types::{ExtractMethod, SourceRef};
+
+    fn ranking() -> RankingConfig {
+        RankingConfig::default()
+    }
+
+    fn unit(values: &[f32]) -> Vec<f32> {
+        let norm = values.iter().map(|v| v * v).sum::<f32>().sqrt();
+        values.iter().map(|v| v / norm).collect()
+    }
+
+    fn example(id: ArticleId, label: &str, value: f64, embedding: &[f32]) -> RatedExample {
+        RatedExample {
+            article_id: id,
+            label: label.into(),
+            title: format!("rated {id}"),
+            value,
+            decay: 1.0,
+            embedding: unit(embedding),
+            feeds: vec![id],
+        }
+    }
+
+    fn article(id: ArticleId, feeds: &[FeedId]) -> Article {
+        Article {
+            id,
+            canonical_url: format!("https://example.com/{id}"),
+            title: format!("Article {id}"),
+            best_entry_id: id,
+            content_html: String::new(),
+            word_count: 1000,
+            excerpt_only: false,
+            image_count: 0,
+            sources: feeds
+                .iter()
+                .map(|feed| SourceRef {
+                    entry_id: id,
+                    feed_id: *feed,
+                    feed_title: format!("feed {feed}"),
+                    category: None,
+                    kind: SourceKind::Feed,
+                })
+                .collect(),
+            first_seen: "2026-08-15T00:00:00Z".parse().unwrap(),
+            url: format!("https://example.com/{id}"),
+            author: None,
+            feed_id: feeds.first().copied().unwrap_or(0),
+            feed_title: String::new(),
+            category: None,
+            published_at: None,
+            comments_url: None,
+            image_urls: vec![],
+            social: vec![],
+            extract_method: ExtractMethod::Miniflux,
+        }
+    }
+
+    fn with_heuristic(value: Option<f64>) -> Signals {
+        Signals {
+            heuristic: value,
+            ..Signals::default()
+        }
+    }
+
+    // --- §9.1 interest z-scores ---
+
+    fn interest_fixture(n: usize) -> (HashMap<ArticleId, Vec<f32>>, HashMap<String, Vec<f32>>) {
+        // Article 1 sits on axis x; the rest sit near axis y with a tiny spread.
+        let mut articles = HashMap::new();
+        articles.insert(1, unit(&[1.0, 0.0, 0.0]));
+        for id in 2..=n as ArticleId {
+            articles.insert(id, unit(&[0.0, 1.0, 0.001 * id as f32]));
+        }
+        // "Broad" is about equally close to everything; "Specific" matches only article 1.
+        let mut interests = HashMap::new();
+        interests.insert("Broad".to_string(), unit(&[1.0, 1.0, 0.0]));
+        interests.insert("Specific".to_string(), unit(&[1.0, 0.0, 0.0]));
+        (articles, interests)
+    }
+
+    #[test]
+    fn specific_interest_with_one_strong_match_beats_a_broad_one() {
+        let (articles, interests) = interest_fixture(40);
+        let matched = interest_matches(&articles, &interests);
+        let strong = &matched[&1];
+        assert_eq!(strong.top_interests[0].name, "Specific");
+        assert!(
+            strong.top_interests[0].z > 3.0,
+            "z = {}",
+            strong.top_interests[0].z
+        );
+        let others = (2..=40)
+            .map(|id| matched[&id].score)
+            .fold(f64::NEG_INFINITY, f64::max);
+        assert!(strong.score > others + 2.0, "{} vs {others}", strong.score);
+        // Raw cosine would have called Broad a near-tie everywhere (≈0.707).
+        assert!((matched[&2].top1_cos - 0.707).abs() < 0.01);
+    }
+
+    #[test]
+    fn interest_falls_back_to_raw_cosine_under_thirty_articles() {
+        let (articles, interests) = interest_fixture(10);
+        let matched = interest_matches(&articles, &interests);
+        for (id, m) in &matched {
+            assert!(
+                (m.score - m.top1_cos).abs() < 1e-9,
+                "article {id} should use raw top-1"
+            );
+        }
+        assert!((matched[&2].score - 0.707).abs() < 0.01);
+    }
+
+    // --- §9.2 preference ---
+
+    #[test]
+    fn one_loved_article_gives_a_positive_knn_to_a_near_neighbour() {
+        let mut ranking = ranking();
+        ranking.knn_floor = 0;
+        ranking.knn_full = 1;
+        let state = PreferenceState::build(vec![example(1, "loved", 1.0, &[1.0, 0.0])], &ranking);
+        let (knn, neighbours) = state.knn(&unit(&[0.9, 0.1]), &ranking);
+        assert!(knn.unwrap() > 0.9);
+        assert_eq!(neighbours.len(), 1);
+        assert_eq!(neighbours[0].label, "loved");
+        let (far, _) = state.knn(&unit(&[0.0, 1.0]), &ranking);
+        assert!(far.unwrap().abs() < 1e-6);
+    }
+
+    #[test]
+    fn two_unrelated_loved_clusters_both_score_high() {
+        let mut ranking = ranking();
+        ranking.knn_floor = 0;
+        ranking.knn_full = 1;
+        ranking.neighbour_k = 2;
+        let state = PreferenceState::build(
+            vec![
+                example(1, "loved", 1.0, &[1.0, 0.0, 0.0]),
+                example(2, "loved", 1.0, &[0.98, 0.02, 0.0]),
+                example(3, "loved", 1.0, &[0.0, 1.0, 0.0]),
+                example(4, "loved", 1.0, &[0.0, 0.98, 0.02]),
+            ],
+            &ranking,
+        );
+        let (near_a, _) = state.knn(&unit(&[1.0, 0.0, 0.0]), &ranking);
+        let (near_b, _) = state.knn(&unit(&[0.0, 1.0, 0.0]), &ranking);
+        assert!(near_a.unwrap() > 0.95, "{near_a:?}");
+        assert!(near_b.unwrap() > 0.95, "{near_b:?}");
+        // A centroid would have put both at ~0.7.
+    }
+
+    #[test]
+    fn good_carries_a_third_of_loved() {
+        let loved = example(1, "loved", 1.0, &[1.0, 0.0]);
+        let good = example(2, "good", 0.35, &[1.0, 0.0]);
+        assert!((good.weight() / loved.weight() - 0.35).abs() < 1e-9);
+
+        // A mixed neighbourhood: the far example pulls the mean down by 0.35× as
+        // much weight when it is merely "good" as when it is "loved".
+        let mut ranking = ranking();
+        ranking.knn_floor = 0;
+        ranking.knn_full = 1;
+        let near = example(1, "loved", 1.0, &[1.0, 0.0]);
+        let candidate = unit(&[1.0, 0.0]);
+        let both_loved = PreferenceState::build(
+            vec![near.clone(), example(2, "loved", 1.0, &[0.0, 1.0])],
+            &ranking,
+        );
+        let one_good =
+            PreferenceState::build(vec![near, example(2, "good", 0.35, &[0.0, 1.0])], &ranking);
+        let pull_loved = 1.0 - both_loved.knn(&candidate, &ranking).0.unwrap();
+        let pull_good = 1.0 - one_good.knn(&candidate, &ranking).0.unwrap();
+        assert!(pull_good < pull_loved);
+        // Weighted means: 0.5 vs 1/1.35 → pulls 0.5 vs 0.35/1.35.
+        assert!((pull_good / pull_loved - 0.35 / 1.35 / 0.5).abs() < 1e-9);
+    }
+
+    #[test]
+    fn negatives_subtract_with_the_negative_coefficient() {
+        let mut ranking = ranking();
+        ranking.knn_floor = 0;
+        ranking.knn_full = 1;
+        let state =
+            PreferenceState::build(vec![example(1, "not_for_me", -1.0, &[1.0, 0.0])], &ranking);
+        let (knn, _) = state.knn(&unit(&[1.0, 0.0]), &ranking);
+        assert!((knn.unwrap() + ranking.negative_coefficient).abs() < 1e-9);
+    }
+
+    #[test]
+    fn decay_halves_at_the_half_life() {
+        assert!((decay(60.0, 60.0) - 0.5).abs() < 1e-12);
+        assert!((decay(0.0, 60.0) - 1.0).abs() < 1e-12);
+        assert!((decay(120.0, 60.0) - 0.25).abs() < 1e-12);
+    }
+
+    #[test]
+    fn gate_is_zero_below_floor_one_at_full_and_linear_between() {
+        assert_eq!(gate(0, 8, 25), 0.0);
+        assert_eq!(gate(8, 8, 25), 0.0);
+        assert_eq!(gate(25, 8, 25), 1.0);
+        assert_eq!(gate(100, 8, 25), 1.0);
+        assert!((gate(16, 8, 24) - 0.5).abs() < 1e-9);
+        assert!((gate(9, 8, 25) - 1.0 / 17.0).abs() < 1e-9);
+    }
+
+    #[test]
+    fn knn_is_absent_when_the_gate_is_closed() {
+        let ranking = ranking(); // knn_floor 8
+        let state = PreferenceState::build(vec![example(1, "loved", 1.0, &[1.0, 0.0])], &ranking);
+        assert_eq!(state.knn_gate, 0.0);
+        assert_eq!(state.knn(&unit(&[1.0, 0.0]), &ranking), (None, Vec::new()));
+    }
+
+    // --- §9.3 feed affinity ---
+
+    #[test]
+    fn feed_credit_sums_to_one_across_direct_feeds() {
+        let mut rated = example(1, "loved", 1.0, &[1.0, 0.0]);
+        rated.feeds = vec![10, 20, 30];
+        let state = PreferenceState::build(vec![rated], &ranking());
+        let total: f64 = [10, 20, 30]
+            .iter()
+            .map(|feed| state.feed_credit(*feed).unwrap().0)
+            .sum();
+        assert!((total - 1.0).abs() < 1e-9);
+        assert!((state.feed_credit(10).unwrap().0 - 1.0 / 3.0).abs() < 1e-9);
+    }
+
+    #[test]
+    fn feed_affinity_is_the_mean_over_rated_feeds() {
+        let mut ranking = ranking();
+        ranking.feed_floor = 0;
+        ranking.feed_full = 1;
+        let mut loved = example(1, "loved", 1.0, &[1.0, 0.0]);
+        loved.feeds = vec![10];
+        let mut down = example(2, "not_for_me", -1.0, &[1.0, 0.0]);
+        down.feeds = vec![20];
+        let state = PreferenceState::build(vec![loved, down], &ranking);
+        // feed 10: (1+1)/(1+0+2) = 2/3; feed 20: (0+1)/(0+1+2) = 1/3; unrated 99 ignored.
+        let both = state.feed(&article(7, &[10, 20, 99])).unwrap();
+        assert!((both - 0.5).abs() < 1e-9, "{both}");
+        let best = state.feed(&article(8, &[10])).unwrap();
+        assert!((best - 2.0 / 3.0).abs() < 1e-9);
+        assert_eq!(state.feed(&article(9, &[99])), None);
+    }
+
+    #[test]
+    fn feed_is_absent_when_the_gate_is_closed() {
+        let ranking = ranking(); // feed_floor 15
+        let mut loved = example(1, "loved", 1.0, &[1.0, 0.0]);
+        loved.feeds = vec![10];
+        let state = PreferenceState::build(vec![loved], &ranking);
+        assert_eq!(state.feed_gate, 0.0);
+        assert_eq!(state.feed(&article(7, &[10])), None);
+    }
+
+    #[test]
+    fn direct_feeds_fall_back_to_the_best_entry_feed() {
+        let mut a = article(1, &[]);
+        a.feed_id = 42;
+        assert_eq!(direct_feeds(&a), vec![42]);
+        assert_eq!(direct_feeds(&article(2, &[5, 3, 5])), vec![3, 5]);
+    }
+
+    // --- §12.2 normalization, §12.4 blend ---
+
+    #[test]
+    fn constant_signal_normalizes_to_half_for_everyone() {
+        let mut values = [with_heuristic(Some(7.0)), with_heuristic(Some(7.0))];
+        let mut refs = values.iter_mut().collect::<Vec<_>>();
+        normalize(&mut refs);
+        assert!(values.iter().all(|v| v.norm["heuristic"] == 0.5));
+        let mut single = [with_heuristic(Some(3.0))];
+        let mut refs = single.iter_mut().collect::<Vec<_>>();
+        normalize(&mut refs);
+        assert_eq!(single[0].norm["heuristic"], 0.5);
+    }
+
+    #[test]
+    fn ties_get_equal_percentiles_without_an_id_ramp() {
+        let mut values = (0..400)
+            .map(|_| with_heuristic(Some(0.0)))
+            .collect::<Vec<_>>();
+        let mut refs = values.iter_mut().collect::<Vec<_>>();
+        normalize(&mut refs);
+        assert!(values.iter().all(|v| v.norm["heuristic"] == 0.5));
+
+        let mut mixed = [
+            with_heuristic(Some(1.0)),
+            with_heuristic(Some(2.0)),
+            with_heuristic(Some(2.0)),
+            with_heuristic(Some(3.0)),
+        ];
+        let mut refs = mixed.iter_mut().collect::<Vec<_>>();
+        normalize(&mut refs);
+        assert_eq!(mixed[0].norm["heuristic"], 0.25);
+        assert_eq!(mixed[1].norm["heuristic"], 0.625);
+        assert_eq!(mixed[2].norm["heuristic"], 0.625);
+        assert_eq!(mixed[3].norm["heuristic"], 1.0);
+    }
+
+    #[test]
+    fn absent_values_do_not_shift_present_values() {
+        let mut values = [
+            with_heuristic(Some(1.0)),
+            with_heuristic(Some(2.0)),
+            with_heuristic(None),
+        ];
+        let mut refs = values.iter_mut().collect::<Vec<_>>();
+        normalize(&mut refs);
+        assert!(!values[2].norm.contains_key("heuristic"));
+        // n_present = 2: the absent third value does not widen the scale.
+        assert_eq!(values[0].norm["heuristic"], 0.5);
+        assert_eq!(values[1].norm["heuristic"], 1.0);
+    }
+
+    #[test]
+    fn effective_weights_sum_to_one_and_missing_signals_are_skipped() {
+        let mut signals = Signals {
+            interest: Some(1.0),
+            heuristic: Some(2.0),
+            norm: BTreeMap::from([("interest".into(), 0.8), ("heuristic".into(), 0.4)]),
+            ..Signals::default()
+        };
+        let blend = preliminary_blend(&mut signals, &PreliminaryWeights::default()).unwrap();
+        assert!((signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
+        assert!(!signals.weights.contains_key("knn"));
+        assert!(!signals.weights.contains_key("social"));
+        // 0.35/0.55 × 0.8 + 0.20/0.55 × 0.4 = 0.6545…
+        assert!((blend - 65.4545).abs() < 0.01, "{blend}");
+
+        let mut only_heuristic = Signals {
+            heuristic: Some(2.0),
+            norm: BTreeMap::from([("heuristic".into(), 0.4)]),
+            ..Signals::default()
+        };
+        let blend = preliminary_blend(&mut only_heuristic, &PreliminaryWeights::default());
+        assert!((blend.unwrap() - 40.0).abs() < 1e-9);
+        assert_eq!(only_heuristic.weights["heuristic"], 1.0);
+    }
+
+    #[test]
+    fn learned_weights_are_multiplied_by_their_gate() {
+        let mut signals = Signals {
+            knn: Some(0.5),
+            heuristic: Some(2.0),
+            knn_gate: 0.5,
+            norm: BTreeMap::from([("knn".into(), 1.0), ("heuristic".into(), 0.0)]),
+            ..Signals::default()
+        };
+        preliminary_blend(&mut signals, &PreliminaryWeights::default());
+        // knn 0.25 × 0.5 = 0.125 against heuristic 0.20.
+        assert!((signals.weights["knn"] - 0.125 / 0.325).abs() < 1e-9);
+    }
+
+    #[test]
+    fn compute_scores_every_article_and_leaves_ungated_signals_absent() {
+        let articles = vec![article(1, &[10]), article(2, &[20]), article(3, &[30])];
+        let mut embeddings = HashMap::new();
+        embeddings.insert(1, unit(&[1.0, 0.0]));
+        embeddings.insert(2, unit(&[0.0, 1.0]));
+        let mut interests = HashMap::new();
+        interests.insert("Axis".to_string(), unit(&[1.0, 0.0]));
+        let state = PreferenceState::build(vec![example(9, "loved", 1.0, &[1.0, 0.0])], &ranking());
+        let signals = compute(&articles, &embeddings, &interests, &state, &ranking());
+        assert_eq!(signals.len(), 3);
+        assert!(signals[&1].interest.is_some());
+        assert!(signals[&3].interest.is_none(), "no embedding → absent");
+        assert!(
+            signals
+                .values()
+                .all(|s| s.knn.is_none() && s.feed.is_none())
+        );
+        assert!(
+            signals
+                .values()
+                .all(|s| s.heuristic.is_some() && s.preliminary.is_some())
+        );
+    }
+}

src/db.rs

@@ -2,9 +2,10 @@
 //!
 //! Runtime queries only — no `sqlx::query!` macros (implementation notes §1).
 //! Timestamps are stored as RFC3339 UTC strings and dates as `YYYY-MM-DD`
-//! (implementation notes §2). Every write is an idempotent upsert so that
+//! (implementation notes §2). Pipeline writes are idempotent upserts so that
-//! `generate --date X` can be re-run safely (implementation notes §12).
+//! `generate --date X` can be re-run safely; feedback events are append-only.
 
+use std::collections::BTreeMap;
 use std::path::Path;
 use std::str::FromStr;
 use std::time::Duration;
@@ -15,8 +16,8 @@ use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePoolOptions, S
 use sqlx::{Row, SqlitePool};
 
 use crate::types::{
-    Article, ArticleId, Entry, EntryId, FeedId, FeedPrior, LlmScore, Pick, Rating, SocialRef,
+    Article, ArticleId, Entry, EntryId, Facets, Pick, RatedArticle, RatingEvent, SocialRef,
-    SocialSource, SourceRef, Vote,
+    SocialSource, SourceRef,
 };
 
 /// Embedded migrations from `./migrations` (implementation notes §1).
@@ -24,9 +25,9 @@ pub static MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations");
 
 /// `kv` key holding the ingest watermark (§3.1).
 pub const KV_WATERMARK: &str = "ingest_watermark";
-/// `kv` key holding the current taste profile document (§3.6).
+/// `kv` key holding the current system-prompt profile document (§8.4).
 pub const KV_TASTE_PROFILE: &str = "taste_profile";
-/// `kv` key holding the taste profile version/build time (§3.6).
+/// `kv` key holding the profile version/build time (§8.2).
 pub const KV_PROFILE_VERSION: &str = "profile_version";
 
 #[derive(Debug, thiserror::Error)]
@@ -43,6 +44,12 @@ pub enum DbError {
     },
     #[error("malformed value in column `{column}`: {value}")]
     Decode { column: &'static str, value: String },
+    #[error("malformed JSON in column `{column}`: {source}")]
+    Json {
+        column: &'static str,
+        #[source]
+        source: serde_json::Error,
+    },
 }
 
 type Result<T> = std::result::Result<T, DbError>;
@@ -100,7 +107,7 @@ impl Db {
     }
 
     // -----------------------------------------------------------------
-    // kv (§3.1 watermark, §3.6 taste profile)
+    // kv (§3.1 watermark, §8 profile)
     // -----------------------------------------------------------------
 
     pub async fn kv_get(&self, key: &str) -> Result<Option<String>> {
@@ -321,21 +328,43 @@ impl Db {
         Ok(rows.iter().map(|r| r.get::<i64, _>("article_id")).collect())
     }
 
-    /// Articles the LLM scored below `threshold` within the last `days` (§3.5).
+    /// Article ids published before this issue date; same-date regeneration is allowed (§8.1).
-    pub async fn recently_low_scored_ids(
+    pub async fn previously_published_ids_before(&self, date: Date) -> Result<Vec<ArticleId>> {
-        &self,
+        let rows =
-        threshold: f64,
+            sqlx::query("SELECT DISTINCT article_id FROM issue_articles WHERE issue_date < ?")
-        since: Date,
+                .bind(date.to_string())
-    ) -> Result<Vec<ArticleId>> {
+                .fetch_all(&self.pool)
+                .await?;
+        Ok(rows
+            .iter()
+            .map(|row| row.get::<i64, _>("article_id"))
+            .collect())
+    }
+
+    /// Published article ids first seen at or after `since` (`features backfill`).
+    pub async fn published_article_ids_since(&self, since: Timestamp) -> Result<Vec<ArticleId>> {
         let rows = sqlx::query(
-            "SELECT DISTINCT article_id FROM scores
+            "SELECT DISTINCT ia.article_id FROM issue_articles ia
-             WHERE llm_score IS NOT NULL AND llm_score < ? AND run_date >= ?",
+             JOIN articles a ON a.id = ia.article_id
+             WHERE a.first_seen >= ?
+             ORDER BY ia.article_id",
         )
-        .bind(threshold)
+        .bind(fmt_ts(since))
-        .bind(since.to_string())
         .fetch_all(&self.pool)
         .await?;
-        Ok(rows.iter().map(|r| r.get::<i64, _>("article_id")).collect())
+        Ok(rows
+            .iter()
+            .map(|row| row.get::<i64, _>("article_id"))
+            .collect())
+    }
+
+    /// Every article id first seen at or after `since` (`features backfill --all`).
+    pub async fn article_ids_since(&self, since: Timestamp) -> Result<Vec<ArticleId>> {
+        let rows = sqlx::query("SELECT id FROM articles WHERE first_seen >= ? ORDER BY id")
+            .bind(fmt_ts(since))
+            .fetch_all(&self.pool)
+            .await?;
+        Ok(rows.iter().map(|row| row.get::<i64, _>("id")).collect())
     }
 
     // -----------------------------------------------------------------
@@ -376,37 +405,6 @@ impl Db {
         rows.iter().map(social_from_row).collect()
     }
 
-    // -----------------------------------------------------------------
-    // scores (§3.5, §3.6)
-    // -----------------------------------------------------------------
-
-    pub async fn upsert_score(
-        &self,
-        article_id: ArticleId,
-        run_date: Date,
-        prefilter_score: Option<f64>,
-        llm: Option<&LlmScore>,
-    ) -> Result<()> {
-        sqlx::query(
-            "INSERT INTO scores (article_id, run_date, prefilter_score, llm_score, llm_category, rationale)
-             VALUES (?, ?, ?, ?, ?, ?)
-             ON CONFLICT(article_id, run_date) DO UPDATE SET
-                 prefilter_score = COALESCE(excluded.prefilter_score, scores.prefilter_score),
-                 llm_score = COALESCE(excluded.llm_score, scores.llm_score),
-                 llm_category = COALESCE(excluded.llm_category, scores.llm_category),
-                 rationale = COALESCE(excluded.rationale, scores.rationale)",
-        )
-        .bind(article_id)
-        .bind(run_date.to_string())
-        .bind(prefilter_score)
-        .bind(llm.map(|l| l.score))
-        .bind(llm.map(|l| l.category.as_str()))
-        .bind(llm.map(|l| l.rationale.as_str()))
-        .execute(&self.pool)
-        .await?;
-        Ok(())
-    }
-
     // -----------------------------------------------------------------
     // issues + lineup (§3.10)
     // -----------------------------------------------------------------
@@ -480,8 +478,8 @@ impl Db {
             .await?;
         for pick in picks {
             sqlx::query(
-                "INSERT INTO issue_articles (issue_date, article_id, section, position, is_lead, summary)
+                "INSERT INTO issue_articles (issue_date, article_id, section, position, is_lead, summary, why)
-                 VALUES (?, ?, ?, ?, ?, ?)",
+                 VALUES (?, ?, ?, ?, ?, ?, ?)",
             )
             .bind(date.to_string())
             .bind(pick.article.id)
@@ -489,6 +487,7 @@ impl Db {
             .bind(pick.position)
             .bind(pick.is_lead)
             .bind(pick.summary.as_deref())
+            .bind(pick.why.as_deref())
             .execute(&mut *tx)
             .await?;
         }
@@ -535,117 +534,108 @@ impl Db {
     }
 
     // -----------------------------------------------------------------
-    // ratings + feed priors (§3.9)
+    // append-only rating events (§6.2)
     // -----------------------------------------------------------------
 
-    /// Idempotent upsert of a reader vote (§3.9). Returns true if it changed anything.
+    /// Append one feedback event and return its database id.
-    pub async fn upsert_rating(&self, rating: &Rating) -> Result<bool> {
+    pub async fn append_rating_event(&self, event: &RatingEvent) -> Result<i64> {
-        let res = sqlx::query(
+        let row = sqlx::query(
-            "INSERT INTO ratings (issue_date, article_id, vote, rated_at)
+            "INSERT INTO rating_events
-             VALUES (?, ?, ?, ?)
+                 (article_id, issue_date, kind, source, label, value, note, event_at)
-             ON CONFLICT(issue_date, article_id) DO UPDATE SET
+             VALUES (?, ?, ?, ?, ?, ?, ?, ?)
-                 vote = excluded.vote,
+             RETURNING id",
-                 rated_at = excluded.rated_at
-             WHERE ratings.vote != excluded.vote",
         )
-        .bind(rating.issue_date.to_string())
+        .bind(event.article_id)
-        .bind(rating.article_id)
+        .bind(event.issue_date.map(|date| date.to_string()))
-        .bind(rating.vote.as_i64())
+        .bind(&event.kind)
-        .bind(fmt_ts(rating.rated_at))
+        .bind(&event.source)
-        .execute(&self.pool)
+        .bind(&event.label)
+        .bind(event.value)
+        .bind(event.note.as_deref())
+        .bind(fmt_ts(event.event_at))
+        .fetch_one(&self.pool)
         .await?;
-        Ok(res.rows_affected() > 0)
+        Ok(row.get("id"))
     }
 
-    /// Every rating joined to the feed that carried the article (§3.9 priors).
+    /// Latest issue containing an article, used to attach CLI feedback when possible.
-    pub async fn ratings_with_feed(&self) -> Result<Vec<(FeedId, Vote)>> {
+    pub async fn latest_issue_date_for_article(
+        &self,
+        article_id: ArticleId,
+    ) -> Result<Option<Date>> {
+        let row = sqlx::query(
+            "SELECT issue_date FROM issue_articles
+             WHERE article_id = ? ORDER BY issue_date DESC LIMIT 1",
+        )
+        .bind(article_id)
+        .fetch_optional(&self.pool)
+        .await?;
+        row.map(|row| {
+            parse_date(
+                "issue_articles.issue_date",
+                &row.get::<String, _>("issue_date"),
+            )
+        })
+        .transpose()
+    }
+
+    /// Current explicit verdicts, newest first. A latest `cleared` event removes
+    /// its article from this learned set (§6.2).
+    pub async fn current_ratings(&self, lookback_days: i64) -> Result<Vec<RatedArticle>> {
+        self.latest_explicit_ratings(lookback_days, false).await
+    }
+
+    /// Current explicit events including `cleared`, for the ratings CLI.
+    pub async fn current_ratings_including_cleared(
+        &self,
+        lookback_days: i64,
+    ) -> Result<Vec<RatedArticle>> {
+        self.latest_explicit_ratings(lookback_days, true).await
+    }
+
+    async fn latest_explicit_ratings(
+        &self,
+        lookback_days: i64,
+        include_cleared: bool,
+    ) -> Result<Vec<RatedArticle>> {
+        let since = Timestamp::now()
+            .checked_sub(jiff::Span::new().hours(lookback_days.max(0).saturating_mul(24)))
+            .unwrap_or(Timestamp::UNIX_EPOCH);
         let rows = sqlx::query(
-            "SELECT e.feed_id AS feed_id, r.vote AS vote
+            "WITH ranked AS (
-             FROM ratings r
+                 SELECT re.*,
+                        ROW_NUMBER() OVER (
+                            PARTITION BY re.article_id
+                            ORDER BY re.event_at DESC, re.id DESC
+                        ) AS event_rank
+                 FROM rating_events re
+                 WHERE re.kind = 'explicit' AND re.event_at >= ?
+             )
+             SELECT r.article_id, r.issue_date, r.label, r.value, r.note, r.event_at,
+                    COALESCE(a.title, '') AS title,
+                    COALESCE(e.feed_title, '') AS feed_title,
+                    (SELECT ia.summary FROM issue_articles ia
+                     WHERE ia.article_id = r.article_id
+                     ORDER BY ia.issue_date DESC LIMIT 1) AS summary,
+                    aa.facets_json AS facets_json
+             FROM ranked r
              JOIN articles a ON a.id = r.article_id
-             JOIN entries e ON e.id = a.best_entry_id",
+             LEFT JOIN entries e ON e.id = a.best_entry_id
+             LEFT JOIN article_assessments aa
+                    ON aa.article_id = r.article_id AND aa.stage = 'deep'
+             WHERE r.event_rank = 1 AND (? OR r.label != 'cleared')
+             ORDER BY r.event_at DESC, r.id DESC",
         )
+        .bind(fmt_ts(since))
+        .bind(include_cleared)
         .fetch_all(&self.pool)
         .await?;
-        Ok(rows
-            .iter()
-            .map(|r| {
-                let vote = if r.get::<i64, _>("vote") >= 0 {
-                    Vote::Up
-                } else {
-                    Vote::Down
-                };
-                (r.get::<i64, _>("feed_id"), vote)
-            })
-            .collect())
-    }
 
-    /// Recent ratings with article titles, for the weekly profile rewrite (§3.6).
+        rows.iter().map(rated_article_from_row).collect()
-    pub async fn recent_ratings_detailed(&self, since: Date) -> Result<Vec<(Rating, String)>> {
-        let rows = sqlx::query(
-            "SELECT r.issue_date AS issue_date, r.article_id AS article_id, r.vote AS vote,
-                    r.rated_at AS rated_at, a.title AS title
-             FROM ratings r JOIN articles a ON a.id = r.article_id
-             WHERE r.issue_date >= ? ORDER BY r.rated_at DESC",
-        )
-        .bind(since.to_string())
-        .fetch_all(&self.pool)
-        .await?;
-        rows.iter()
-            .map(|r| {
-                let rating = Rating {
-                    issue_date: parse_date(
-                        "ratings.issue_date",
-                        &r.get::<String, _>("issue_date"),
-                    )?,
-                    article_id: r.get::<i64, _>("article_id"),
-                    vote: if r.get::<i64, _>("vote") >= 0 {
-                        Vote::Up
-                    } else {
-                        Vote::Down
-                    },
-                    rated_at: parse_ts("ratings.rated_at", &r.get::<String, _>("rated_at"))?,
-                };
-                Ok((rating, r.get::<String, _>("title")))
-            })
-            .collect()
-    }
-
-    pub async fn upsert_feed_prior(&self, prior: &FeedPrior) -> Result<()> {
-        sqlx::query(
-            "INSERT INTO feed_priors (feed_id, upvotes, downvotes, included)
-             VALUES (?, ?, ?, ?)
-             ON CONFLICT(feed_id) DO UPDATE SET
-                 upvotes = excluded.upvotes,
-                 downvotes = excluded.downvotes,
-                 included = excluded.included",
-        )
-        .bind(prior.feed_id)
-        .bind(prior.upvotes)
-        .bind(prior.downvotes)
-        .bind(prior.included)
-        .execute(&self.pool)
-        .await?;
-        Ok(())
-    }
-
-    pub async fn feed_priors(&self) -> Result<Vec<FeedPrior>> {
-        let rows = sqlx::query("SELECT feed_id, upvotes, downvotes, included FROM feed_priors")
-            .fetch_all(&self.pool)
-            .await?;
-        Ok(rows
-            .iter()
-            .map(|r| FeedPrior {
-                feed_id: r.get::<i64, _>("feed_id"),
-                upvotes: r.get::<i64, _>("upvotes"),
-                downvotes: r.get::<i64, _>("downvotes"),
-                included: r.get::<i64, _>("included"),
-            })
-            .collect())
     }
 
     // -----------------------------------------------------------------
-    // runs (§3.6 cost guardrail, §3.13)
+    // runs (§3.13, plan §7.6)
     // -----------------------------------------------------------------
 
     /// Insert a `running` row at the top of `generate`; returns `runs.id`.
@@ -665,7 +655,7 @@ impl Db {
         sqlx::query(
             "UPDATE runs SET finished_at = ?, entries_fetched = ?, candidates = ?, selected = ?,
                              input_tokens = ?, cached_tokens = ?, output_tokens = ?, cost_usd = ?,
-                             status = ?, error = ?
+                             status = ?, error = ?, provider_costs_json = ?, config_json = ?
              WHERE id = ?",
         )
         .bind(report.finished_at.map(fmt_ts))
@@ -678,20 +668,55 @@ impl Db {
         .bind(report.cost_usd)
         .bind(report.status.as_str())
         .bind(report.error.as_deref())
+        .bind(
+            serde_json::to_string(&report.provider_costs).map_err(|source| DbError::Json {
+                column: "runs.provider_costs_json",
+                source,
+            })?,
+        )
+        .bind(
+            serde_json::to_string(&report.config_json).map_err(|source| DbError::Json {
+                column: "runs.config_json",
+                source,
+            })?,
+        )
         .bind(id)
         .execute(&self.pool)
         .await?;
         Ok(())
     }
 
-    /// Total spend recorded for a date, for the `max_daily_usd` guardrail (§3.6).
+    /// Earlier provider spend on the UTC date containing this run's start (§5).
-    pub async fn spend_for_date(&self, date: Date) -> Result<f64> {
+    pub async fn provider_spend_for_utc_day(
-        let row =
+        &self,
-            sqlx::query("SELECT COALESCE(SUM(cost_usd), 0.0) AS total FROM runs WHERE date = ?")
+        started_at: Timestamp,
-                .bind(date.to_string())
+    ) -> Result<BTreeMap<String, f64>> {
-                .fetch_one(&self.pool)
+        let utc_date = started_at
-                .await?;
+            .to_zoned(jiff::tz::TimeZone::UTC)
-        Ok(row.get::<f64, _>("total"))
+            .date()
+            .to_string();
+        let rows = sqlx::query(
+            "SELECT provider_costs_json FROM runs
+             WHERE substr(started_at, 1, 10) = ? AND started_at < ?
+               AND provider_costs_json IS NOT NULL",
+        )
+        .bind(utc_date)
+        .bind(fmt_ts(started_at))
+        .fetch_all(&self.pool)
+        .await?;
+        let mut totals = BTreeMap::new();
+        for row in rows {
+            let raw = row.get::<String, _>("provider_costs_json");
+            let providers: BTreeMap<String, crate::report::ProviderUsage> =
+                serde_json::from_str(&raw).map_err(|source| DbError::Json {
+                    column: "runs.provider_costs_json",
+                    source,
+                })?;
+            for (provider, usage) in providers {
+                *totals.entry(provider).or_insert(0.0) += usage.cost_usd;
+            }
+        }
+        Ok(totals)
     }
 }
 
@@ -787,6 +812,34 @@ fn article_from_row(row: &sqlx::sqlite::SqliteRow) -> Result<Article> {
     })
 }
 
+fn rated_article_from_row(row: &sqlx::sqlite::SqliteRow) -> Result<RatedArticle> {
+    let issue_date = row
+        .get::<Option<String>, _>("issue_date")
+        .map(|raw| parse_date("rating_events.issue_date", &raw))
+        .transpose()?;
+    let facets = row.get::<Option<String>, _>("facets_json").and_then(|raw| {
+        match serde_json::from_str::<Facets>(&raw) {
+            Ok(facets) => Some(facets),
+            Err(error) => {
+                tracing::warn!(%error, "ignoring malformed assessment facets");
+                None
+            }
+        }
+    });
+    Ok(RatedArticle {
+        article_id: row.get("article_id"),
+        issue_date,
+        title: row.get("title"),
+        feed_title: row.get("feed_title"),
+        summary: row.get("summary"),
+        facets,
+        note: row.get("note"),
+        value: row.get("value"),
+        label: row.get("label"),
+        event_at: parse_ts("rating_events.event_at", &row.get::<String, _>("event_at"))?,
+    })
+}
+
 fn social_from_row(row: &sqlx::sqlite::SqliteRow) -> Result<SocialRef> {
     let raw: String = row.get("source");
     let source = SocialSource::parse(&raw).ok_or(DbError::Decode {
@@ -969,7 +1022,7 @@ mod tests {
         report.counts.entries_fetched = 412;
         report.counts.candidates = 120;
         report.counts.selected = 20;
-        report.finish(ts("2026-08-15T05:36:00Z"), 0.14, 0.0028, 0.28);
+        report.finish(ts("2026-08-15T05:36:00Z"));
         db.finish_run(run_id, &report).await.unwrap();
 
         db.upsert_issue(
@@ -987,24 +1040,270 @@ mod tests {
         let next: Date = "2026-08-16".parse().unwrap();
         assert_eq!(db.next_issue_number(next).await.unwrap(), 2);
         assert_eq!(db.recent_reports(5).await.unwrap().len(), 1);
-        assert_eq!(db.spend_for_date(date).await.unwrap(), 0.0);
+        // No provider spend was recorded, so the budget-day preload is empty.
+        let spend = db
+            .provider_spend_for_utc_day(ts("2026-08-15T23:00:00Z"))
+            .await
+            .unwrap();
+        assert!(spend.values().all(|usd| *usd == 0.0));
+    }
+
+    async fn record_run(db: &Db, date: Date, started: &str, deepseek: f64, anthropic: f64) {
+        use crate::report::{ProviderUsage, RunReport};
+        let started_at = ts(started);
+        let run_id = db.start_run(date, started_at).await.unwrap();
+        let mut report = RunReport::new(date, started_at);
+        report.provider_costs.insert(
+            "deepseek".into(),
+            ProviderUsage {
+                usage: crate::types::TokenUsage {
+                    input_tokens: 10,
+                    ..Default::default()
+                },
+                cost_usd: deepseek,
+            },
+        );
+        report.provider_costs.insert(
+            "anthropic".into(),
+            ProviderUsage {
+                usage: crate::types::TokenUsage::default(),
+                cost_usd: anthropic,
+            },
+        );
+        report.config_json = serde_json::json!({"models": {"editor": "claude-opus-5"}});
+        report.finish(started_at);
+        db.finish_run(run_id, &report).await.unwrap();
     }
 
     #[tokio::test]
-    async fn ratings_upsert_is_idempotent() {
+    async fn provider_spend_is_summed_by_the_utc_day_of_started_at() {
         let (_dir, db) = temp_db().await;
-        let rating = Rating {
+        let date: Date = "2026-08-15".parse().unwrap();
-            issue_date: "2026-08-15".parse().unwrap(),
+        record_run(&db, date, "2026-08-15T03:00:00Z", 0.10, 0.50).await;
-            article_id: 1,
+        record_run(&db, date, "2026-08-15T23:30:00Z", 0.05, 0.0).await;
-            vote: Vote::Up,
+        // Nominal issue date 08-15 in New York, but already 08-16 in UTC: a
-            rated_at: ts("2026-08-15T12:00:00Z"),
+        // different budget day (§5).
+        record_run(&db, date, "2026-08-16T01:00:00Z", 1.0, 1.0).await;
+
+        let spend = db
+            .provider_spend_for_utc_day(ts("2026-08-15T23:45:00Z"))
+            .await
+            .unwrap();
+        assert!((spend["deepseek"] - 0.15).abs() < 1e-9);
+        assert!((spend["anthropic"] - 0.5).abs() < 1e-9);
+        // Only runs that started earlier than this one count.
+        let spend = db
+            .provider_spend_for_utc_day(ts("2026-08-15T12:00:00Z"))
+            .await
+            .unwrap();
+        assert!((spend["deepseek"] - 0.10).abs() < 1e-9);
+        let spend = db
+            .provider_spend_for_utc_day(ts("2026-08-17T12:00:00Z"))
+            .await
+            .unwrap();
+        assert!(spend.is_empty());
+
+        // `provider_costs_json` and `config_json` were written and round-trip.
+        let row =
+            sqlx::query("SELECT provider_costs_json, config_json FROM runs ORDER BY id LIMIT 1")
+                .fetch_one(&db.pool)
+                .await
+                .unwrap();
+        let costs: BTreeMap<String, crate::report::ProviderUsage> =
+            serde_json::from_str(&row.get::<String, _>("provider_costs_json")).unwrap();
+        assert_eq!(costs["deepseek"].usage.input_tokens, 10);
+        assert!((costs["anthropic"].cost_usd - 0.5).abs() < 1e-9);
+        let config: serde_json::Value =
+            serde_json::from_str(&row.get::<String, _>("config_json")).unwrap();
+        assert_eq!(config["models"]["editor"], "claude-opus-5");
+    }
+
+    #[tokio::test]
+    async fn latest_explicit_rating_wins_and_clear_removes_it() {
+        let (_dir, db) = temp_db().await;
+        db.upsert_entry(&sample_entry(1)).await.unwrap();
+        let article = Article {
+            id: 0,
+            canonical_url: "https://example.com/1".into(),
+            title: "Story 1".into(),
+            best_entry_id: 1,
+            content_html: "<p>body</p>".into(),
+            word_count: 900,
+            excerpt_only: false,
+            image_count: 0,
+            sources: vec![],
+            first_seen: ts("2026-08-15T05:30:00Z"),
+            url: "https://example.com/1".into(),
+            author: None,
+            feed_id: 7,
+            feed_title: "Hacker News".into(),
+            category: None,
+            published_at: None,
+            comments_url: None,
+            image_urls: vec![],
+            social: vec![],
+            extract_method: ExtractMethod::Miniflux,
         };
-        assert!(db.upsert_rating(&rating).await.unwrap());
+        let article_id = db.upsert_article(&article).await.unwrap();
-        assert!(!db.upsert_rating(&rating).await.unwrap());
+        for (date, number, summary) in [
-        let flipped = Rating {
+            ("2026-08-14", 1, "Older summary"),
-            vote: Vote::Down,
+            ("2026-08-15", 2, "Newest summary"),
-            ..rating.clone()
+        ] {
+            db.upsert_issue(
+                date.parse().unwrap(),
+                number,
+                ts("2026-08-15T05:30:00Z"),
+                None,
+                None,
+                None,
+                None,
+                None,
+            )
+            .await
+            .unwrap();
+            sqlx::query(
+                "INSERT INTO issue_articles
+                     (issue_date, article_id, section, position, is_lead, summary)
+                 VALUES (?, ?, 'Top Stories', 1, 0, ?)",
+            )
+            .bind(date)
+            .bind(article_id)
+            .bind(summary)
+            .execute(db.pool())
+            .await
+            .unwrap();
+        }
+        sqlx::query(
+            "INSERT INTO article_assessments
+                 (article_id, stage, model, prompt_version, facets_json, assessed_at)
+             VALUES (?, 'deep', 'mock', 1, ?, '2026-08-15T11:00:00Z')",
+        )
+        .bind(article_id)
+        .bind(r#"{"format":"analysis_essay","depth":"deep","evidence":null,"commerciality":null,"topic_group":"software_engineering","technicality":"advanced","locality":null,"specific_topics":null}"#)
+        .execute(db.pool())
+        .await
+        .unwrap();
+        let event = |label: &str, value: f64, at: &str| RatingEvent {
+            id: 0,
+            article_id,
+            issue_date: Some("2026-08-15".parse().unwrap()),
+            kind: "explicit".into(),
+            source: "cli".into(),
+            label: label.into(),
+            value,
+            note: None,
+            event_at: ts(at),
         };
-        assert!(db.upsert_rating(&flipped).await.unwrap());
+        db.append_rating_event(&event("loved", 1.0, "2026-08-15T12:00:00Z"))
+            .await
+            .unwrap();
+        db.append_rating_event(&event("good", 0.35, "2026-08-15T13:00:00Z"))
+            .await
+            .unwrap();
+        let mut implicit = event("read_fully", 0.5, "2026-08-15T13:30:00Z");
+        implicit.kind = "implicit".into();
+        implicit.source = "bookorbit".into();
+        db.append_rating_event(&implicit).await.unwrap();
+        let ratings = db.current_ratings(36500).await.unwrap();
+        assert_eq!(ratings.len(), 1);
+        assert_eq!(ratings[0].label, "good");
+        assert_eq!(ratings[0].feed_title, "Hacker News");
+        assert_eq!(ratings[0].summary.as_deref(), Some("Newest summary"));
+        assert_eq!(
+            ratings[0]
+                .facets
+                .as_ref()
+                .and_then(|facets| facets.format.as_deref()),
+            Some("analysis_essay")
+        );
+
+        db.append_rating_event(&event("cleared", 0.0, "2026-08-15T14:00:00Z"))
+            .await
+            .unwrap();
+        assert!(db.current_ratings(36500).await.unwrap().is_empty());
+        let events = db.current_ratings_including_cleared(36500).await.unwrap();
+        assert_eq!(events.len(), 1);
+        assert_eq!(events[0].label, "cleared");
+        let sources: Vec<String> =
+            sqlx::query_scalar("SELECT source FROM rating_events ORDER BY id")
+                .fetch_all(db.pool())
+                .await
+                .unwrap();
+        assert_eq!(sources, ["cli", "cli", "bookorbit", "cli"]);
+    }
+
+    #[tokio::test]
+    async fn curation_v2_migration_copies_ratings_and_drops_old_tables() {
+        let pool = SqlitePoolOptions::new()
+            .max_connections(1)
+            .connect("sqlite::memory:")
+            .await
+            .unwrap();
+        sqlx::raw_sql(include_str!("../migrations/0001_init.sql"))
+            .execute(&pool)
+            .await
+            .unwrap();
+        sqlx::query(
+            "INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
+             (1, 'https://example.com/loved', 'Loved', '2026-08-15T00:00:00Z'),
+             (2, 'https://example.com/down', 'Down', '2026-08-15T00:00:00Z')",
+        )
+        .execute(&pool)
+        .await
+        .unwrap();
+        sqlx::query(
+            "INSERT INTO ratings (issue_date, article_id, vote, rated_at) VALUES
+             ('2026-08-15', 1, 1, '2026-08-15T12:00:00Z'),
+             ('2026-08-15', 2, -1, '2026-08-15T13:00:00Z')",
+        )
+        .execute(&pool)
+        .await
+        .unwrap();
+
+        sqlx::raw_sql(include_str!("../migrations/0002_curation_v2.sql"))
+            .execute(&pool)
+            .await
+            .unwrap();
+        sqlx::raw_sql(include_str!("../migrations/0003_drop_scores.sql"))
+            .execute(&pool)
+            .await
+            .unwrap();
+
+        let rows = sqlx::query(
+            "SELECT article_id, issue_date, kind, source, label, value, event_at
+             FROM rating_events ORDER BY article_id",
+        )
+        .fetch_all(&pool)
+        .await
+        .unwrap();
+        assert_eq!(rows.len(), 2);
+        assert_eq!(rows[0].get::<String, _>("label"), "loved");
+        assert_eq!(rows[0].get::<f64, _>("value"), 1.0);
+        assert_eq!(rows[1].get::<String, _>("label"), "not_for_me");
+        assert_eq!(rows[1].get::<f64, _>("value"), -1.0);
+        for row in &rows {
+            assert_eq!(row.get::<String, _>("kind"), "explicit");
+            assert_eq!(row.get::<String, _>("source"), "migration");
+            assert_eq!(row.get::<String, _>("issue_date"), "2026-08-15");
+        }
+        assert_eq!(rows[0].get::<String, _>("event_at"), "2026-08-15T12:00:00Z");
+
+        let tables: Vec<String> =
+            sqlx::query_scalar("SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name")
+                .fetch_all(&pool)
+                .await
+                .unwrap();
+        assert!(!tables.iter().any(|table| table == "ratings"));
+        assert!(!tables.iter().any(|table| table == "feed_priors"));
+        assert!(!tables.iter().any(|table| table == "scores"));
+        for expected in [
+            "rating_events",
+            "article_embeddings",
+            "interest_embeddings",
+            "article_assessments",
+            "candidate_runs",
+        ] {
+            assert!(tables.iter().any(|table| table == expected), "{expected}");
+        }
     }
 }

src/dedupe.rs

@@ -162,7 +162,7 @@ fn is_enclosure_only(raw_content: &str) -> bool {
         || lower.contains("<video")
         || lower.contains("<embed")
         || lower.contains("<iframe");
-    embeds && crate::extract::word_count(raw_content) < 25
+    embeds && crate::html::word_count(raw_content) < 25
 }
 
 /// Classify which kind of feed an entry arrived through, for the sources list (§3.2, §3.5).
@@ -345,7 +345,7 @@ fn build_article(members: Vec<(Entry, String)>, feed_urls: &FeedUrls) -> Article
     let best = members
         .iter()
         .enumerate()
-        .max_by_key(|(_, (entry, _))| (crate::extract::word_count(&entry.raw_content), -(entry.id)))
+        .max_by_key(|(_, (entry, _))| (crate::html::word_count(&entry.raw_content), -(entry.id)))
         .map(|(i, _)| i)
         .unwrap_or(0);
     let (best_entry, canonical) = &members[best];
@@ -385,7 +385,7 @@ fn build_article(members: Vec<(Entry, String)>, feed_urls: &FeedUrls) -> Article
         .filter_map(|(e, _)| e.author.clone())
         .find(|a| !a.trim().is_empty());
 
-    let word_count = crate::extract::word_count(&best_entry.raw_content);
+    let word_count = crate::html::word_count(&best_entry.raw_content);
 
     Article {
         id: 0,

src/epub/chapters.rs

@@ -0,0 +1,895 @@
+//! Rendering one chapter of XHTML at a time (spec §3.10).
+//!
+//! Each function here turns part of an [`Issue`] into a [`Chapter`]: the front
+//! page, the index, section title pages, articles, discussions, the world
+//! briefing and the colophon. [`super::build::render_all`] puts them in order;
+//! [`super::build::assemble`] zips them.
+
+use askama::Template;
+
+use crate::comments;
+use crate::html::{text_escape, to_xhtml};
+use crate::images;
+use crate::types::{
+    BehindThePaper, Edition, ImageAsset, Issue, NearMiss, Pick, SocialRef, Vote,
+    WORLD_BRIEFING_SECTION,
+};
+use crate::world;
+
+use super::EpubError;
+use super::build::Chapter;
+
+/// Characters of the hex HMAC kept in rating links (§3.9).
+pub const TOKEN_LEN: usize = crate::auth::TOKEN_LEN;
+
+// One implementation, shared with the verifying side in [`crate::server`]:
+// see [`crate::auth`] for the formula and the pinned test vector (§3.9).
+pub use crate::auth::{rating_message, rating_token, rating_url};
+
+#[derive(Template)]
+#[template(path = "front_page.xhtml", escape = "html")]
+struct FrontPage {
+    title: String,
+    display_date: String,
+    issue_number: i64,
+    stats_line: String,
+    body_html: String,
+}
+
+struct IndexEntry {
+    href: String,
+    title: String,
+    source: String,
+    reading_minutes: i64,
+    summary: String,
+    why: Option<String>,
+}
+
+struct IndexSection {
+    name: String,
+    entries: Vec<IndexEntry>,
+}
+
+#[derive(Template)]
+#[template(path = "in_this_issue.xhtml", escape = "html")]
+struct InThisIssue {
+    title: String,
+    stats_line: String,
+    sections: Vec<IndexSection>,
+}
+
+#[derive(Template)]
+#[template(path = "section.xhtml", escape = "html")]
+struct SectionPage {
+    title: String,
+    name: String,
+}
+
+struct RatingLinks {
+    loved_url: String,
+    good_url: String,
+    not_for_me_url: String,
+}
+
+#[derive(Template)]
+#[template(path = "chapter.xhtml", escape = "html")]
+struct ArticleChapter {
+    title: String,
+    article_title: String,
+    byline: Option<String>,
+    meta_line: String,
+    social_line: Option<String>,
+    why: Option<String>,
+    summary: Option<String>,
+    excerpt_only: bool,
+    body_html: String,
+    rating: Option<RatingLinks>,
+    read_online_url: String,
+    discussion_href: Option<String>,
+}
+
+#[derive(Template)]
+#[template(path = "discussion.xhtml", escape = "html")]
+struct DiscussionChapter {
+    title: String,
+    heading: String,
+    body_html: String,
+    article_href: String,
+}
+
+#[derive(Template)]
+#[template(path = "world_briefing.xhtml", escape = "html")]
+struct WorldBriefingChapter {
+    title: String,
+    display_date: String,
+    body_html: String,
+}
+
+#[derive(Template)]
+#[template(path = "behind.xhtml", escape = "html")]
+struct BehindChapter {
+    title: String,
+    summary_line: String,
+    admitted_line: String,
+    learned_line: String,
+    near_misses: Vec<String>,
+    models_line: String,
+}
+
+#[derive(Template)]
+#[template(path = "colophon.xhtml", escape = "html")]
+struct ColophonChapter {
+    title: String,
+    issue_number: i64,
+    display_date: String,
+    generated_at: String,
+    bulk_model: String,
+    editor_model: String,
+    summaries_model: String,
+    provider_costs: Vec<ProviderCostLine>,
+    entries_fetched: i64,
+    feeds_seen: i64,
+    candidates: i64,
+    article_count: i64,
+    section_count: i64,
+    total_words: i64,
+    reading_line: String,
+    cost_usd: String,
+    generator_version: String,
+}
+
+struct ProviderCostLine {
+    provider: String,
+    cost: String,
+}
+
+// ---------------------------------------------------------------------------
+// Chapters (§3.10)
+// ---------------------------------------------------------------------------
+
+/// Prepare article markup for XHTML: rewrite images, sanitize, self-close voids.
+pub fn prepare_body(html: &str, images_: &[ImageAsset]) -> String {
+    // Sanitize first: image rewriting emits our own trusted markup (including the
+    // `image-placeholder` class, which ammonia would otherwise strip).
+    let cleaned = ammonia::clean(html);
+    let rewritten = images::rewrite_img_srcs(&cleaned, images_);
+    to_xhtml(&rewritten)
+}
+
+/// "~2h 15m read" for the colophon (§3.10).
+fn reading_line(minutes: i64) -> String {
+    let (h, m) = (minutes / 60, minutes % 60);
+    if h > 0 {
+        format!("~{h}h {m}m read")
+    } else {
+        format!("~{m}m read")
+    }
+}
+
+/// "▲ 342 on HN · 210 comments" (§3.10).
+pub fn social_line(social: &[SocialRef]) -> Option<String> {
+    let mut parts: Vec<String> = Vec::new();
+    let mut comments = 0i64;
+    for entry in social {
+        if entry.score > 0 {
+            parts.push(format!(
+                "\u{25b2} {} on {}",
+                entry.score,
+                entry.source.display_name()
+            ));
+        }
+        comments += entry.num_comments.max(0);
+    }
+    if comments > 0 {
+        let noun = if comments == 1 { "comment" } else { "comments" };
+        parts.push(format!("{comments} {noun}"));
+    }
+    if parts.is_empty() {
+        None
+    } else {
+        Some(parts.join(" \u{00b7} "))
+    }
+}
+
+fn article_href(pick: &Pick) -> String {
+    format!("{}.xhtml", pick.article.chapter_id())
+}
+
+fn discussion_href(pick: &Pick) -> String {
+    format!("disc-{}.xhtml", pick.article.best_entry_id)
+}
+
+fn section_href(name: &str) -> String {
+    let slug: String = name
+        .chars()
+        .map(|c| {
+            if c.is_ascii_alphanumeric() {
+                c.to_ascii_lowercase()
+            } else {
+                '-'
+            }
+        })
+        .collect();
+    let slug = slug
+        .split('-')
+        .filter(|s| !s.is_empty())
+        .collect::<Vec<_>>()
+        .join("-");
+    format!("sec-{slug}.xhtml")
+}
+
+fn summary_for<'a>(issue: &'a Issue, pick: &'a Pick) -> Option<&'a str> {
+    pick.summary
+        .as_deref()
+        .or_else(|| {
+            issue
+                .editorial
+                .summaries
+                .get(&pick.article.id)
+                .map(|s| s.as_str())
+        })
+        .filter(|s| !s.trim().is_empty())
+}
+
+fn published_display(pick: &Pick) -> Option<String> {
+    pick.article
+        .published_at
+        .map(|ts| ts.to_zoned(jiff::tz::TimeZone::UTC).date().to_string())
+}
+
+/// The Brief (§14.2) under the masthead, plus the issue stats line (§3.10).
+pub fn render_front_page(issue: &Issue) -> Result<Chapter, EpubError> {
+    let body = issue.editorial.front_page_html.trim();
+    let body_html = if body.is_empty() {
+        format!(
+            "<p>{} of reading, chosen overnight.</p>",
+            text_escape(&issue.meta.stats_line())
+        )
+    } else {
+        to_xhtml(&ammonia::clean(body))
+    };
+    let tpl = FrontPage {
+        title: "The Brief".into(),
+        display_date: issue.meta.display_date.clone(),
+        issue_number: issue.meta.issue_number,
+        stats_line: issue.meta.stats_line(),
+        body_html,
+    };
+    Ok(Chapter {
+        id: "front".into(),
+        href: "front.xhtml".into(),
+        title: "The Brief".into(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    })
+}
+
+/// Section names in issue order, with the reserved world section removed (§3.6).
+pub fn section_names(issue: &Issue) -> Vec<String> {
+    let mut names: Vec<String> = issue
+        .lineup
+        .section_order
+        .iter()
+        .filter(|s| s.as_str() != WORLD_BRIEFING_SECTION)
+        .cloned()
+        .collect();
+    for pick in &issue.lineup.picks {
+        if pick.section != WORLD_BRIEFING_SECTION && !names.contains(&pick.section) {
+            names.push(pick.section.clone());
+        }
+    }
+    names.retain(|n| issue.lineup.picks.iter().any(|p| &p.section == n));
+    names
+}
+
+/// "In This Issue": per section, each article's title, source, reading time and
+/// summary, linked to its chapter (§3.10).
+pub fn render_in_this_issue(issue: &Issue) -> Result<Chapter, EpubError> {
+    let mut sections = Vec::new();
+    for name in section_names(issue) {
+        let entries = issue
+            .lineup
+            .section_picks(&name)
+            .into_iter()
+            .map(|pick| IndexEntry {
+                href: article_href(pick),
+                title: pick.article.title.clone(),
+                source: pick.article.feed_title.clone(),
+                reading_minutes: pick.article.reading_minutes(),
+                summary: summary_for(issue, pick).unwrap_or_default().to_string(),
+                why: pick.why.clone(),
+            })
+            .collect();
+        sections.push(IndexSection { name, entries });
+    }
+    if issue.world_briefing.is_some() {
+        sections.push(IndexSection {
+            name: WORLD_BRIEFING_SECTION.to_string(),
+            entries: vec![IndexEntry {
+                href: "world.xhtml".into(),
+                title: "World Briefing".into(),
+                source: "Wikipedia Current Events".into(),
+                reading_minutes: 3,
+                summary: "The day's events, as recorded by the Current Events portal.".into(),
+                why: None,
+            }],
+        });
+    }
+    let tpl = InThisIssue {
+        title: "In This Issue".into(),
+        stats_line: issue.meta.stats_line(),
+        sections,
+    };
+    Ok(Chapter {
+        id: "in-this-issue".into(),
+        href: "in-this-issue.xhtml".into(),
+        title: "In This Issue".into(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    })
+}
+
+/// A section title page: the name only (§14.2 removed the LLM intros).
+pub fn render_section_page(name: &str) -> Result<Chapter, EpubError> {
+    let tpl = SectionPage {
+        title: name.to_string(),
+        name: name.to_string(),
+    };
+    Ok(Chapter {
+        id: format!("sec-{name}"),
+        href: section_href(name),
+        title: name.to_string(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    })
+}
+
+/// One article chapter: header, cleaned body with embedded images, rating footer (§3.10).
+pub fn render_article(
+    issue: &Issue,
+    pick: &Pick,
+    images_: &[ImageAsset],
+    edition: Edition,
+    public_url: &str,
+    hmac_secret: Option<&str>,
+) -> Result<Chapter, EpubError> {
+    let article = &pick.article;
+    let mut meta_parts = vec![article.feed_title.clone()];
+    if let Some(date) = published_display(pick) {
+        meta_parts.push(date);
+    }
+    meta_parts.push(format!(
+        "{} min read \u{00b7} {} words",
+        article.reading_minutes(),
+        article.word_count
+    ));
+
+    // The X4 has no browser, so rating links are pointless there (§7).
+    let rating = match (hmac_secret, edition) {
+        (Some(secret), Edition::Standard) if !secret.is_empty() => Some(RatingLinks {
+            loved_url: rating_url(public_url, secret, issue.meta.date, article.id, Vote::Loved),
+            good_url: rating_url(public_url, secret, issue.meta.date, article.id, Vote::Good),
+            not_for_me_url: rating_url(
+                public_url,
+                secret,
+                issue.meta.date,
+                article.id,
+                Vote::NotForMe,
+            ),
+        }),
+        _ => None,
+    };
+
+    let tpl = ArticleChapter {
+        title: article.title.clone(),
+        article_title: article.title.clone(),
+        byline: article.author.as_ref().map(|a| format!("By {a}")),
+        meta_line: meta_parts.join(" \u{00b7} "),
+        social_line: social_line(&article.social),
+        why: pick.why.clone(),
+        summary: summary_for(issue, pick).map(str::to_string),
+        excerpt_only: article.excerpt_only,
+        body_html: prepare_body(&article.content_html, images_),
+        rating,
+        read_online_url: article.url.clone(),
+        discussion_href: pick.discussion.as_ref().map(|_| discussion_href(pick)),
+    };
+    Ok(Chapter {
+        id: article.chapter_id(),
+        href: article_href(pick),
+        title: article.title.clone(),
+        xhtml: tpl.render()?,
+        toc_level: 2,
+    })
+}
+
+/// The discussion chapter that follows an article when it has comments (§3.7).
+pub fn render_discussion(pick: &Pick) -> Result<Option<Chapter>, EpubError> {
+    let Some(discussion) = &pick.discussion else {
+        return Ok(None);
+    };
+    if discussion.threads.is_empty() {
+        return Ok(None);
+    }
+    let title = comments::chapter_title(&pick.article.title, discussion);
+    let tpl = DiscussionChapter {
+        title: title.clone(),
+        heading: title.clone(),
+        body_html: comments::render_xhtml(discussion, &pick.article.title),
+        article_href: article_href(pick),
+    };
+    Ok(Some(Chapter {
+        id: format!("disc-{}", pick.article.best_entry_id),
+        href: discussion_href(pick),
+        title,
+        xhtml: tpl.render()?,
+        toc_level: 3,
+    }))
+}
+
+/// The Wikipedia Current Events section chapter (§3.8).
+pub fn render_world_briefing(issue: &Issue) -> Result<Option<Chapter>, EpubError> {
+    let Some(briefing) = &issue.world_briefing else {
+        return Ok(None);
+    };
+    // The briefing may cover an earlier day than the issue: the portal page for
+    // the issue's own date is still an empty stub at 05:30 (§3.8). Dateline the
+    // section with the day it actually reports on, not the masthead date.
+    let tpl = WorldBriefingChapter {
+        title: WORLD_BRIEFING_SECTION.to_string(),
+        display_date: crate::pipeline::display_date(briefing.date),
+        body_html: world::render_xhtml(briefing),
+    };
+    Ok(Some(Chapter {
+        id: "world".into(),
+        href: "world.xhtml".into(),
+        title: WORLD_BRIEFING_SECTION.to_string(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    }))
+}
+
+/// "1,465" — thousands separators for the counts in Behind the paper.
+fn thousands(n: i64) -> String {
+    let digits = n.abs().to_string();
+    let mut out = String::with_capacity(digits.len() + digits.len() / 3);
+    for (i, c) in digits.chars().enumerate() {
+        if i > 0 && (digits.len() - i).is_multiple_of(3) {
+            out.push(',');
+        }
+        out.push(c);
+    }
+    if n < 0 { format!("-{out}") } else { out }
+}
+
+/// `Considered 412 articles from 1,465 feeds · 398 eligible · …` (§15.1).
+pub fn behind_summary_line(b: &BehindThePaper) -> String {
+    format!(
+        "Considered {} articles from {} feeds \u{00b7} {} eligible \u{00b7} {} triaged \u{00b7} {} read closely \u{00b7} {} shortlisted \u{00b7} {} selected.",
+        thousands(b.considered),
+        thousands(b.feeds_seen),
+        thousands(b.eligible),
+        thousands(b.triaged),
+        thousands(b.read_closely),
+        thousands(b.shortlisted),
+        thousands(b.selected),
+    )
+}
+
+/// `Admitted via: triage 60 · interests 20 · your ratings 12 · exploration 5 · blend 23.`
+pub fn behind_admitted_line(b: &BehindThePaper) -> String {
+    let count = |name: &str| b.admitted_by.get(name).copied().unwrap_or(0);
+    let mut parts = vec![
+        format!("triage {}", count("triage")),
+        format!("interests {}", count("interest")),
+        format!("your ratings {}", count("knn")),
+        format!("exploration {}", count("exploration")),
+        format!("blend {}", count("blend")),
+    ];
+    if count("auto_include") > 0 {
+        parts.push(format!("always-include {}", count("auto_include")));
+    }
+    format!("Admitted via: {}.", parts.join(" \u{00b7} "))
+}
+
+/// `Learned signals: 14 rated articles with embeddings (neighbour signal at 35%); feed affinity off.`
+pub fn behind_learned_line(b: &BehindThePaper) -> String {
+    let percent = |gate: f64| (gate.clamp(0.0, 1.0) * 100.0).round() as i64;
+    let neighbour = if b.knn_gate > 0.0 {
+        format!("neighbour signal at {}%", percent(b.knn_gate))
+    } else {
+        "neighbour signal off".to_string()
+    };
+    let feed = if b.feed_gate > 0.0 {
+        format!("feed affinity at {}%", percent(b.feed_gate))
+    } else {
+        "feed affinity off".to_string()
+    };
+    format!(
+        "Learned signals: {} rated articles with embeddings ({neighbour}); {feed}.",
+        thousands(b.rated_with_embeddings)
+    )
+}
+
+/// `<title> — <feed> · quality 8.0 · fit 6.5 · shortlisted, not selected`.
+pub fn behind_near_miss_line(miss: &NearMiss) -> String {
+    let mut parts = vec![format!("{} \u{2014} {}", miss.title, miss.feed_title)];
+    if let Some(quality) = miss.quality {
+        parts.push(format!("quality {quality:.1}"));
+    }
+    if let Some(fit) = miss.fit {
+        parts.push(format!("fit {fit:.1}"));
+    }
+    parts.push(match &miss.reason {
+        Some(reason) => format!("{}, {}", miss.stage, reason.replace('_', " ")),
+        None => miss.stage.clone(),
+    });
+    parts.join(" \u{00b7} ")
+}
+
+/// `Models: triage and assessment … · editor and summaries … · embeddings ….
+/// Cost $0.81. Generation 23 min.`
+pub fn behind_models_line(b: &BehindThePaper) -> String {
+    let editorial = if b.models.summaries == b.models.editor {
+        format!("editor and summaries {}", b.models.editor)
+    } else {
+        format!(
+            "editor {} \u{00b7} summaries {}",
+            b.models.editor, b.models.summaries
+        )
+    };
+    let generation = if b.generation_secs < 60 {
+        format!("{} s", b.generation_secs.max(0))
+    } else {
+        format!("{} min", (b.generation_secs as f64 / 60.0).round() as i64)
+    };
+    format!(
+        "Models: triage and assessment {} \u{00b7} {editorial} \u{00b7} embeddings {}. Cost ${:.2}. Generation {generation}.",
+        b.models.bulk, b.embedding_model, b.cost_usd
+    )
+}
+
+/// "Behind the paper": the run's counts, admission mix, learned-signal state,
+/// near misses and models (§15.1). Same text in both editions; no links.
+pub fn render_behind_the_paper(issue: &Issue) -> Result<Chapter, EpubError> {
+    let behind = &issue.behind;
+    let tpl = BehindChapter {
+        title: "Behind the paper".into(),
+        summary_line: behind_summary_line(behind),
+        admitted_line: behind_admitted_line(behind),
+        learned_line: behind_learned_line(behind),
+        near_misses: behind
+            .near_misses
+            .iter()
+            .map(behind_near_miss_line)
+            .collect(),
+        models_line: behind_models_line(behind),
+    };
+    Ok(Chapter {
+        id: "behind".into(),
+        href: "behind.xhtml".into(),
+        title: "Behind the paper".into(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    })
+}
+
+/// Colophon: generation timestamp, models used, token cost, feed counts (§3.10).
+pub fn render_colophon(issue: &Issue) -> Result<Chapter, EpubError> {
+    let colophon = &issue.colophon;
+    let provider_costs = colophon
+        .provider_costs
+        .iter()
+        .map(|(provider, cost)| ProviderCostLine {
+            provider: provider.clone(),
+            cost: format!("${cost:.4}"),
+        })
+        .collect();
+    let tpl = ColophonChapter {
+        title: "Colophon".into(),
+        issue_number: issue.meta.issue_number,
+        display_date: issue.meta.display_date.clone(),
+        generated_at: issue.meta.generated_at.to_string(),
+        bulk_model: colophon.models.bulk.clone(),
+        editor_model: colophon.models.editor.clone(),
+        summaries_model: colophon.models.summaries.clone(),
+        provider_costs,
+        entries_fetched: colophon.entries_fetched,
+        feeds_seen: colophon.feeds_seen,
+        candidates: colophon.candidates,
+        article_count: issue.meta.article_count,
+        section_count: issue.meta.section_count,
+        total_words: issue.meta.total_words,
+        reading_line: reading_line(issue.meta.reading_minutes),
+        cost_usd: format!("${:.4}", colophon.cost_usd),
+        generator_version: if colophon.generator_version.is_empty() {
+            format!("daily-epub {}", env!("CARGO_PKG_VERSION"))
+        } else {
+            colophon.generator_version.clone()
+        },
+    };
+    Ok(Chapter {
+        id: "colophon".into(),
+        href: "colophon.xhtml".into(),
+        title: "Colophon".into(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::epub::fixtures::{self, assert_xml_ok, issue};
+    use crate::types::{SocialRef, Vote};
+    use hmac::{Hmac, KeyInit};
+    use jiff::civil::Date;
+    use sha2::Sha256;
+
+    #[test]
+    fn rating_token_matches_the_spec_vector() {
+        let date: Date = "2026-08-15".parse().unwrap();
+        assert_eq!(
+            rating_message(date, 1234, Vote::Loved),
+            "2026-08-15/1234/loved"
+        );
+        // hex(hmac_sha256("test-secret", "2026-08-15/1234/loved"))[..16]
+        let token = rating_token("test-secret", date, 1234, Vote::Loved);
+        assert_eq!(token.len(), TOKEN_LEN);
+        assert!(token.chars().all(|c| c.is_ascii_hexdigit()));
+
+        // Independently computed reference value.
+        use hmac::Mac;
+        let mut mac = Hmac::<Sha256>::new_from_slice(b"test-secret").unwrap();
+        mac.update(b"2026-08-15/1234/loved");
+        let expected: String = hex::encode(mac.finalize().into_bytes())
+            .chars()
+            .take(16)
+            .collect();
+        assert_eq!(token, expected);
+
+        // Different vote, article and secret all change the token.
+        assert_ne!(
+            token,
+            rating_token("test-secret", date, 1234, Vote::NotForMe)
+        );
+        assert_ne!(token, rating_token("test-secret", date, 1235, Vote::Loved));
+        assert_ne!(token, rating_token("other-secret", date, 1234, Vote::Loved));
+    }
+
+    /// The EPUB signs the links and `server.rs` verifies them: one formula, or no
+    /// rating ever lands. This is the vector `server::tests` pins from its side
+    /// (`VECTOR_SECRET` / `VECTOR_TOKEN_UP`) — change one, change both (§3.9).
+    #[test]
+    fn epub_and_server_share_one_token_vector() {
+        let date: Date = "2026-08-15".parse().unwrap();
+        let token = rating_token("test-secret", date, 42, Vote::Loved);
+        assert_eq!(token, "cece96767d6c5f8a");
+        assert!(crate::server::verify_token(
+            "test-secret",
+            date,
+            42,
+            Vote::Loved,
+            &token
+        ));
+    }
+
+    #[test]
+    fn rating_url_has_the_spec_shape() {
+        let date: Date = "2026-08-15".parse().unwrap();
+        let url = rating_url(
+            "https://daily.hallada.net/",
+            "s3cret",
+            date,
+            99,
+            Vote::NotForMe,
+        );
+        let token = rating_token("s3cret", date, 99, Vote::NotForMe);
+        assert_eq!(
+            url,
+            format!("https://daily.hallada.net/r/2026-08-15/99/down?t={token}")
+        );
+    }
+
+    #[test]
+    fn social_line_matches_the_spec_example() {
+        let refs = vec![SocialRef {
+            article_id: 1,
+            source: crate::types::SocialSource::Hn,
+            item_id: None,
+            score: 342,
+            num_comments: 210,
+            item_url: None,
+            fetched_at: fixtures::timestamp(),
+        }];
+        assert_eq!(
+            social_line(&refs).as_deref(),
+            Some("\u{25b2} 342 on HN \u{00b7} 210 comments")
+        );
+        assert!(social_line(&[]).is_none());
+    }
+
+    #[test]
+    fn hrefs_are_deterministic() {
+        let issue = issue();
+        let pick = &issue.lineup.picks[0];
+        assert_eq!(article_href(pick), "art-1001.xhtml");
+        assert_eq!(discussion_href(pick), "disc-1001.xhtml");
+        assert_eq!(
+            section_href("Tech & Engineering"),
+            "sec-tech-engineering.xhtml"
+        );
+    }
+
+    #[test]
+    fn front_page_renders_stats_and_editorial() {
+        let issue = issue();
+        let chapter = render_front_page(&issue).unwrap();
+        assert_eq!(chapter.href, "front.xhtml");
+        assert!(chapter.xhtml.contains("The Brief"));
+        assert!(chapter.xhtml.contains("2 articles"));
+        assert!(chapter.xhtml.contains("both worth your coffee"));
+        assert!(chapter.xhtml.contains("No. 42"));
+        assert_xml_ok(&chapter.xhtml);
+    }
+
+    #[test]
+    fn in_this_issue_links_every_pick() {
+        let issue = issue();
+        let chapter = render_in_this_issue(&issue).unwrap();
+        assert!(chapter.xhtml.contains("href=\"art-1001.xhtml\""));
+        assert!(chapter.xhtml.contains("href=\"art-1002.xhtml\""));
+        assert!(chapter.xhtml.contains("Example Feed"));
+        assert!(chapter.xhtml.contains("6 min read"));
+        assert!(chapter.xhtml.contains("What it argues"));
+        assert!(chapter.xhtml.contains("A short abstract"));
+        // Titles are escaped (askama emits numeric references), never injected raw.
+        assert!(chapter.xhtml.contains("A Niche Delight &#38; Other Tales"));
+        assert_xml_ok(&chapter.xhtml);
+    }
+
+    #[test]
+    fn article_chapter_has_header_body_and_footer() {
+        let issue = issue();
+        let pick = &issue.lineup.picks[0];
+        let chapter = render_article(
+            &issue,
+            pick,
+            &[],
+            Edition::Standard,
+            "https://daily.hallada.net",
+            Some("s3cret"),
+        )
+        .unwrap();
+        assert_eq!(chapter.id, "art-1001");
+        assert_eq!(chapter.toc_level, 2);
+        assert!(chapter.xhtml.contains("By A. Writer"));
+        assert!(chapter.xhtml.contains("Example Feed"));
+        assert!(chapter.xhtml.contains("6 min read"));
+        assert!(chapter.xhtml.contains("\u{25b2} 342 on HN"));
+        assert!(chapter.xhtml.contains("/r/2026-08-15/1/loved?t="));
+        assert!(chapter.xhtml.contains("/r/2026-08-15/1/good?t="));
+        assert!(chapter.xhtml.contains("/r/2026-08-15/1/down?t="));
+        assert!(chapter.xhtml.contains("Read online"));
+        assert!(chapter.xhtml.contains("href=\"disc-1001.xhtml\""));
+        // The un-downloaded image degrades to a placeholder.
+        assert!(
+            chapter
+                .xhtml
+                .contains("[image: A chart of the daily figures]")
+        );
+        assert_xml_ok(&chapter.xhtml);
+    }
+
+    #[test]
+    fn x4_articles_omit_rating_links() {
+        let issue = issue();
+        let chapter = render_article(
+            &issue,
+            &issue.lineup.picks[0],
+            &[],
+            Edition::X4,
+            "https://daily.hallada.net",
+            Some("s3cret"),
+        )
+        .unwrap();
+        assert!(!chapter.xhtml.contains("/r/2026-08-15/"));
+        assert!(chapter.xhtml.contains("Read online"));
+    }
+
+    #[test]
+    fn discussion_chapter_nests_under_its_article() {
+        let issue = issue();
+        let chapter = render_discussion(&issue.lineup.picks[0]).unwrap().unwrap();
+        assert_eq!(chapter.id, "disc-1001");
+        assert_eq!(chapter.toc_level, 3);
+        assert!(
+            chapter
+                .title
+                .starts_with("\u{1f4ac} Discussion: The Lead Story")
+        );
+        assert!(chapter.xhtml.contains("alice"));
+        assert!(chapter.xhtml.contains("blockquote class=\"comment\""));
+        assert!(chapter.xhtml.contains("href=\"art-1001.xhtml\""));
+        assert_xml_ok(&chapter.xhtml);
+        assert!(render_discussion(&issue.lineup.picks[1]).unwrap().is_none());
+    }
+
+    #[test]
+    fn behind_the_paper_lines_follow_the_plan_shape() {
+        let issue = issue();
+        let behind = &issue.behind;
+        assert_eq!(
+            behind_summary_line(behind),
+            "Considered 412 articles from 1,465 feeds \u{00b7} 398 eligible \u{00b7} 398 triaged \u{00b7} 120 read closely \u{00b7} 60 shortlisted \u{00b7} 2 selected."
+        );
+        assert_eq!(
+            behind_admitted_line(behind),
+            "Admitted via: triage 60 \u{00b7} interests 20 \u{00b7} your ratings 12 \u{00b7} exploration 5 \u{00b7} blend 23."
+        );
+        assert_eq!(
+            behind_learned_line(behind),
+            "Learned signals: 14 rated articles with embeddings (neighbour signal at 35%); feed affinity off."
+        );
+        assert_eq!(
+            behind_near_miss_line(&behind.near_misses[0]),
+            "The One That Got Away \u{2014} Example Feed \u{00b7} quality 8.0 \u{00b7} fit 6.5 \u{00b7} shortlisted, not selected"
+        );
+        assert_eq!(
+            behind_near_miss_line(&behind.near_misses[1]),
+            "Never Read Closely \u{2014} Other Feed \u{00b7} triaged, not admitted"
+        );
+        assert_eq!(
+            behind_models_line(behind),
+            "Models: triage and assessment deepseek-v4-flash \u{00b7} editor and summaries claude-opus-5 \u{00b7} embeddings voyage-4-lite. Cost $0.81. Generation 23 min."
+        );
+        let chapter = render_behind_the_paper(&issue).unwrap();
+        assert_eq!(chapter.id, "behind");
+        assert_eq!(chapter.href, "behind.xhtml");
+        assert!(chapter.xhtml.contains("Behind the paper"));
+        assert!(chapter.xhtml.contains("1,465 feeds"));
+        assert!(chapter.xhtml.contains("The One That Got Away"));
+        assert!(!chapter.xhtml.contains("<a "), "no links in the chapter");
+        assert_xml_ok(&chapter.xhtml);
+
+        // Auto-includes are named only when there were any; a short run is in
+        // seconds; a differing summaries model is listed separately.
+        let mut short = behind.clone();
+        short.admitted_by.insert("auto_include".into(), 2);
+        short.generation_secs = 48;
+        short.feed_gate = 0.6;
+        short.models.summaries = "deepseek-v4-flash".into();
+        short.near_misses.clear();
+        assert!(behind_admitted_line(&short).ends_with("blend 23 \u{00b7} always-include 2."));
+        assert!(behind_learned_line(&short).ends_with("feed affinity at 60%."));
+        assert!(
+            behind_models_line(&short)
+                .contains("editor claude-opus-5 \u{00b7} summaries deepseek-v4-flash")
+        );
+        assert!(behind_models_line(&short).ends_with("Generation 48 s."));
+        let mut issue = issue;
+        issue.behind = short;
+        let chapter = render_behind_the_paper(&issue).unwrap();
+        assert!(chapter.xhtml.contains("None recorded"));
+        assert_xml_ok(&chapter.xhtml);
+        assert_eq!(thousands(0), "0");
+        assert_eq!(thousands(999), "999");
+        assert_eq!(thousands(1_000), "1,000");
+        assert_eq!(thousands(1_234_567), "1,234,567");
+    }
+
+    #[test]
+    fn world_and_colophon_chapters_render() {
+        let issue = issue();
+        let world = render_world_briefing(&issue).unwrap().unwrap();
+        assert!(world.xhtml.contains("Something happened somewhere"));
+        assert!(world.xhtml.contains("CC BY-SA"));
+        assert_xml_ok(&world.xhtml);
+
+        let colophon = render_colophon(&issue).unwrap();
+        assert!(colophon.xhtml.contains("deepseek-v4-flash"));
+        assert!(colophon.xhtml.contains("431"));
+        assert!(colophon.xhtml.contains("$0.0731"));
+        assert_xml_ok(&colophon.xhtml);
+    }
+}

src/epub/cover.rs

@@ -0,0 +1,423 @@
+//! The generated cover image and cover page (spec §3.10).
+//!
+//! The cover is drawn as an SVG from a template, rasterized with resvg and
+//! encoded per edition. If font resolution fails the run still gets a cover —
+//! a text-free variant that keeps the editions apart in a library thumbnail
+//! grid (notes §3: nothing here fails a build).
+
+use askama::Template;
+
+use crate::types::{Edition, Issue};
+
+use super::EpubError;
+use super::build::Chapter;
+use super::x4;
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct CoverAsset {
+    pub bytes: Vec<u8>,
+    pub filename: &'static str,
+    pub mime: &'static str,
+}
+
+pub fn cover_href(edition: Edition) -> &'static str {
+    match edition {
+        Edition::Standard => "cover.png",
+        Edition::X4 => "cover.jpg",
+    }
+}
+
+#[derive(Template)]
+#[template(path = "cover.svg", escape = "html")]
+struct CoverSvg {
+    width: i64,
+    height: i64,
+    margin: i64,
+    inner_width: i64,
+    inner_height: i64,
+    border: i64,
+    hairline: i64,
+    center_x: i64,
+    masthead_y: i64,
+    masthead_size: i64,
+    rule_x1: i64,
+    rule_x2: i64,
+    rule_y: i64,
+    rule2_y: i64,
+    weekday: String,
+    weekday_y: i64,
+    weekday_size: i64,
+    long_date: String,
+    date_y: i64,
+    date_size: i64,
+    issue_number: i64,
+    issue_y: i64,
+    issue_size: i64,
+    stats_line: String,
+    stats_y: i64,
+    stats_size: i64,
+    /// Empty for the standard edition — see [`cover_badge`].
+    edition_tag: String,
+    badge_x: i64,
+    badge_y: i64,
+    badge_width: i64,
+    badge_height: i64,
+    badge_radius: i64,
+    badge_text_y: i64,
+    badge_size: i64,
+    footer: String,
+    footer_y: i64,
+    footer_size: i64,
+}
+
+#[derive(Template)]
+#[template(path = "cover_page.xhtml", escape = "html")]
+struct CoverPage {
+    title: String,
+    alt: String,
+    cover_href: &'static str,
+}
+
+// ---------------------------------------------------------------------------
+// Cover (§3.10)
+// ---------------------------------------------------------------------------
+
+/// Cover pixel size per edition: 1200×1600 standard, 480×800 X4 (§3.10).
+pub fn cover_size(edition: Edition) -> (u32, u32) {
+    match edition {
+        Edition::Standard => (1200, 1600),
+        Edition::X4 => x4::X4_SCREEN,
+    }
+}
+
+/// Split "Friday, August 15, 2026" into ("Friday", "August 15, 2026").
+fn split_display_date(display: &str) -> (String, String) {
+    match display.split_once(", ") {
+        Some((weekday, rest)) => (weekday.to_string(), rest.to_string()),
+        None => (String::new(), display.to_string()),
+    }
+}
+
+/// Reversed badge printed on the cover so the two editions are told apart at
+/// thumbnail size, where the title is unreadable (§3.10).
+///
+/// Empty for the standard edition: an unmarked cover is the default, and the
+/// presence of the slab is itself the signal.
+pub fn cover_badge(edition: Edition) -> &'static str {
+    match edition {
+        Edition::Standard => "",
+        Edition::X4 => "X4 EDITION",
+    }
+}
+
+fn cover_svg(
+    issue: &Issue,
+    edition: Edition,
+    width: u32,
+    height: u32,
+) -> Result<String, EpubError> {
+    let w = i64::from(width);
+    let h = i64::from(height);
+    let margin = w / 15;
+    let (weekday, long_date) = split_display_date(&issue.meta.display_date);
+    // A solid bar between the stats line and the footer: at 100px wide in a
+    // library grid the black slab is the only thing still legible.
+    let badge_height = h / 16;
+    let badge_width = w * 2 / 5;
+    let tpl = CoverSvg {
+        width: w,
+        height: h,
+        margin,
+        inner_width: w - 2 * margin,
+        inner_height: h - 2 * margin,
+        border: (w / 300).max(2),
+        hairline: (w / 600).max(1),
+        center_x: w / 2,
+        masthead_y: h * 30 / 100,
+        masthead_size: w / 9,
+        rule_x1: margin + w / 12,
+        rule_x2: w - margin - w / 12,
+        rule_y: h * 34 / 100,
+        rule2_y: h * 53 / 100,
+        weekday,
+        weekday_y: h * 42 / 100,
+        weekday_size: w / 28,
+        long_date,
+        date_y: h * 47 / 100,
+        date_size: w / 22,
+        issue_number: issue.meta.issue_number,
+        issue_y: h * 60 / 100,
+        issue_size: w / 18,
+        stats_line: issue.meta.stats_line(),
+        stats_y: h * 66 / 100,
+        stats_size: w / 32,
+        edition_tag: cover_badge(edition).to_string(),
+        badge_x: (w - badge_width) / 2,
+        badge_y: h * 73 / 100,
+        badge_width,
+        badge_height,
+        badge_radius: badge_height / 2,
+        badge_text_y: h * 73 / 100 + badge_height * 7 / 10,
+        badge_size: badge_height * 11 / 20,
+        footer: "Assembled overnight \u{00b7} read offline".to_string(),
+        footer_y: h - margin - h / 25,
+        footer_size: w / 40,
+    };
+    Ok(tpl.render()?)
+}
+
+/// Render the standard cover as RGB PNG and the X4 cover as baseline RGB JPEG.
+pub fn render_cover(issue: &Issue, edition: Edition) -> Result<CoverAsset, EpubError> {
+    let (width, height) = cover_size(edition);
+    let svg = cover_svg(issue, edition, width, height)?;
+    let pixmap = match rasterize(&svg, width, height) {
+        Some(pixmap) => pixmap,
+        None => {
+            tracing::warn!("no usable system fonts: falling back to a geometric cover");
+            draw_fallback_cover(width, height, edition)
+                .ok_or_else(|| EpubError::Build("could not allocate the cover".into()))?
+        }
+    };
+    Ok(CoverAsset {
+        bytes: encode_cover(pixmap, edition)?,
+        filename: cover_href(edition),
+        mime: if edition == Edition::X4 {
+            "image/jpeg"
+        } else {
+            "image/png"
+        },
+    })
+}
+
+fn rasterize(svg: &str, width: u32, height: u32) -> Option<tiny_skia::Pixmap> {
+    let mut options = resvg::usvg::Options::default();
+    options.fontdb_mut().load_system_fonts();
+    if options.fontdb.is_empty() {
+        return None;
+    }
+    let tree = resvg::usvg::Tree::from_str(svg, &options)
+        .map_err(|e| tracing::warn!("cover svg did not parse: {e}"))
+        .ok()?;
+    let mut pixmap = tiny_skia::Pixmap::new(width, height)?;
+    pixmap.fill(tiny_skia::Color::WHITE);
+    let size = tree.size();
+    let transform = tiny_skia::Transform::from_scale(
+        width as f32 / size.width(),
+        height as f32 / size.height(),
+    );
+    resvg::render(&tree, transform, &mut pixmap.as_mut());
+    Some(pixmap)
+}
+
+/// Text-free cover used when font resolution fails — the build never fails (§3.10).
+///
+/// The badge cannot carry its lettering here, but the slab itself still keeps
+/// the editions apart in a thumbnail grid.
+fn draw_fallback_cover(width: u32, height: u32, edition: Edition) -> Option<tiny_skia::Pixmap> {
+    let mut pixmap = tiny_skia::Pixmap::new(width, height)?;
+    pixmap.fill(tiny_skia::Color::WHITE);
+    let mut paint = tiny_skia::Paint::default();
+    paint.set_color(tiny_skia::Color::BLACK);
+    paint.anti_alias = true;
+
+    let w = width as f32;
+    let h = height as f32;
+    let margin = w / 15.0;
+    let rect = |x: f32, y: f32, rw: f32, rh: f32, pixmap: &mut tiny_skia::Pixmap| {
+        if let Some(r) = tiny_skia::Rect::from_xywh(x, y, rw, rh) {
+            let path = tiny_skia::PathBuilder::from_rect(r);
+            pixmap.fill_path(
+                &path,
+                &paint,
+                tiny_skia::FillRule::Winding,
+                tiny_skia::Transform::identity(),
+                None,
+            );
+        }
+    };
+    // Frame.
+    let border = (w / 300.0).max(2.0);
+    rect(margin, margin, w - 2.0 * margin, border, &mut pixmap);
+    rect(
+        margin,
+        h - margin - border,
+        w - 2.0 * margin,
+        border,
+        &mut pixmap,
+    );
+    rect(margin, margin, border, h - 2.0 * margin, &mut pixmap);
+    rect(
+        w - margin - border,
+        margin,
+        border,
+        h - 2.0 * margin,
+        &mut pixmap,
+    );
+    // Masthead slab plus body rules — a newspaper silhouette.
+    rect(
+        margin * 2.0,
+        h * 0.26,
+        w - 4.0 * margin,
+        h * 0.035,
+        &mut pixmap,
+    );
+    for i in 0..8 {
+        let y = h * 0.42 + (i as f32) * h * 0.045;
+        let inset = if i % 3 == 2 {
+            margin * 4.0
+        } else {
+            margin * 2.0
+        };
+        rect(inset, y, w - 2.0 * inset, border, &mut pixmap);
+    }
+    if !cover_badge(edition).is_empty() {
+        let badge_height = h / 16.0;
+        let badge_width = w * 2.0 / 5.0;
+        rect(
+            (w - badge_width) / 2.0,
+            h * 0.73,
+            badge_width,
+            badge_height,
+            &mut pixmap,
+        );
+    }
+    Some(pixmap)
+}
+
+fn encode_cover(pixmap: tiny_skia::Pixmap, edition: Edition) -> Result<Vec<u8>, EpubError> {
+    let (w, h) = (pixmap.width(), pixmap.height());
+    let rgba = image::RgbaImage::from_raw(w, h, pixmap.take_demultiplied())
+        .ok_or_else(|| EpubError::Build("cover pixel buffer had the wrong size".into()))?;
+    let rgb = image::DynamicImage::ImageRgba8(rgba).to_rgb8();
+    let mut bytes = Vec::new();
+    match edition {
+        Edition::Standard => image::DynamicImage::ImageRgb8(rgb).write_to(
+            &mut std::io::Cursor::new(&mut bytes),
+            image::ImageFormat::Png,
+        ),
+        Edition::X4 => image::codecs::jpeg::JpegEncoder::new_with_quality(&mut bytes, 92)
+            .encode_image(&image::DynamicImage::ImageRgb8(rgb)),
+    }
+    .map_err(|e| EpubError::Build(format!("cover encoding failed: {e}")))?;
+    Ok(bytes)
+}
+
+pub fn render_cover_page(issue: &Issue, edition: Edition) -> Result<Chapter, EpubError> {
+    let tpl = CoverPage {
+        title: issue.meta.title_for(edition),
+        alt: format!(
+            "The Daily EPUB, {} \u{2014} No. {}",
+            issue.meta.display_date, issue.meta.issue_number
+        ),
+        cover_href: cover_href(edition),
+    };
+    Ok(Chapter {
+        id: "cover".into(),
+        href: "cover.xhtml".into(),
+        title: "Cover".into(),
+        xhtml: tpl.render()?,
+        toc_level: 1,
+    })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::epub::fixtures::{self, issue};
+
+    #[test]
+    fn covers_rasterize_for_both_editions() {
+        let issue = issue();
+        for edition in [Edition::Standard, Edition::X4] {
+            let cover = render_cover(&issue, edition).expect("cover");
+            let decoded = image::load_from_memory(&cover.bytes).expect("cover is a valid image");
+            assert_eq!(
+                (decoded.width(), decoded.height()),
+                cover_size(edition),
+                "cover size for {edition:?}"
+            );
+            assert_eq!(decoded.color(), image::ColorType::Rgb8);
+            if edition == Edition::X4 {
+                assert_eq!(cover.filename, "cover.jpg");
+                assert_eq!(cover.mime, "image/jpeg");
+                assert!(
+                    cover.bytes.windows(2).any(|marker| marker == [0xff, 0xc0]),
+                    "baseline SOF0 missing"
+                );
+                assert!(
+                    !cover.bytes.windows(2).any(|marker| marker == [0xff, 0xc2]),
+                    "progressive SOF2 present"
+                );
+            } else {
+                assert_eq!(cover.filename, "cover.png");
+                assert_eq!(cover.mime, "image/png");
+            }
+        }
+    }
+
+    #[test]
+    fn fallback_cover_is_drawn_without_fonts() {
+        let png = encode_cover(
+            draw_fallback_cover(480, 800, Edition::X4).unwrap(),
+            Edition::X4,
+        )
+        .unwrap();
+        let decoded = image::load_from_memory(&png).unwrap();
+        assert_eq!((decoded.width(), decoded.height()), (480, 800));
+        // Some ink actually landed on the page.
+        let gray = decoded.to_luma8();
+        assert!(gray.pixels().any(|p| p[0] < 32));
+        assert!(gray.pixels().any(|p| p[0] > 224));
+
+        // The badge slab is the only per-edition mark the text-free fallback can
+        // draw, so the two editions still differ without any fonts installed.
+        let standard = draw_fallback_cover(480, 800, Edition::Standard).unwrap();
+        let x4 = draw_fallback_cover(480, 800, Edition::X4).unwrap();
+        assert_ne!(standard.data(), x4.data());
+        assert!(badge_ink(&x4) > badge_ink(&standard));
+    }
+
+    /// Dark pixels inside the badge rectangle.
+    fn badge_ink(pixmap: &tiny_skia::Pixmap) -> usize {
+        let (w, h) = (pixmap.width() as usize, pixmap.height() as usize);
+        let (x0, x1) = (w * 3 / 10, w * 7 / 10);
+        let (y0, y1) = (h * 74 / 100, h * 78 / 100);
+        let mut dark = 0;
+        for y in y0..y1 {
+            for x in x0..x1 {
+                if pixmap.data()[(y * w + x) * 4] < 32 {
+                    dark += 1;
+                }
+            }
+        }
+        dark
+    }
+
+    /// At thumbnail size the title is unreadable, so the cover itself has to
+    /// say which edition it is (§3.10).
+    #[test]
+    fn only_the_x4_cover_carries_the_edition_badge() {
+        let issue = fixtures::issue();
+        let x4 = cover_svg(&issue, Edition::X4, 480, 800).unwrap();
+        assert!(x4.contains(">X4 EDITION<"), "{x4}");
+        // White appears twice: the page ground, and the badge text reversed out
+        // of the slab.
+        assert_eq!(x4.matches("fill=\"#ffffff\"").count(), 2, "{x4}");
+
+        let standard = cover_svg(&issue, Edition::Standard, 1200, 1600).unwrap();
+        assert!(!standard.contains("X4"));
+        assert_eq!(standard.matches("fill=\"#ffffff\"").count(), 1);
+
+        // The badge sits between the stats line and the footer, inside the frame.
+        let cover = render_cover(&issue, Edition::X4).unwrap();
+        let gray = image::load_from_memory(&cover.bytes).unwrap().to_luma8();
+        let dark_in_badge = (584..634)
+            .flat_map(|y| (144..336).map(move |x| (x, y)))
+            .filter(|&(x, y)| gray.get_pixel(x, y)[0] < 32)
+            .count();
+        assert!(
+            dark_in_badge > 4000,
+            "badge slab is missing: {dark_in_badge}"
+        );
+    }
+}

src/epub/fixtures.rs

@@ -0,0 +1,236 @@
+//! A synthetic, fully offline [`crate::types::Issue`] for tests.
+//!
+//! Lives here rather than inside a `#[cfg(test)]` block because the integration
+//! tests in `tests/` can only see the public API.
+
+use std::collections::BTreeMap;
+
+use jiff::Timestamp;
+
+use crate::types::*;
+
+pub fn timestamp() -> Timestamp {
+    "2026-08-15T05:30:00Z".parse().expect("fixed timestamp")
+}
+
+pub fn article(id: ArticleId, entry_id: EntryId, title: &str) -> Article {
+    Article {
+        id,
+        canonical_url: format!("https://example.com/{entry_id}"),
+        title: title.to_string(),
+        best_entry_id: entry_id,
+        content_html: format!(
+            "<p>Body of <em>{title}</em> with an image.</p><img src=\"https://img.example/{entry_id}.png\" alt=\"A chart of the daily figures\"><p>More words &amp; things.</p>"
+        ),
+        word_count: 1200,
+        excerpt_only: false,
+        image_count: 1,
+        sources: vec![SourceRef {
+            entry_id,
+            feed_id: 7,
+            feed_title: "Example Feed".into(),
+            category: Some("Tech".into()),
+            kind: SourceKind::Feed,
+        }],
+        first_seen: timestamp(),
+        url: format!("https://example.com/{entry_id}"),
+        author: Some("A. Writer".into()),
+        feed_id: 7,
+        feed_title: "Example Feed".into(),
+        category: Some("Tech".into()),
+        published_at: Some(timestamp()),
+        comments_url: None,
+        image_urls: vec![format!("https://img.example/{entry_id}.png")],
+        social: vec![SocialRef {
+            article_id: id,
+            source: SocialSource::Hn,
+            item_id: Some("40100000".into()),
+            score: 342,
+            num_comments: 210,
+            item_url: Some("https://news.ycombinator.com/item?id=40100000".into()),
+            fetched_at: timestamp(),
+        }],
+        extract_method: ExtractMethod::Readability,
+    }
+}
+
+pub fn discussion(article_id: ArticleId, entry_id: EntryId) -> Discussion {
+    Discussion {
+        article_id,
+        chapter_id: format!("disc-{entry_id}"),
+        threads: vec![CommentThread {
+            source: SocialSource::Hn,
+            item_url: "https://news.ycombinator.com/item?id=40100000".into(),
+            total_comments: 210,
+            comments: vec![Comment {
+                author: "alice".into(),
+                points: Some(61),
+                text_html: "<p>The write path is the interesting part.</p>".into(),
+                depth: 0,
+                children: vec![Comment {
+                    author: "bob".into(),
+                    points: Some(24),
+                    text_html: "<p>Agreed.</p>".into(),
+                    depth: 1,
+                    children: vec![],
+                }],
+            }],
+        }],
+    }
+}
+
+/// A synthetic two-article issue, one of them carrying a discussion.
+pub fn issue() -> Issue {
+    let lead = Pick {
+        article: article(1, 1001, "The Lead Story"),
+        section: "Top Stories".into(),
+        position: 0,
+        is_lead: true,
+        why: Some("The systems story with enough operational detail to matter".into()),
+        summary: Some("What it argues, and why it is worth the time.".into()),
+        llm: None,
+        discussion: Some(discussion(1, 1001)),
+    };
+    let second = Pick {
+        article: article(2, 1002, "A Niche Delight & Other Tales"),
+        section: "Niche Corner".into(),
+        position: 0,
+        is_lead: false,
+        why: Some("A small-scene delight outside the usual technical orbit".into()),
+        summary: None,
+        llm: None,
+        discussion: None,
+    };
+    let mut summaries = BTreeMap::new();
+    summaries.insert(2, "A short abstract for the second piece.".to_string());
+
+    Issue {
+        meta: IssueMeta {
+            date: "2026-08-15".parse().expect("fixed date"),
+            issue_number: 42,
+            generated_at: timestamp(),
+            display_date: "Friday, August 15, 2026".into(),
+            article_count: 2,
+            section_count: 2,
+            total_words: 2400,
+            reading_minutes: 11,
+        },
+        lineup: Lineup {
+            date: "2026-08-15".parse().expect("fixed date"),
+            picks: vec![lead, second],
+            section_order: vec!["Top Stories".into(), "Niche Corner".into()],
+        },
+        editorial: Editorial {
+            front_page_html: "<p>Two stories today, both worth your coffee.</p>".into(),
+            summaries,
+        },
+        world_briefing: Some(WorldBriefing {
+            date: "2026-08-15".parse().expect("fixed date"),
+            source_url: "https://en.wikipedia.org/wiki/Portal:Current_events/2026_August_15".into(),
+            overview: Some("A concise view of the day.".into()),
+            sections: vec![WorldBriefingSection {
+                title: "Top Stories".into(),
+                events: vec![WorldEvent {
+                    id: "s1-e1".into(),
+                    source_text: "Something happened somewhere.".into(),
+                    links: vec![],
+                    children: vec![],
+                    summary: Some("The event in context.".into()),
+                }],
+            }],
+        }),
+        colophon: Colophon {
+            provider_costs: BTreeMap::from([
+                ("deepseek".into(), 0.0231),
+                ("anthropic".into(), 0.05),
+            ]),
+            models: Models {
+                bulk: "deepseek-v4-flash".into(),
+                editor: "claude-opus-5".into(),
+                summaries: "claude-opus-5".into(),
+            },
+            entries_fetched: 431,
+            feeds_seen: 92,
+            candidates: 120,
+            cost_usd: 0.0731,
+            generator_version: "daily-epub 0.1.0".into(),
+        },
+        behind: BehindThePaper {
+            considered: 412,
+            feeds_seen: 1465,
+            eligible: 398,
+            triaged: 398,
+            read_closely: 120,
+            shortlisted: 60,
+            selected: 2,
+            admitted_by: BTreeMap::from([
+                ("triage".to_string(), 60),
+                ("interest".to_string(), 20),
+                ("knn".to_string(), 12),
+                ("exploration".to_string(), 5),
+                ("blend".to_string(), 23),
+            ]),
+            rated_with_embeddings: 14,
+            knn_gate: 0.35,
+            feed_gate: 0.0,
+            near_misses: vec![
+                NearMiss {
+                    article_id: 3,
+                    title: "The One That Got Away".into(),
+                    feed_title: "Example Feed".into(),
+                    quality: Some(8.0),
+                    fit: Some(6.5),
+                    stage: "shortlisted".into(),
+                    reason: Some("not_selected".into()),
+                },
+                NearMiss {
+                    article_id: 4,
+                    title: "Never Read Closely".into(),
+                    feed_title: "Other Feed".into(),
+                    quality: None,
+                    fit: None,
+                    stage: "triaged".into(),
+                    reason: Some("not_admitted".into()),
+                },
+            ],
+            models: Models {
+                bulk: "deepseek-v4-flash".into(),
+                editor: "claude-opus-5".into(),
+                summaries: "claude-opus-5".into(),
+            },
+            embedding_model: "voyage-4-lite".into(),
+            cost_usd: 0.81,
+            generation_secs: 23 * 60 + 12,
+        },
+    }
+}
+
+/// A crude XHTML well-formedness check for rendered chapters.
+///
+/// The real proof is `epubcheck`, which cannot run in a unit test; this catches
+/// the mistakes that actually happen — an unclosed void element or a raw
+/// `&nbsp;`, both of which make an EPUB3 content document unparseable.
+pub fn assert_xml_ok(xhtml: &str) {
+    // A crude well-formedness check: the document parses as XML only if every
+    // tag is closed, so compare open/close counts for the elements we emit.
+    assert!(xhtml.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>"));
+    assert!(xhtml.contains("xmlns=\"http://www.w3.org/1999/xhtml\""));
+    assert!(xhtml.trim_end().ends_with("</html>"));
+    for tag in ["html", "head", "body", "div", "p"] {
+        let opens = xhtml.matches(&format!("<{tag}")).count();
+        let closes = xhtml.matches(&format!("</{tag}>")).count();
+        assert_eq!(opens, closes, "unbalanced <{tag}> in\n{xhtml}");
+    }
+    assert!(!xhtml.contains("&nbsp;"));
+    for void in ["<br>", "<hr>", "<img "] {
+        if void == "<img " {
+            for (i, _) in xhtml.match_indices("<img ") {
+                let tail = &xhtml[i..];
+                let end = tail.find('>').unwrap_or(0);
+                assert!(tail[..end].ends_with('/'), "unclosed <img> in\n{xhtml}");
+            }
+        } else {
+            assert!(!xhtml.contains(void), "unclosed {void} in\n{xhtml}");
+        }
+    }
+}

src/epub/images.rs

@@ -1,715 +0,0 @@
-//! Image download and re-encoding (spec §3.10 "Images").
-//!
-//! Failed downloads degrade to a `[image: alt text]` placeholder paragraph — the
-//! run never fails because of an image (notes §3).
-
-use std::collections::HashMap;
-use std::io::Cursor;
-use std::time::Duration;
-
-use futures::StreamExt;
-use image::{DynamicImage, GenericImageView, ImageFormat};
-
-use crate::types::{Edition, ImageAsset, Pick};
-
-/// Per-image download timeout (§3.10).
-pub const DOWNLOAD_TIMEOUT_SECS: u64 = 10;
-/// Per-image size cap (§3.10).
-pub const MAX_IMAGE_BYTES: usize = 5 * 1024 * 1024;
-/// Concurrent downloads (§3.10).
-pub const CONCURRENCY: usize = 8;
-/// Whole-issue asset budget (§3.10).
-pub const ISSUE_ASSET_BUDGET_BYTES: usize = 25 * 1024 * 1024;
-/// Images smaller than this in either dimension are decorative — skipped (§3.10).
-pub const MIN_DIMENSION_PX: u32 = 24;
-/// Images referenced per article are already capped at 12 by extraction (§3.3).
-pub const MAX_IMAGES_PER_ARTICLE: usize = 12;
-
-/// HTML void elements: XHTML requires them self-closed (§3.10 "valid XHTML").
-pub const VOID_ELEMENTS: &[&str] = &[
-    "area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source",
-    "track", "wbr",
-];
-
-/// Per-edition re-encoding parameters (§3.10).
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub struct ImageProfile {
-    pub max_width: u32,
-    pub max_height: u32,
-    pub jpeg_quality: u8,
-    pub grayscale: bool,
-}
-
-impl ImageProfile {
-    /// Standard edition: max width 1200px, JPEG q80, color (§3.10).
-    pub const STANDARD: ImageProfile = ImageProfile {
-        max_width: 1200,
-        max_height: 4000,
-        jpeg_quality: 80,
-        grayscale: false,
-    };
-
-    /// X4 edition: grayscale Luma8, fit within 480×800, JPEG q70 (§3.10).
-    pub const X4: ImageProfile = ImageProfile {
-        max_width: 480,
-        max_height: 800,
-        jpeg_quality: 70,
-        grayscale: true,
-    };
-
-    pub fn for_edition(edition: Edition) -> Self {
-        match edition {
-            Edition::Standard => Self::STANDARD,
-            Edition::X4 => Self::X4,
-        }
-    }
-}
-
-/// One `<img>` found in article markup, with the caption of its `<figure>` if any.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct ImgRef {
-    pub src: String,
-    pub alt: String,
-    pub caption: Option<String>,
-}
-
-/// Collect `<img>` references (src, alt, enclosing figcaption) from article markup.
-pub fn extract_img_refs(html: &str) -> Vec<ImgRef> {
-    let doc = scraper::Html::parse_fragment(html);
-    let Ok(img_sel) = scraper::Selector::parse("img") else {
-        return Vec::new();
-    };
-    let cap_sel = scraper::Selector::parse("figcaption").ok();
-
-    let mut out = Vec::new();
-    let mut seen: Vec<String> = Vec::new();
-    for el in doc.select(&img_sel) {
-        let Some(src) = el.value().attr("src") else {
-            continue;
-        };
-        let src = src.trim();
-        if src.is_empty() || src.starts_with("data:") {
-            continue;
-        }
-        if seen.iter().any(|s| s == src) {
-            continue;
-        }
-        seen.push(src.to_string());
-        let alt = el
-            .value()
-            .attr("alt")
-            .unwrap_or_default()
-            .trim()
-            .to_string();
-        // Walk up to an enclosing <figure> and take its caption, if any.
-        let mut caption = None;
-        if let Some(cap_sel) = &cap_sel {
-            let mut cursor = el.parent();
-            while let Some(node) = cursor {
-                if let Some(elem) = scraper::ElementRef::wrap(node) {
-                    if elem.value().name() == "figure" {
-                        caption = elem.select(cap_sel).next().map(|c| {
-                            c.text()
-                                .collect::<String>()
-                                .split_whitespace()
-                                .collect::<Vec<_>>()
-                                .join(" ")
-                        });
-                        break;
-                    }
-                    cursor = elem.parent();
-                } else {
-                    break;
-                }
-            }
-        }
-        out.push(ImgRef {
-            src: src.to_string(),
-            alt,
-            caption: caption.filter(|c| !c.is_empty()),
-        });
-    }
-    out
-}
-
-/// Download one image, honoring the timeout and size cap (§3.10).
-pub async fn download(http: &reqwest::Client, url: &str) -> Option<Vec<u8>> {
-    let resp = http
-        .get(url)
-        .timeout(Duration::from_secs(DOWNLOAD_TIMEOUT_SECS))
-        .send()
-        .await
-        .map_err(|e| tracing::debug!(url, "image download failed: {e}"))
-        .ok()?;
-    if !resp.status().is_success() {
-        tracing::debug!(url, status = %resp.status(), "image download rejected");
-        return None;
-    }
-    if let Some(len) = resp.content_length()
-        && len as usize > MAX_IMAGE_BYTES
-    {
-        tracing::debug!(url, len, "image exceeds the size cap");
-        return None;
-    }
-    let mut resp = resp;
-    let mut buf: Vec<u8> = Vec::new();
-    loop {
-        match resp.chunk().await {
-            Ok(Some(chunk)) => {
-                if buf.len() + chunk.len() > MAX_IMAGE_BYTES {
-                    tracing::debug!(url, "image exceeds the size cap mid-stream");
-                    return None;
-                }
-                buf.extend_from_slice(&chunk);
-            }
-            Ok(None) => break,
-            Err(e) => {
-                tracing::debug!(url, "image download interrupted: {e}");
-                return None;
-            }
-        }
-    }
-    if buf.is_empty() { None } else { Some(buf) }
-}
-
-/// Decode, resize/grayscale, flatten transparency to white and re-encode (§3.10).
-///
-/// Line art with transparency is kept as PNG after flattening; everything else
-/// becomes JPEG. Returns `None` for undecodable sources (SVG/WebP without support).
-pub fn reencode(bytes: &[u8], profile: ImageProfile) -> Option<(Vec<u8>, &'static str)> {
-    let format = image::guess_format(bytes).ok();
-    let decoded = image::load_from_memory(bytes)
-        .map_err(|e| tracing::debug!("undecodable image: {e}"))
-        .ok()?;
-
-    let (w, h) = decoded.dimensions();
-    if w < MIN_DIMENSION_PX || h < MIN_DIMENSION_PX {
-        tracing::debug!(w, h, "skipping decorative image");
-        return None;
-    }
-
-    let has_alpha = decoded.color().has_alpha();
-    let flattened = if has_alpha {
-        flatten_to_white(&decoded)
-    } else {
-        decoded
-    };
-
-    let resized = if w > profile.max_width || h > profile.max_height {
-        flattened.resize(
-            profile.max_width,
-            profile.max_height,
-            image::imageops::FilterType::Lanczos3,
-        )
-    } else {
-        flattened
-    };
-
-    // Keep line art (PNG source, few distinct tones) lossless; everything else
-    // becomes JPEG, which is far smaller for photographs (§3.10).
-    let keep_png = format == Some(ImageFormat::Png) && is_line_art(&resized);
-
-    let mut out = Cursor::new(Vec::new());
-    // NB: encode the concrete buffer, not the `DynamicImage` — the latter always
-    // reports RGBA pixels, which would silently re-colorize a grayscale image.
-    if profile.grayscale {
-        let gray = resized.to_luma8();
-        if keep_png {
-            DynamicImage::ImageLuma8(gray)
-                .write_to(&mut out, ImageFormat::Png)
-                .ok()?;
-            return Some((out.into_inner(), "image/png"));
-        }
-        let mut enc =
-            image::codecs::jpeg::JpegEncoder::new_with_quality(&mut out, profile.jpeg_quality);
-        enc.encode_image(&gray).ok()?;
-        return Some((out.into_inner(), "image/jpeg"));
-    }
-
-    let rgb = resized.to_rgb8();
-    if keep_png {
-        DynamicImage::ImageRgb8(rgb)
-            .write_to(&mut out, ImageFormat::Png)
-            .ok()?;
-        return Some((out.into_inner(), "image/png"));
-    }
-    let mut enc =
-        image::codecs::jpeg::JpegEncoder::new_with_quality(&mut out, profile.jpeg_quality);
-    enc.encode_image(&rgb).ok()?;
-    Some((out.into_inner(), "image/jpeg"))
-}
-
-/// Composite over an opaque white page — e-ink has no transparency (§3.10).
-fn flatten_to_white(img: &DynamicImage) -> DynamicImage {
-    let rgba = img.to_rgba8();
-    let mut rgb = image::RgbImage::new(rgba.width(), rgba.height());
-    for (x, y, px) in rgba.enumerate_pixels() {
-        let a = f32::from(px[3]) / 255.0;
-        let blend = |c: u8| {
-            ((f32::from(c) * a) + 255.0 * (1.0 - a))
-                .round()
-                .clamp(0.0, 255.0) as u8
-        };
-        rgb.put_pixel(x, y, image::Rgb([blend(px[0]), blend(px[1]), blend(px[2])]));
-    }
-    DynamicImage::ImageRgb8(rgb)
-}
-
-/// Cheap line-art test: few distinct colors (diagrams, logos, screenshots of text).
-fn is_line_art(img: &DynamicImage) -> bool {
-    const SAMPLE_LIMIT: usize = 20_000;
-    const DISTINCT_LIMIT: usize = 64;
-    let rgb = img.to_rgb8();
-    let mut distinct: Vec<[u8; 3]> = Vec::with_capacity(DISTINCT_LIMIT + 1);
-    for (i, px) in rgb.pixels().enumerate() {
-        if i >= SAMPLE_LIMIT {
-            break;
-        }
-        let c = [px[0], px[1], px[2]];
-        if !distinct.contains(&c) {
-            distinct.push(c);
-            if distinct.len() > DISTINCT_LIMIT {
-                return false;
-            }
-        }
-    }
-    true
-}
-
-/// Everything needed to fetch one image, in deterministic issue order.
-#[derive(Debug, Clone)]
-struct PendingImage {
-    id: String,
-    url: String,
-    alt: String,
-    caption: Option<String>,
-}
-
-fn pending_for_pick(pick: &Pick) -> Vec<PendingImage> {
-    let entry_id = pick.article.best_entry_id;
-    let mut refs = extract_img_refs(&pick.article.content_html);
-    if refs.is_empty() {
-        refs = pick
-            .article
-            .image_urls
-            .iter()
-            .map(|u| ImgRef {
-                src: u.clone(),
-                alt: String::new(),
-                caption: None,
-            })
-            .collect();
-    }
-    refs.into_iter()
-        .filter(|r| r.src.starts_with("http://") || r.src.starts_with("https://"))
-        .take(MAX_IMAGES_PER_ARTICLE)
-        .enumerate()
-        .map(|(i, r)| PendingImage {
-            id: format!("img-{entry_id}-{i}"),
-            url: r.src,
-            alt: r.alt,
-            caption: r.caption,
-        })
-        .collect()
-}
-
-/// Download and re-encode every image referenced by the lineup for one edition,
-/// respecting [`ISSUE_ASSET_BUDGET_BYTES`] (§3.10).
-pub async fn collect_for_issue(
-    http: &reqwest::Client,
-    picks: &[Pick],
-    edition: Edition,
-) -> Vec<ImageAsset> {
-    let profile = ImageProfile::for_edition(edition);
-    let pending: Vec<PendingImage> = picks.iter().flat_map(pending_for_pick).collect();
-    if pending.is_empty() {
-        return Vec::new();
-    }
-    tracing::info!(count = pending.len(), ?edition, "downloading issue images");
-
-    let results: Vec<Option<(PendingImage, Vec<u8>, &'static str)>> =
-        futures::stream::iter(pending.into_iter().map(|p| {
-            let http = http.clone();
-            async move {
-                let raw = download(&http, &p.url).await?;
-                let (bytes, mime) = tokio::task::spawn_blocking(move || reencode(&raw, profile))
-                    .await
-                    .ok()
-                    .flatten()?;
-                Some((p, bytes, mime))
-            }
-        }))
-        .buffered(CONCURRENCY)
-        .collect()
-        .await;
-
-    let mut assets = Vec::new();
-    let mut budget_used = 0usize;
-    let mut skipped = 0usize;
-    for result in results.into_iter().flatten() {
-        let (pending, bytes, mime) = result;
-        if budget_used + bytes.len() > ISSUE_ASSET_BUDGET_BYTES {
-            skipped += 1;
-            continue;
-        }
-        budget_used += bytes.len();
-        let ext = if mime == "image/png" { "png" } else { "jpg" };
-        assets.push(ImageAsset {
-            href: format!("images/{}.{ext}", pending.id),
-            id: pending.id,
-            mime: mime.to_string(),
-            data: bytes,
-            alt: pending.alt,
-            caption: pending.caption,
-            source_url: pending.url,
-        });
-    }
-    if skipped > 0 {
-        tracing::warn!(skipped, budget_used, "issue image budget exhausted");
-    }
-    tracing::info!(
-        embedded = assets.len(),
-        bytes = budget_used,
-        "issue images ready"
-    );
-    assets
-}
-
-// ---------------------------------------------------------------------------
-// Markup rewriting
-// ---------------------------------------------------------------------------
-
-/// End index (exclusive) of the tag starting at `start` (`html[start] == '<'`),
-/// respecting quoted attribute values and comments.
-pub(crate) fn tag_end(html: &str, start: usize) -> Option<usize> {
-    let rest = &html[start..];
-    if rest.starts_with("<!--") {
-        return rest.find("-->").map(|i| start + i + 3);
-    }
-    let mut quote: Option<char> = None;
-    for (i, c) in rest.char_indices().skip(1) {
-        match (quote, c) {
-            (Some(q), c) if c == q => quote = None,
-            (Some(_), _) => {}
-            (None, '"') | (None, '\'') => quote = Some(c),
-            (None, '>') => return Some(start + i + c.len_utf8()),
-            (None, _) => {}
-        }
-    }
-    None
-}
-
-/// Lowercased element name of a tag body such as `img src="…"`.
-pub(crate) fn tag_name(inner: &str) -> String {
-    inner
-        .trim_start_matches('/')
-        .split(|c: char| c.is_whitespace() || c == '/' || c == '>')
-        .next()
-        .unwrap_or("")
-        .to_ascii_lowercase()
-}
-
-/// Parse `name="value"` pairs out of a tag body.
-fn parse_attrs(inner: &str) -> Vec<(String, String)> {
-    let mut attrs = Vec::new();
-    let bytes: Vec<char> = inner.chars().collect();
-    let mut i = 0;
-    // Skip the element name.
-    while i < bytes.len() && !bytes[i].is_whitespace() {
-        i += 1;
-    }
-    while i < bytes.len() {
-        while i < bytes.len() && (bytes[i].is_whitespace() || bytes[i] == '/') {
-            i += 1;
-        }
-        let name_start = i;
-        while i < bytes.len() && !bytes[i].is_whitespace() && bytes[i] != '=' && bytes[i] != '/' {
-            i += 1;
-        }
-        if i == name_start {
-            break;
-        }
-        let name: String = bytes[name_start..i]
-            .iter()
-            .collect::<String>()
-            .to_ascii_lowercase();
-        while i < bytes.len() && bytes[i].is_whitespace() {
-            i += 1;
-        }
-        let mut value = String::new();
-        if i < bytes.len() && bytes[i] == '=' {
-            i += 1;
-            while i < bytes.len() && bytes[i].is_whitespace() {
-                i += 1;
-            }
-            if i < bytes.len() && (bytes[i] == '"' || bytes[i] == '\'') {
-                let quote = bytes[i];
-                i += 1;
-                while i < bytes.len() && bytes[i] != quote {
-                    value.push(bytes[i]);
-                    i += 1;
-                }
-                i += 1;
-            } else {
-                while i < bytes.len() && !bytes[i].is_whitespace() && bytes[i] != '>' {
-                    value.push(bytes[i]);
-                    i += 1;
-                }
-            }
-        }
-        attrs.push((name, value));
-    }
-    attrs
-}
-
-/// Escape a string for use inside a double-quoted XML attribute.
-fn attr_escape(s: &str) -> String {
-    let mut out = String::with_capacity(s.len());
-    for c in s.chars() {
-        match c {
-            '&' => out.push_str("&amp;"),
-            '<' => out.push_str("&lt;"),
-            '>' => out.push_str("&gt;"),
-            '"' => out.push_str("&quot;"),
-            _ => out.push(c),
-        }
-    }
-    out
-}
-
-/// Escape a string for XML text content.
-pub fn text_escape(s: &str) -> String {
-    let mut out = String::with_capacity(s.len());
-    for c in s.chars() {
-        match c {
-            '&' => out.push_str("&amp;"),
-            '<' => out.push_str("&lt;"),
-            '>' => out.push_str("&gt;"),
-            _ => out.push(c),
-        }
-    }
-    out
-}
-
-/// Rewrite `<img src>` to the embedded hrefs, replacing misses with the
-/// `[image: alt]` placeholder paragraph (§3.10).
-pub fn rewrite_img_srcs(html: &str, assets: &[ImageAsset]) -> String {
-    let by_url: HashMap<&str, &ImageAsset> =
-        assets.iter().map(|a| (a.source_url.as_str(), a)).collect();
-    let mut out = String::with_capacity(html.len());
-    let mut cursor = 0usize;
-    while let Some(rel) = html[cursor..].find('<') {
-        let start = cursor + rel;
-        out.push_str(&html[cursor..start]);
-        let Some(end) = tag_end(html, start) else {
-            out.push_str(&html[start..]);
-            return out;
-        };
-        let raw = &html[start..end];
-        let inner = raw
-            .trim_start_matches('<')
-            .trim_end_matches('>')
-            .trim_end_matches('/');
-        if tag_name(inner) == "img" {
-            let attrs = parse_attrs(inner);
-            let src = attrs
-                .iter()
-                .find(|(k, _)| k == "src")
-                .map(|(_, v)| v.trim().to_string())
-                .unwrap_or_default();
-            let alt = attrs
-                .iter()
-                .find(|(k, _)| k == "alt")
-                .map(|(_, v)| v.trim().to_string())
-                .unwrap_or_default();
-            match by_url.get(src.as_str()) {
-                Some(asset) => {
-                    let alt = if alt.is_empty() { &asset.alt } else { &alt };
-                    out.push_str(&format!(
-                        "<img src=\"{}\" alt=\"{}\"/>",
-                        attr_escape(&asset.href),
-                        attr_escape(alt)
-                    ));
-                }
-                None => {
-                    let label = if alt.is_empty() {
-                        "image unavailable"
-                    } else {
-                        &alt
-                    };
-                    out.push_str(&format!(
-                        "<p class=\"image-placeholder\">[image: {}]</p>",
-                        text_escape(label)
-                    ));
-                }
-            }
-        } else {
-            out.push_str(raw);
-        }
-        cursor = end;
-    }
-    out.push_str(&html[cursor..]);
-    out
-}
-
-/// Self-close HTML void elements and normalize `&nbsp;` so the markup parses as
-/// XML — EPUB3 content documents are XHTML (§3.10).
-pub fn to_xhtml(html: &str) -> String {
-    let mut out = String::with_capacity(html.len());
-    let mut cursor = 0usize;
-    while let Some(rel) = html[cursor..].find('<') {
-        let start = cursor + rel;
-        out.push_str(&html[cursor..start]);
-        let Some(end) = tag_end(html, start) else {
-            out.push_str(&html[start..]);
-            cursor = html.len();
-            break;
-        };
-        let raw = &html[start..end];
-        let inner = raw.trim_start_matches('<').trim_end_matches('>');
-        let name = tag_name(inner);
-        if VOID_ELEMENTS.contains(&name.as_str()) && !inner.trim_end().ends_with('/') {
-            out.push('<');
-            out.push_str(inner.trim_end());
-            out.push_str("/>");
-        } else {
-            out.push_str(raw);
-        }
-        cursor = end;
-    }
-    out.push_str(&html[cursor..]);
-    // html5ever (via ammonia) emits `&nbsp;`, which is undefined in XML.
-    out.replace("&nbsp;", "&#160;")
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn asset(url: &str, href: &str) -> ImageAsset {
-        ImageAsset {
-            id: "img-1-0".into(),
-            href: href.into(),
-            mime: "image/jpeg".into(),
-            data: vec![1, 2, 3],
-            alt: "fallback alt".into(),
-            caption: None,
-            source_url: url.into(),
-        }
-    }
-
-    #[test]
-    fn extracts_img_refs_with_captions() {
-        let html = r#"<p>hi</p>
-            <figure><img src="https://e.g/a.png" alt="A diagram"/>
-            <figcaption>Figure 1:  the thing</figcaption></figure>
-            <img src="https://e.g/b.jpg"/>
-            <img src="data:image/png;base64,zz"/>
-            <img src="https://e.g/a.png" alt="dupe"/>"#;
-        let refs = extract_img_refs(html);
-        assert_eq!(refs.len(), 2);
-        assert_eq!(refs[0].src, "https://e.g/a.png");
-        assert_eq!(refs[0].alt, "A diagram");
-        assert_eq!(refs[0].caption.as_deref(), Some("Figure 1: the thing"));
-        assert_eq!(refs[1].src, "https://e.g/b.jpg");
-        assert!(refs[1].caption.is_none());
-    }
-
-    #[test]
-    fn rewrites_hits_and_placeholders_misses() {
-        let assets = vec![asset("https://e.g/a.png", "images/img-1-0.jpg")];
-        let html = r#"<p>x</p><img src="https://e.g/a.png" alt="Alt &amp; more"><img src="https://e.g/gone.png" alt="Missing">"#;
-        let out = rewrite_img_srcs(html, &assets);
-        assert!(out.contains(r#"<img src="images/img-1-0.jpg" alt="Alt &amp;amp; more"/>"#));
-        assert!(out.contains(r#"<p class="image-placeholder">[image: Missing]</p>"#));
-        assert!(!out.contains("gone.png"));
-    }
-
-    #[test]
-    fn placeholder_falls_back_when_alt_is_missing() {
-        let out = rewrite_img_srcs(r#"<img src="https://e.g/x.png">"#, &[]);
-        assert_eq!(
-            out,
-            r#"<p class="image-placeholder">[image: image unavailable]</p>"#
-        );
-    }
-
-    #[test]
-    fn to_xhtml_self_closes_voids_and_entities() {
-        let html = "<p>a<br>b<hr>c&nbsp;d<img src=\"x.png\" alt=\"y\"></p><p>e<br/></p>";
-        let out = to_xhtml(html);
-        assert!(out.contains("<br/>"));
-        assert!(out.contains("<hr/>"));
-        assert!(out.contains("<img src=\"x.png\" alt=\"y\"/>"));
-        assert!(out.contains("&#160;"));
-        assert!(!out.contains("&nbsp;"));
-        assert!(!out.contains("<br/ >"));
-        // Already-closed voids are left alone (no double slash).
-        assert_eq!(out.matches("<br/>").count(), 2);
-    }
-
-    #[test]
-    fn tag_scanner_ignores_angle_brackets_in_attributes() {
-        let html = r#"<a title="a > b">x</a>"#;
-        assert_eq!(to_xhtml(html), html);
-    }
-
-    #[test]
-    fn reencode_resizes_grayscales_and_encodes() {
-        let mut img = image::RgbaImage::new(200, 100);
-        for (x, y, px) in img.enumerate_pixels_mut() {
-            *px = image::Rgba([(x % 256) as u8, (y % 256) as u8, 128, 255]);
-        }
-        let mut png = Cursor::new(Vec::new());
-        DynamicImage::ImageRgba8(img)
-            .write_to(&mut png, ImageFormat::Png)
-            .unwrap();
-        let raw = png.into_inner();
-
-        let (std_bytes, std_mime) = reencode(&raw, ImageProfile::STANDARD).unwrap();
-        assert_eq!(std_mime, "image/jpeg");
-        let decoded = image::load_from_memory(&std_bytes).unwrap();
-        assert_eq!(decoded.dimensions(), (200, 100), "no upscaling");
-
-        let (x4_bytes, _) = reencode(&raw, ImageProfile::X4).unwrap();
-        let x4 = image::load_from_memory(&x4_bytes).unwrap();
-        assert!(x4.width() <= 480 && x4.height() <= 800);
-        assert_eq!(x4.color(), image::ColorType::L8, "X4 is grayscale Luma8");
-    }
-
-    #[test]
-    fn reencode_skips_decorative_images_and_junk() {
-        let tiny = image::RgbaImage::new(8, 8);
-        let mut png = Cursor::new(Vec::new());
-        DynamicImage::ImageRgba8(tiny)
-            .write_to(&mut png, ImageFormat::Png)
-            .unwrap();
-        assert!(reencode(&png.into_inner(), ImageProfile::STANDARD).is_none());
-        assert!(reencode(b"<svg>not an image</svg>", ImageProfile::STANDARD).is_none());
-    }
-
-    #[test]
-    fn line_art_png_stays_png_and_is_flattened() {
-        let mut img = image::RgbaImage::new(120, 60);
-        for (x, _y, px) in img.enumerate_pixels_mut() {
-            *px = if x % 12 == 0 {
-                image::Rgba([0, 0, 0, 255])
-            } else {
-                image::Rgba([255, 255, 255, 0])
-            };
-        }
-        let mut png = Cursor::new(Vec::new());
-        DynamicImage::ImageRgba8(img)
-            .write_to(&mut png, ImageFormat::Png)
-            .unwrap();
-        let (bytes, mime) = reencode(&png.into_inner(), ImageProfile::STANDARD).unwrap();
-        assert_eq!(mime, "image/png");
-        let decoded = image::load_from_memory(&bytes).unwrap();
-        assert!(!decoded.color().has_alpha(), "transparency is flattened");
-        // Transparent pixels became white.
-        assert_eq!(
-            decoded.to_rgb8().get_pixel(1, 1),
-            &image::Rgb([255, 255, 255])
-        );
-    }
-}

src/epub/mod.rs

@@ -5,24 +5,17 @@
 //! `art-{entry_id}` so rating links stay stable across regenerations.
 
 pub mod build;
-pub mod images;
+pub mod chapters;
+pub mod cover;
+pub mod fixtures;
 pub mod x4;
 
 use std::path::{Path, PathBuf};
 
 use crate::config::Config;
+use crate::images;
 use crate::types::{Artifact, Edition, ImageAsset, Issue};
 
-/// Chapter order inside an issue (§3.10).
-pub const CHAPTER_ORDER: &[&str] = &[
-    "cover",
-    "from-the-editor",
-    "in-this-issue",
-    "sections",
-    "world-briefing",
-    "colophon",
-];
-
 #[derive(Debug, thiserror::Error)]
 pub enum EpubError {
     #[error("epub build failed: {0}")]
@@ -189,6 +182,7 @@ mod tests {
                 "OEBPS/disc-1001.xhtml",
                 "OEBPS/art-1002.xhtml",
                 "OEBPS/world.xhtml",
+                "OEBPS/behind.xhtml",
                 "OEBPS/colophon.xhtml",
             ] {
                 assert!(

src/epub/templates/README.md

@@ -8,12 +8,13 @@ the crate root points askama here (`dirs = ["src/epub/templates"]`).
 |---|---|---|
 | `base.xhtml` | — | Shared XHTML skeleton (`{% block body_class %}`, `{% block content %}`) |
 | `cover_page.xhtml` | `CoverPage` | Page that displays the rasterized cover image |
-| `front_page.xhtml` | `FrontPage` | "From the Editor" + issue stats line |
+| `front_page.xhtml` | `FrontPage` | "The Brief" + issue stats line |
 | `in_this_issue.xhtml` | `InThisIssue` | Introduction chapter: per-section linked index |
-| `section.xhtml` | `SectionPage` | Section title page + LLM intro |
+| `section.xhtml` | `SectionPage` | Section title page (name only) |
 | `chapter.xhtml` | `ArticleChapter` | Article: header, body, rating/read-online footer |
 | `discussion.xhtml` | `DiscussionChapter` | Comment chapter (§3.7); body from `comments::render_xhtml` |
 | `world_briefing.xhtml` | `WorldBriefingChapter` | Wikipedia Current Events (§3.8), body from `world::render_xhtml` |
+| `behind.xhtml` | `BehindChapter` | "Behind the paper": run counts, admission mix, near misses, models (§15.1) |
 | `colophon.xhtml` | `ColophonChapter` | Back matter: models, cost, counts |
 | `cover.svg` | `CoverSvg` | Typographic cover, rasterized with resvg + tiny-skia |
 | `style.css` | — | Standard-edition stylesheet, embedded as `stylesheet.css` |

src/epub/templates/behind.xhtml

@@ -0,0 +1,20 @@
+{% extends "base.xhtml" %}
+{% block body_class %}behind{% endblock %}
+{% block content %}
+      <h1>Behind the paper</h1>
+      <p class="fact-line">{{ summary_line }}</p>
+      <p class="fact-line">{{ admitted_line }}</p>
+      <p class="fact-line">{{ learned_line }}</p>
+      <h2>Near misses</h2>
+      <p class="fact-line"><em>Highest utility not selected.</em></p>
+{% if near_misses.is_empty() %}
+      <p class="fact-line">None recorded for this run.</p>
+{% else %}
+      <ul class="near-misses">
+{% for line in near_misses %}
+        <li>{{ line }}</li>
+{% endfor %}
+      </ul>
+{% endif %}
+      <p class="fact-line">{{ models_line }}</p>
+{% endblock %}

src/epub/templates/chapter.xhtml

@@ -7,6 +7,9 @@
         <p class="byline">{{ line }}</p>
 {% endif %}
         <p class="meta">{{ meta_line }}</p>
+{% if let Some(text) = why %}
+        <p class="why"><em>Why it&#39;s here: {{ text }}</em></p>
+{% endif %}
 {% if let Some(line) = social_line %}
         <p class="social">{{ line }}</p>
 {% endif %}
@@ -24,9 +27,10 @@
       <hr class="rule"/>
       <div class="article-footer">
 {% if let Some(links) = rating %}
-        <p class="rating">Was this a good pick? <a href="{{ links.up_url }}">[ &#128077; Yes ]</a> &#183; <a href="{{ links.down_url }}">[ &#128078; No ]</a></p>
+        <p class="rating">Was this a good pick? &#160; <a href="{{ links.loved_url }}">[ Loved it ]</a> &#160; <a href="{{ links.good_url }}">[ Good ]</a> &#160; <a href="{{ links.not_for_me_url }}">[ Not for me ]</a> &#160;&#160;&#160; <a href="{{ read_online_url }}">Read online &#8599;</a></p>
-{% endif %}
+{% else %}
         <p class="read-online"><a href="{{ read_online_url }}">Read online &#8599;</a></p>
+{% endif %}
 {% if let Some(href) = discussion_href %}
         <p class="see-discussion"><a href="{{ href }}">&#128172; Read the discussion</a></p>
 {% endif %}

src/epub/templates/colophon.xhtml

@@ -9,12 +9,17 @@
       </p>
       <p class="fact-line"><strong>Issue:</strong> No. {{ issue_number }} &#183; {{ display_date }}</p>
       <p class="fact-line"><strong>Generated:</strong> {{ generated_at }}</p>
-      <p class="fact-line"><strong>Curation model:</strong> {{ model }}</p>
+      <p class="fact-line"><strong>Bulk model:</strong> {{ bulk_model }}</p>
+      <p class="fact-line"><strong>Editor model:</strong> {{ editor_model }}</p>
+      <p class="fact-line"><strong>Summaries model:</strong> {{ summaries_model }}</p>
       <p class="fact-line"><strong>Entries considered:</strong> {{ entries_fetched }} from {{ feeds_seen }} feeds</p>
       <p class="fact-line"><strong>Candidates scored:</strong> {{ candidates }}</p>
       <p class="fact-line"><strong>Articles selected:</strong> {{ article_count }} across {{ section_count }} sections</p>
       <p class="fact-line"><strong>Words:</strong> {{ total_words }} &#183; {{ reading_line }}</p>
-      <p class="fact-line"><strong>Token cost:</strong> {{ cost_usd }}</p>
+{% for line in provider_costs %}
+      <p class="fact-line"><strong>{{ line.provider }} cost:</strong> {{ line.cost }}</p>
+{% endfor %}
+      <p class="fact-line"><strong>Total token cost:</strong> {{ cost_usd }}</p>
       <p class="fact-line"><strong>Generator:</strong> {{ generator_version }}</p>
       <p class="attribution">
         Article text belongs to its authors and publications; excerpts and links are

src/epub/templates/front_page.xhtml

@@ -4,7 +4,7 @@
       <h1 class="masthead">The Daily EPUB</h1>
       <p class="dateline">{{ display_date }} &#183; No. {{ issue_number }}</p>
       <hr class="rule"/>
-      <h2 class="kicker">From the Editor</h2>
+      <h2 class="kicker">The Brief</h2>
       <div class="editorial">
 {{ body_html|safe }}
       </div>

src/epub/templates/in_this_issue.xhtml

@@ -12,6 +12,9 @@
           <p class="index-meta">{{ entry.source }} &#183; {{ entry.reading_minutes }} min read</p>
 {% if !entry.summary.is_empty() %}
           <p class="index-summary">{{ entry.summary }}</p>
+{% endif %}
+{% if let Some(text) = entry.why %}
+          <p class="index-why"><em>Why it&#39;s here: {{ text }}</em></p>
 {% endif %}
         </li>
 {% endfor %}

src/epub/templates/section.xhtml

@@ -3,7 +3,4 @@
 {% block content %}
       <h1 class="section-title">{{ name }}</h1>
       <hr class="rule"/>
-{% if let Some(text) = intro %}
-      <p class="section-intro">{{ text }}</p>
-{% endif %}
 {% endblock %}

src/epub/templates/style-x4.css

@@ -190,3 +190,5 @@ p.comment-line {
 .fact-line {
   margin: 0 0 0.35em 0;
 }
+
+.why, .index-why { font-size: 0.9em; font-style: italic; }

src/epub/templates/style.css

@@ -224,6 +224,10 @@ hr.rule {
   font-size: 0.9em;
 }
 
+.rating {
+  font-size: 0.8em;
+}
+
 .rating a {
   text-decoration: none;
 }
@@ -292,3 +296,5 @@ blockquote.comment blockquote.comment {
 .fact-line {
   margin: 0 0 0.35em 0;
 }
+
+.why, .index-why { font-size: 0.9em; font-style: italic; }

src/epub/x4.rs

@@ -10,7 +10,7 @@ use std::path::{Path, PathBuf};
 
 use crate::config::XtcConfig;
 
-use super::images::tag_end;
+use crate::html::tag_end;
 
 /// Native X4 screen size, used for the cover and image fitting (§3.10).
 pub const X4_SCREEN: (u32, u32) = (480, 800);

src/extract.rs

@@ -1,24 +1,29 @@
-//! Content extraction, sanitization and word counting (spec §3.3).
+//! Getting an article's body text (spec §3.3).
 //!
 //! Priority order per article: Miniflux content if it looks like full text →
 //! fetch + `dom_smoothie` readability → feed excerpt with a "(excerpt only)" note.
+//! Whichever wins is sanitized down to the tag subset the EPUB templates accept.
+//!
+//! Image handling lives in [`crate::images`]; this module calls into
+//! [`crate::images::normalize`] before readability and before sanitizing, and
+//! reads image URLs back out with [`crate::images::collect_image_urls`].
 
 use std::collections::HashSet;
 use std::sync::Arc;
 use std::time::Duration;
 
 use futures::StreamExt;
-use scraper::{Html, Node, Selector};
 use url::Url;
 
+use crate::html::word_count;
+use crate::images::normalize::{normalize_img_tags, prepare_for_readability};
+use crate::images::refs::collect_image_urls;
 use crate::types::{Article, ExtractMethod, Extracted};
 
 /// Word count at or above which Miniflux content is treated as full text (§3.3).
 pub const FULL_TEXT_MIN_WORDS: i64 = 250;
 /// Maximum bytes downloaded when fetching an article page (§3.3).
 pub const MAX_FETCH_BYTES: usize = 3 * 1024 * 1024;
-/// Maximum images collected per article (§3.3).
-pub const MAX_IMAGES_PER_ARTICLE: usize = 12;
 /// Note appended to bodies we could only excerpt (§3.3).
 pub const EXCERPT_NOTE: &str = "(excerpt only — read online)";
 
@@ -130,20 +135,37 @@ impl Extractor {
         let _guard = span.enter();
 
         // 1. Miniflux content, when it already looks like full text.
-        let feed_html = sanitize_with_base(&article.content_html, &article.url);
+        let feed_html =
+            sanitize_with_base(&normalize_img_tags(&article.content_html), &article.url);
         let feed_words = word_count(&feed_html);
         if feed_words >= FULL_TEXT_MIN_WORDS {
-            return self.finish(article, feed_html, feed_words, ExtractMethod::Miniflux);
+            return self.finish(
+                article,
+                feed_html,
+                feed_words,
+                ExtractMethod::Miniflux,
+                &article.url,
+            );
         }
 
         // 2. Fetch the page and run readability over it.
         if self.can_fetch() {
             match self.fetch_readable(&article.url).await {
-                Ok(html) => {
+                Ok(page) => {
-                    let clean = sanitize_with_base(&html, &article.url);
+                    // Relative URLs in the markup belong to the page we ended up
+                    // on, not the one we asked for: shortener and syndication
+                    // links land on another host entirely.
+                    let clean =
+                        sanitize_with_base(&normalize_img_tags(&page.html), &page.final_url);
                     let words = word_count(&clean);
                     if words > feed_words && words > 0 {
-                        return self.finish(article, clean, words, ExtractMethod::Readability);
+                        return self.finish(
+                            article,
+                            clean,
+                            words,
+                            ExtractMethod::Readability,
+                            &page.final_url,
+                        );
                     }
                     tracing::debug!(words, feed_words, "readability was not an improvement");
                 }
@@ -160,20 +182,24 @@ impl Extractor {
             format!("{feed_html}<p>{EXCERPT_NOTE}</p>")
         };
         let words = word_count(&body);
-        let mut out = self.finish(article, body, words, ExtractMethod::Excerpt);
+        let mut out = self.finish(article, body, words, ExtractMethod::Excerpt, &article.url);
         out.excerpt_only = true;
         out
     }
 
     /// Assemble the [`Extracted`] value once a body has been chosen.
+    ///
+    /// `base_url` is what the body's relative URLs resolve against — the page we
+    /// landed on for a fetched article, the article URL otherwise.
     fn finish(
         &self,
         article: &Article,
         content_html: String,
         words: i64,
         method: ExtractMethod,
+        base_url: &str,
     ) -> Extracted {
-        let image_urls = collect_image_urls(&content_html, &article.url);
+        let image_urls = collect_image_urls(&content_html, base_url);
         let excerpt_only = method == ExtractMethod::Excerpt
             || looks_paywalled(&article.url, words, &self.paywall_domains);
         Extracted {
@@ -220,7 +246,10 @@ impl Extractor {
 
     /// Fetch `url` (10s timeout, desktop UA, [`MAX_FETCH_BYTES`] cap) and run
     /// `dom_smoothie` readability over it (§3.3).
-    pub async fn fetch_readable(&self, url: &str) -> Result<String, ExtractError> {
+    ///
+    /// Returns the URL the fetch actually landed on alongside the markup, so
+    /// callers resolve relative links against the right origin.
+    pub async fn fetch_readable(&self, url: &str) -> Result<Page, ExtractError> {
         let Some(http) = &self.http else {
             return Err(ExtractError::FetchDisabled);
         };
@@ -237,6 +266,7 @@ impl Extractor {
         if !response.status().is_success() {
             return Err(ExtractError::Status(response.status().as_u16()));
         }
+        let final_url = response.url().to_string();
         if let Some(ct) = response
             .headers()
             .get(reqwest::header::CONTENT_TYPE)
@@ -256,17 +286,36 @@ impl Extractor {
             body.extend_from_slice(&chunk);
         }
         let html = String::from_utf8_lossy(&body).into_owned();
-        readability(&html, url)
+        Ok(Page {
+            html: readability(&html, &final_url)?,
+            final_url,
+        })
     }
 }
 
+/// An article page after fetching and readability (§3.3).
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Page {
+    /// Readability's main-content markup.
+    pub html: String,
+    /// Where the fetch ended up, after any redirects — the base for relative URLs.
+    pub final_url: String,
+}
+
 /// Run `dom_smoothie` over a fetched page and return its main-content HTML (§3.3).
+///
+/// The page is normalized first ([`prepare_for_readability`]) so that readability
+/// sees plain, already-resolved `<img src>` elements. Left to itself it damages
+/// them in two ways: it deletes whole subtrees (lightbox `<button>` wrappers take
+/// their image with them) and its lazy-image heuristic overwrites a perfectly
+/// good `src` with whatever other attribute happens to contain `.jpg`.
 pub fn readability(html: &str, url: &str) -> Result<String, ExtractError> {
+    let html = prepare_for_readability(html);
     let config = dom_smoothie::Config {
         max_elements_to_parse: 60_000,
         ..Default::default()
     };
-    let mut readability = dom_smoothie::Readability::new(html, Some(url), Some(config))
+    let mut readability = dom_smoothie::Readability::new(html.as_str(), Some(url), Some(config))
         .map_err(|_| ExtractError::NoContent)?;
     let parsed = readability.parse().map_err(|_| ExtractError::NoContent)?;
     let content = parsed.content.to_string();
@@ -371,75 +420,6 @@ pub fn sanitize_with_base(html: &str, base_url: &str) -> String {
     }
 }
 
-// ---------------------------------------------------------------------------
-// Text measurement (§3.3)
-// ---------------------------------------------------------------------------
-
-/// Visible text of an HTML fragment, entities decoded, `script`/`style` skipped.
-pub fn html_to_text(html: &str) -> String {
-    let document = Html::parse_fragment(html);
-    let mut out = String::with_capacity(html.len() / 2);
-    for node in document.tree.nodes() {
-        let Node::Text(text) = node.value() else {
-            continue;
-        };
-        let hidden = node.ancestors().any(|a| match a.value() {
-            Node::Element(e) => matches!(e.name(), "script" | "style" | "noscript"),
-            _ => false,
-        });
-        if hidden {
-            continue;
-        }
-        out.push_str(text);
-        out.push(' ');
-    }
-    out
-}
-
-/// Count words in rendered text (tags stripped) (§3.3).
-pub fn word_count(html: &str) -> i64 {
-    html_to_text(html)
-        .split_whitespace()
-        .filter(|w| w.chars().any(char::is_alphanumeric))
-        .count() as i64
-}
-
-/// Absolute image URLs referenced by `html`, resolved against `base_url`,
-/// capped at [`MAX_IMAGES_PER_ARTICLE`] (§3.3).
-pub fn collect_image_urls(html: &str, base_url: &str) -> Vec<String> {
-    let Ok(selector) = Selector::parse("img") else {
-        return Vec::new();
-    };
-    let base = Url::parse(base_url).ok();
-    let document = Html::parse_fragment(html);
-    let mut seen: HashSet<String> = HashSet::new();
-    let mut out: Vec<String> = Vec::new();
-    for element in document.select(&selector) {
-        let raw = element
-            .value()
-            .attr("src")
-            .or_else(|| element.value().attr("data-src"))
-            .map(str::trim)
-            .filter(|s| !s.is_empty());
-        let Some(raw) = raw else { continue };
-        let resolved = match Url::parse(raw) {
-            Ok(u) => Some(u),
-            Err(_) => base.as_ref().and_then(|b| b.join(raw).ok()),
-        };
-        let Some(url) = resolved.filter(|u| matches!(u.scheme(), "http" | "https")) else {
-            continue;
-        };
-        let url = url.to_string();
-        if seen.insert(url.clone()) {
-            out.push(url);
-        }
-        if out.len() >= MAX_IMAGES_PER_ARTICLE {
-            break;
-        }
-    }
-    out
-}
-
 /// Heuristic paywall detection: very short text on a known paywall domain (§3.3).
 ///
 /// Two rules: anything under [`EXCERPT_MAX_WORDS`] is a stub whatever the host,
@@ -588,31 +568,19 @@ mod tests {
         assert!(sanitize_with_base(html, "not a url").contains("/next"));
     }
 
-    #[test]
+    /// Feeds carry the same lazy markup pages do, so the same normalization runs
-    fn word_count_ignores_markup_and_script() {
+    /// on Miniflux content before it is sanitized.
-        assert_eq!(word_count("<p>one two three</p>"), 3);
+    #[tokio::test]
-        assert_eq!(word_count("<p>a</p><script>b c d e</script>"), 1);
+    async fn feed_content_is_normalized_too() {
-        assert_eq!(word_count("<p>&amp; &mdash; ok</p>"), 1);
+        let extractor = Extractor::offline(vec![]);
-        assert_eq!(word_count(""), 0);
+        let body = format!(
-        assert_eq!(word_count("<p></p>"), 0);
+            r#"<p>{}</p><img src="data:image/gif;base64,zz" data-src="https://cdn.dev/real.jpg"/>"#,
-    }
+            "lorem ".repeat(400)
-
+        );
-    #[test]
+        let article = article("https://blog.dev/p", &body);
-    fn image_collection_resolves_and_caps() {
+        let out = extractor.extract(&article).await;
-        let mut html = String::from(r#"<img src="/a.png"><img src="https://cdn.dev/b.png">"#);
+        assert_eq!(out.method, ExtractMethod::Miniflux);
-        html.push_str(r#"<img data-src="c.png"><img src="/a.png"><img src="data:image/png;x">"#);
+        assert_eq!(out.image_urls, ["https://cdn.dev/real.jpg"]);
-        for i in 0..20 {
-            html.push_str(&format!(r#"<img src="/n{i}.png">"#));
-        }
-        let urls = collect_image_urls(&html, "https://blog.dev/posts/one");
-        assert_eq!(urls.len(), MAX_IMAGES_PER_ARTICLE);
-        assert_eq!(urls[0], "https://blog.dev/a.png");
-        assert_eq!(urls[1], "https://cdn.dev/b.png");
-        assert_eq!(urls[2], "https://blog.dev/posts/c.png");
-        // Duplicates and data: URIs never appear.
-        assert_eq!(urls.iter().filter(|u| u.ends_with("/a.png")).count(), 1);
-        assert!(!urls.iter().any(|u| u.starts_with("data:")));
-        assert!(collect_image_urls("<p>none</p>", "https://blog.dev").is_empty());
     }
 
     #[test]
@@ -685,6 +653,41 @@ mod tests {
         assert!(articles[1].excerpt_only);
     }
 
+    /// A redirected article's relative image URLs belong to where it landed.
+    ///
+    /// `postgr.es/p/9sl` redirects to `boringsql.com`; resolving its `/images/…`
+    /// against the shortener produced two 404s and two lost charts.
+    #[tokio::test]
+    async fn relative_urls_resolve_against_the_url_we_landed_on() {
+        use axum::response::{Html, Redirect};
+        use axum::routing::get;
+
+        let page = format!(
+            "<html><body><article><h1>Post</h1>{}\
+             <img src=\"images/chart.svg\" alt=\"A chart\"/></article></body></html>",
+            "<p>Body copy that readability will happily keep. </p>".repeat(40)
+        );
+        let app = axum::Router::new()
+            .route("/p/9sl", get(|| async { Redirect::to("/posts/real/") }))
+            .route("/posts/real/", get(move || async move { Html(page) }));
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+        let addr = listener.local_addr().unwrap();
+        let server = tokio::spawn(async move { axum::serve(listener, app).await });
+
+        let http = crate::http::build_client(Duration::from_secs(5)).unwrap();
+        let extractor = Extractor::new(http, vec![]);
+        let article = article(&format!("http://{addr}/p/9sl"), "<p>stub</p>");
+        let out = extractor.extract(&article).await;
+        server.abort();
+
+        assert_eq!(out.method, ExtractMethod::Readability);
+        assert_eq!(
+            out.image_urls,
+            [format!("http://{addr}/posts/real/images/chart.svg")],
+            "the shortener path must not be the base"
+        );
+    }
+
     #[tokio::test]
     async fn offline_extractor_never_fetches() {
         let extractor = Extractor::offline(vec![]);

src/html.rs

@@ -0,0 +1,400 @@
+//! Generic HTML markup utilities shared across the pipeline.
+//!
+//! Everything here is about *markup as text*: scanning tags without building a
+//! DOM, escaping for XML output, and reading a fragment as prose. It knows
+//! nothing about articles, images or EPUBs — those modules build on top of it.
+//!
+//! There are two ways to look at HTML in this crate. `scraper` parses a real
+//! DOM and is the right tool when structure matters (walking up to an enclosing
+//! `<figure>`, say). The scanners here walk the string instead, which is what
+//! you want when the job is to rewrite tags in place and hand back markup that
+//! is otherwise byte-identical.
+
+use scraper::{Html, Node};
+
+/// HTML void elements: XHTML requires them self-closed (§3.10 "valid XHTML").
+pub const VOID_ELEMENTS: &[&str] = &[
+    "area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source",
+    "track", "wbr",
+];
+
+// ---------------------------------------------------------------------------
+// Tag scanning
+// ---------------------------------------------------------------------------
+
+/// End index (exclusive) of the tag starting at `start` (`html[start] == '<'`),
+/// respecting quoted attribute values and comments.
+pub fn tag_end(html: &str, start: usize) -> Option<usize> {
+    let rest = &html[start..];
+    if rest.starts_with("<!--") {
+        return rest.find("-->").map(|i| start + i + 3);
+    }
+    let mut quote: Option<char> = None;
+    for (i, c) in rest.char_indices().skip(1) {
+        match (quote, c) {
+            (Some(q), c) if c == q => quote = None,
+            (Some(_), _) => {}
+            (None, '"') | (None, '\'') => quote = Some(c),
+            (None, '>') => return Some(start + i + c.len_utf8()),
+            (None, _) => {}
+        }
+    }
+    None
+}
+
+/// Lowercased element name of a tag body such as `img src="…"`.
+pub fn tag_name(inner: &str) -> String {
+    inner
+        .trim_start_matches('/')
+        .split(|c: char| c.is_whitespace() || c == '/' || c == '>')
+        .next()
+        .unwrap_or("")
+        .to_ascii_lowercase()
+}
+
+/// Parse `name="value"` pairs out of a tag body, with entity-decoded values.
+///
+/// Decoding matters: this scanner reads markup that ammonia has serialized, and
+/// ammonia writes `&` in a URL as `&amp;`. A raw comparison against a URL that
+/// came out of a real HTML parser would never match (§3.10).
+pub fn parse_attrs(inner: &str) -> Vec<(String, String)> {
+    let mut attrs = Vec::new();
+    let bytes: Vec<char> = inner.chars().collect();
+    let mut i = 0;
+    // Skip the element name.
+    while i < bytes.len() && !bytes[i].is_whitespace() {
+        i += 1;
+    }
+    while i < bytes.len() {
+        while i < bytes.len() && (bytes[i].is_whitespace() || bytes[i] == '/') {
+            i += 1;
+        }
+        let name_start = i;
+        while i < bytes.len() && !bytes[i].is_whitespace() && bytes[i] != '=' && bytes[i] != '/' {
+            i += 1;
+        }
+        if i == name_start {
+            break;
+        }
+        let name: String = bytes[name_start..i]
+            .iter()
+            .collect::<String>()
+            .to_ascii_lowercase();
+        while i < bytes.len() && bytes[i].is_whitespace() {
+            i += 1;
+        }
+        let mut value = String::new();
+        if i < bytes.len() && bytes[i] == '=' {
+            i += 1;
+            while i < bytes.len() && bytes[i].is_whitespace() {
+                i += 1;
+            }
+            if i < bytes.len() && (bytes[i] == '"' || bytes[i] == '\'') {
+                let quote = bytes[i];
+                i += 1;
+                while i < bytes.len() && bytes[i] != quote {
+                    value.push(bytes[i]);
+                    i += 1;
+                }
+                i += 1;
+            } else {
+                while i < bytes.len() && !bytes[i].is_whitespace() && bytes[i] != '>' {
+                    value.push(bytes[i]);
+                    i += 1;
+                }
+            }
+        }
+        attrs.push((name, decode_entities(&value)));
+    }
+    attrs
+}
+
+/// Decode the handful of entities an HTML serializer emits inside attributes.
+///
+/// Numeric forms are included because feeds and WordPress write `&#038;` for
+/// `&`; anything else is left alone rather than guessed at.
+pub fn decode_entities(s: &str) -> String {
+    if !s.contains('&') {
+        return s.to_string();
+    }
+    let mut out = String::with_capacity(s.len());
+    let mut rest = s;
+    while let Some(i) = rest.find('&') {
+        out.push_str(&rest[..i]);
+        let tail = &rest[i..];
+        // Entity names are ASCII, but malformed input is not necessarily so.
+        // Search the bounded byte window for the ASCII delimiter instead of
+        // slicing at byte 12, which may fall in the middle of a UTF-8 scalar.
+        let Some(end) = tail
+            .as_bytes()
+            .iter()
+            .take(12)
+            .position(|&byte| byte == b';')
+        else {
+            out.push('&');
+            rest = &tail[1..];
+            continue;
+        };
+        let entity = &tail[1..end];
+        let decoded = match entity {
+            "amp" => Some('&'),
+            "lt" => Some('<'),
+            "gt" => Some('>'),
+            "quot" => Some('"'),
+            "apos" | "#39" => Some('\''),
+            "nbsp" => Some('\u{a0}'),
+            _ => entity
+                .strip_prefix('#')
+                .and_then(|n| match n.strip_prefix(['x', 'X']) {
+                    Some(hex) => u32::from_str_radix(hex, 16).ok(),
+                    None => n.parse::<u32>().ok(),
+                })
+                .and_then(char::from_u32),
+        };
+        match decoded {
+            Some(c) => {
+                out.push(c);
+                rest = &tail[end + 1..];
+            }
+            None => {
+                out.push('&');
+                rest = &tail[1..];
+            }
+        }
+    }
+    out.push_str(rest);
+    out
+}
+
+/// Prefix of `s` that fits within `max_bytes` without splitting a UTF-8
+/// character.
+///
+/// This is intended for byte-budgeted logs and excerpts. The returned string
+/// may be shorter than the limit by up to three bytes.
+pub fn truncate_utf8(s: &str, max_bytes: usize) -> &str {
+    let mut end = max_bytes.min(s.len());
+    while !s.is_char_boundary(end) {
+        end -= 1;
+    }
+    // `end` is explicitly adjusted to a character boundary above.
+    s.get(..end).unwrap_or_default()
+}
+
+/// Escape a string for use inside a double-quoted XML attribute.
+pub fn attr_escape(s: &str) -> String {
+    let mut out = String::with_capacity(s.len());
+    for c in s.chars() {
+        match c {
+            '&' => out.push_str("&amp;"),
+            '<' => out.push_str("&lt;"),
+            '>' => out.push_str("&gt;"),
+            '"' => out.push_str("&quot;"),
+            _ => out.push(c),
+        }
+    }
+    out
+}
+
+/// Escape a string for XML text content.
+pub fn text_escape(s: &str) -> String {
+    let mut out = String::with_capacity(s.len());
+    for c in s.chars() {
+        match c {
+            '&' => out.push_str("&amp;"),
+            '<' => out.push_str("&lt;"),
+            '>' => out.push_str("&gt;"),
+            _ => out.push(c),
+        }
+    }
+    out
+}
+
+// ---------------------------------------------------------------------------
+// XHTML output
+// ---------------------------------------------------------------------------
+
+/// Self-close HTML void elements and normalize `&nbsp;` so the markup parses as
+/// XML — EPUB3 content documents are XHTML (§3.10).
+pub fn to_xhtml(html: &str) -> String {
+    let mut out = String::with_capacity(html.len());
+    let mut cursor = 0usize;
+    while let Some(rel) = html[cursor..].find('<') {
+        let start = cursor + rel;
+        out.push_str(&html[cursor..start]);
+        let Some(end) = tag_end(html, start) else {
+            out.push_str(&html[start..]);
+            cursor = html.len();
+            break;
+        };
+        let raw = &html[start..end];
+        let inner = raw.trim_start_matches('<').trim_end_matches('>');
+        let name = tag_name(inner);
+        if VOID_ELEMENTS.contains(&name.as_str()) && !inner.trim_end().ends_with('/') {
+            out.push('<');
+            out.push_str(inner.trim_end());
+            out.push_str("/>");
+        } else {
+            out.push_str(raw);
+        }
+        cursor = end;
+    }
+    out.push_str(&html[cursor..]);
+    // html5ever (via ammonia) emits `&nbsp;`, which is undefined in XML.
+    out.replace("&nbsp;", "&#160;")
+}
+
+// ---------------------------------------------------------------------------
+// Reading markup as text
+// ---------------------------------------------------------------------------
+
+/// Visible text of an HTML fragment, entities decoded, `script`/`style` skipped.
+pub fn html_to_text(html: &str) -> String {
+    let document = Html::parse_fragment(html);
+    let mut out = String::with_capacity(html.len() / 2);
+    for node in document.tree.nodes() {
+        let Node::Text(text) = node.value() else {
+            continue;
+        };
+        let hidden = node.ancestors().any(|a| match a.value() {
+            Node::Element(e) => matches!(e.name(), "script" | "style" | "noscript"),
+            _ => false,
+        });
+        if hidden {
+            continue;
+        }
+        out.push_str(text);
+        out.push(' ');
+    }
+    out
+}
+
+/// Count words in rendered text (tags stripped) (§3.3).
+pub fn word_count(html: &str) -> i64 {
+    html_to_text(html)
+        .split_whitespace()
+        .filter(|w| w.chars().any(char::is_alphanumeric))
+        .count() as i64
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn tag_scanner_finds_the_end_of_awkward_tags() {
+        // A `>` inside a quoted attribute is not the end of the tag.
+        let html = r#"<a title="a > b">x</a>"#;
+        assert_eq!(
+            tag_end(html, 0),
+            Some(17),
+            "past the closing `>`, not the one in the title"
+        );
+        // Comments end at `-->`, not at the first `>`.
+        let comment = "<!-- a > b -->rest";
+        assert_eq!(tag_end(comment, 0), Some(14));
+        // An unterminated tag has no end.
+        assert_eq!(tag_end("<p class=\"x", 0), None);
+    }
+
+    #[test]
+    fn tag_names_are_lowercased_and_stripped() {
+        assert_eq!(tag_name("IMG src=\"x\""), "img");
+        assert_eq!(tag_name("/DIV"), "div");
+        assert_eq!(tag_name("br/"), "br");
+        assert_eq!(tag_name(""), "");
+    }
+
+    #[test]
+    fn attributes_parse_in_every_spelling() {
+        let attrs = parse_attrs(r#"img src="a.png" ALT='an alt' loading=lazy hidden"#);
+        let get = |k: &str| attrs.iter().find(|(n, _)| n == k).map(|(_, v)| v.as_str());
+        assert_eq!(get("src"), Some("a.png"));
+        assert_eq!(
+            get("alt"),
+            Some("an alt"),
+            "names lowercase, quotes either way"
+        );
+        assert_eq!(get("loading"), Some("lazy"), "unquoted values work");
+        assert_eq!(get("hidden"), Some(""), "valueless attributes are empty");
+    }
+
+    #[test]
+    fn attribute_values_come_back_entity_decoded() {
+        // The reason this matters: a serializer writes `&` in a URL as `&amp;`,
+        // and callers compare against URLs a real parser produced.
+        let attrs = parse_attrs(r#"img src="a.jpg?id=1&amp;w=9&#038;h=2""#);
+        assert_eq!(attrs[0].1, "a.jpg?id=1&w=9&h=2");
+    }
+
+    #[test]
+    fn entity_decoding_covers_named_decimal_and_hex() {
+        assert_eq!(decode_entities("a &amp; b"), "a & b");
+        assert_eq!(decode_entities("&#038;&#x26;"), "&&");
+        assert_eq!(decode_entities("&lt;p&gt;"), "<p>");
+        assert_eq!(decode_entities("caf&#233;"), "café");
+        // Nothing to do, and nothing invented for what we do not know.
+        assert_eq!(decode_entities("plain"), "plain");
+        assert_eq!(decode_entities("&unknown; &"), "&unknown; &");
+    }
+
+    #[test]
+    fn entity_decoding_handles_unicode_at_the_scan_boundary() {
+        // The curly apostrophe begins at byte 11 after `&`. The bounded entity
+        // scan must leave malformed/non-entity text alone rather than slicing
+        // through the apostrophe and panicking.
+        let input = "&abcdefghij’ rest";
+        assert_eq!(decode_entities(input), input);
+
+        // A semicolon after non-ASCII text is likewise safe and remains
+        // unchanged because it is not one of the entities we decode.
+        assert_eq!(decode_entities("&é;"), "&é;");
+    }
+
+    #[test]
+    fn utf8_truncation_respects_byte_limits_and_character_boundaries() {
+        assert_eq!(truncate_utf8("abcdef", 4), "abcd");
+        assert_eq!(truncate_utf8("ab’cd", 4), "ab");
+        assert_eq!(truncate_utf8("ab’cd", 5), "ab’");
+        assert_eq!(truncate_utf8("éclair", 0), "");
+        assert_eq!(truncate_utf8("éclair", usize::MAX), "éclair");
+    }
+
+    #[test]
+    fn escaping_is_the_inverse_that_output_needs() {
+        assert_eq!(
+            attr_escape(r#"a & "b" <c>"#),
+            "a &amp; &quot;b&quot; &lt;c&gt;"
+        );
+        // Text content keeps quotes as they are.
+        assert_eq!(text_escape(r#"a & "b" <c>"#), r#"a &amp; "b" &lt;c&gt;"#);
+    }
+
+    #[test]
+    fn to_xhtml_self_closes_voids_and_entities() {
+        let html = "<p>a<br>b<hr>c&nbsp;d<img src=\"x.png\" alt=\"y\"></p><p>e<br/></p>";
+        let out = to_xhtml(html);
+        assert!(out.contains("<br/>"));
+        assert!(out.contains("<hr/>"));
+        assert!(out.contains("<img src=\"x.png\" alt=\"y\"/>"));
+        assert!(out.contains("&#160;"));
+        assert!(!out.contains("&nbsp;"));
+        assert!(!out.contains("<br/ >"));
+        // Already-closed voids are left alone (no double slash).
+        assert_eq!(out.matches("<br/>").count(), 2);
+    }
+
+    #[test]
+    fn to_xhtml_ignores_angle_brackets_in_attributes() {
+        let html = r#"<a title="a > b">x</a>"#;
+        assert_eq!(to_xhtml(html), html);
+    }
+
+    #[test]
+    fn word_count_ignores_markup_and_script() {
+        assert_eq!(word_count("<p>one two three</p>"), 3);
+        assert_eq!(word_count("<p>a</p><script>b c d e</script>"), 1);
+        assert_eq!(word_count("<p>&amp; &mdash; ok</p>"), 1);
+        assert_eq!(word_count(""), 0);
+        assert_eq!(word_count("<p></p>"), 0);
+    }
+}

src/images/embed.rs

@@ -0,0 +1,193 @@
+//! Pointing article markup at the images actually embedded in the EPUB.
+
+use std::collections::HashMap;
+
+use crate::html::{attr_escape, parse_attrs, tag_end, tag_name, text_escape};
+use crate::types::ImageAsset;
+
+/// Whether an image we could not embed is worth telling the reader about.
+///
+/// Only descriptive alt text qualifies. A filename, a bare label like `red line`
+/// on a divider rule, or no alt at all carries nothing the reader loses by not
+/// seeing the picture — announcing those turns every decorative graphic and
+/// dead link into a line of clutter, which is how the placeholders got out of
+/// hand in the first place.
+fn alt_is_worth_announcing(alt: &str) -> bool {
+    const MIN_DESCRIPTIVE_WORDS: usize = 4;
+    !alt.is_empty()
+        && !is_filename_alt(alt)
+        && alt.split_whitespace().count() >= MIN_DESCRIPTIVE_WORDS
+}
+
+/// True for alt text that is really just the uploaded filename — `IMG_0808.JPG`,
+/// `cut pieces v01.JPG`, `chart-final-2.png`.
+fn is_filename_alt(alt: &str) -> bool {
+    let alt = alt.trim();
+    if alt.contains(' ') && alt.split_whitespace().count() > 4 {
+        return false;
+    }
+    let Some((stem, ext)) = alt.rsplit_once('.') else {
+        return false;
+    };
+    let ext = ext.to_ascii_lowercase();
+    matches!(
+        ext.as_str(),
+        "jpg" | "jpeg" | "png" | "gif" | "webp" | "svg" | "avif" | "bmp" | "heic"
+    ) && !stem.is_empty()
+        && stem
+            .chars()
+            .all(|c| c.is_alphanumeric() || matches!(c, ' ' | '_' | '-' | '.'))
+}
+
+/// Rewrite `<img src>` to the embedded hrefs, replacing misses with the
+/// `[image: alt]` placeholder paragraph (§3.10).
+pub fn rewrite_img_srcs(html: &str, assets: &[ImageAsset]) -> String {
+    let by_url: HashMap<&str, &ImageAsset> =
+        assets.iter().map(|a| (a.source_url.as_str(), a)).collect();
+    let mut out = String::with_capacity(html.len());
+    let mut cursor = 0usize;
+    while let Some(rel) = html[cursor..].find('<') {
+        let start = cursor + rel;
+        out.push_str(&html[cursor..start]);
+        let Some(end) = tag_end(html, start) else {
+            out.push_str(&html[start..]);
+            return out;
+        };
+        let raw = &html[start..end];
+        let inner = raw
+            .trim_start_matches('<')
+            .trim_end_matches('>')
+            .trim_end_matches('/');
+        if tag_name(inner) == "img" {
+            let attrs = parse_attrs(inner);
+            let src = attrs
+                .iter()
+                .find(|(k, _)| k == "src")
+                .map(|(_, v)| v.trim().to_string())
+                .unwrap_or_default();
+            let alt = attrs
+                .iter()
+                .find(|(k, _)| k == "alt")
+                .map(|(_, v)| v.trim().to_string())
+                .unwrap_or_default();
+            match by_url.get(src.as_str()) {
+                Some(asset) => {
+                    let alt = if alt.is_empty() { &asset.alt } else { &alt };
+                    out.push_str(&format!(
+                        "<img src=\"{}\" alt=\"{}\"/>",
+                        attr_escape(&asset.href),
+                        attr_escape(alt)
+                    ));
+                }
+                // An image we could not embed is only worth announcing when its
+                // alt text tells the reader something; otherwise the `<img>`
+                // just goes away. That covers the decorative graphics the
+                // re-encoder deliberately skips as well as genuine misses (§3.10).
+                None if !alt_is_worth_announcing(&alt) => {}
+                None => {
+                    out.push_str(&format!(
+                        "<p class=\"image-placeholder\">[image: {}]</p>",
+                        text_escape(&alt)
+                    ));
+                }
+            }
+        } else {
+            out.push_str(raw);
+        }
+        cursor = end;
+    }
+    out.push_str(&html[cursor..]);
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn asset(url: &str, href: &str) -> ImageAsset {
+        ImageAsset {
+            id: "img-1-0".into(),
+            href: href.into(),
+            mime: "image/jpeg".into(),
+            data: vec![1, 2, 3],
+            alt: "fallback alt".into(),
+            caption: None,
+            source_url: url.into(),
+        }
+    }
+
+    #[test]
+    fn rewrites_hits_and_placeholders_misses() {
+        let assets = vec![asset("https://e.g/a.png", "images/img-1-0.jpg")];
+        let html = r#"<p>x</p><img src="https://e.g/a.png" alt="Alt &amp; more"><img src="https://e.g/gone.png" alt="A chart of missing things">"#;
+        let out = rewrite_img_srcs(html, &assets);
+        // The alt round-trips through one level of escaping, not two.
+        assert!(out.contains(r#"<img src="images/img-1-0.jpg" alt="Alt &amp; more"/>"#));
+        assert!(
+            out.contains(r#"<p class="image-placeholder">[image: A chart of missing things]</p>"#)
+        );
+        assert!(!out.contains("gone.png"));
+    }
+
+    /// The whole point of the fix: ammonia writes `&amp;` into the markup, and
+    /// the asset was keyed on the URL a real parser produced.
+    #[test]
+    fn entity_encoded_urls_still_match_their_asset() {
+        let assets = vec![asset(
+            "https://e.g/a.jpg?id=1&width=980",
+            "images/img-1-0.jpg",
+        )];
+        let html = r#"<img src="https://e.g/a.jpg?id=1&amp;width=980" alt="Chart"/>"#;
+        assert!(rewrite_img_srcs(html, &assets).contains(r#"src="images/img-1-0.jpg""#));
+        // The numeric spelling WordPress emits works too.
+        let html = r#"<img src="https://e.g/a.jpg?id=1&#038;width=980" alt="Chart"/>"#;
+        assert!(rewrite_img_srcs(html, &assets).contains(r#"src="images/img-1-0.jpg""#));
+    }
+
+    #[test]
+    fn unembeddable_images_only_speak_up_when_the_alt_says_something() {
+        // No alt at all: the image simply disappears.
+        assert_eq!(
+            rewrite_img_srcs(r#"<img src="https://e.g/x.png">"#, &[]),
+            ""
+        );
+        // A filename is not a description.
+        assert_eq!(
+            rewrite_img_srcs(r#"<img src="https://e.g/x.png" alt="IMG_0808.JPG">"#, &[]),
+            ""
+        );
+        // Neither is the label on a decorative divider rule.
+        assert_eq!(
+            rewrite_img_srcs(r#"<img src="https://e.g/rule.png" alt="red line">"#, &[]),
+            ""
+        );
+        // A real description is worth keeping.
+        assert!(
+            rewrite_img_srcs(
+                r#"<img src="https://e.g/x.png" alt="A man in a hard hat stands over a well hole">"#,
+                &[]
+            )
+            .contains("[image: A man in a hard hat stands over a well hole]")
+        );
+    }
+
+    #[test]
+    fn filename_alt_detection() {
+        for yes in [
+            "IMG_0808.JPG",
+            "cut pieces v01.JPG",
+            "chart-final-2.png",
+            "diagram.svg",
+        ] {
+            assert!(is_filename_alt(yes), "{yes} should read as a filename");
+        }
+        for no in [
+            "A hydrogen well head",
+            "",
+            "Fig. 3",
+            "The lion-man of Hohlenstein-Stadel, carved from mammoth ivory.",
+        ] {
+            assert!(!is_filename_alt(no), "{no} should read as a description");
+        }
+    }
+}

src/images/encode.rs

@@ -0,0 +1,350 @@
+//! Turning downloaded bytes into something an e-reader can display.
+//!
+//! Every image is decoded, fitted to the edition's profile, flattened onto white
+//! (e-ink has no transparency) and re-encoded. SVG is rasterized on the way in,
+//! because charts and diagrams are frequently vector-only.
+
+use std::io::Cursor;
+
+use image::{DynamicImage, GenericImageView, ImageFormat, ImageReader, Limits};
+
+use crate::types::Edition;
+
+/// Images smaller than this in either dimension are decorative — skipped (§3.10).
+pub const MIN_DIMENSION_PX: u32 = 24;
+/// Maximum width or height accepted from a decoded raster source.
+pub const MAX_DECODED_DIMENSION_PX: u32 = 16_384;
+/// Maximum memory the image decoder may allocate for one source image.
+pub const MAX_DECODE_ALLOC_BYTES: u64 = 512 * 1024 * 1024;
+/// Width an SVG is rendered at when the profile asks for less than this.
+const SVG_FALLBACK_SIZE: u32 = 1000;
+
+/// Per-edition re-encoding parameters (§3.10).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ImageProfile {
+    pub max_width: u32,
+    pub max_height: u32,
+    pub jpeg_quality: u8,
+    pub grayscale: bool,
+}
+
+impl ImageProfile {
+    /// Standard edition: max width 1200px, JPEG q80, color (§3.10).
+    pub const STANDARD: ImageProfile = ImageProfile {
+        max_width: 1200,
+        max_height: 4000,
+        jpeg_quality: 80,
+        grayscale: false,
+    };
+
+    /// X4 edition: grayscale Luma8, fit within 480×800, JPEG q70 (§3.10).
+    pub const X4: ImageProfile = ImageProfile {
+        max_width: 480,
+        max_height: 800,
+        jpeg_quality: 70,
+        grayscale: true,
+    };
+
+    pub fn for_edition(edition: Edition) -> Self {
+        match edition {
+            Edition::Standard => Self::STANDARD,
+            Edition::X4 => Self::X4,
+        }
+    }
+}
+
+/// Decode, resize/grayscale, flatten transparency to white and re-encode (§3.10).
+///
+/// Line art with transparency is kept as PNG after flattening; everything else
+/// becomes JPEG. SVG is rasterized first — charts and diagrams are frequently
+/// vector-only, and dropping them loses the point of the article. Returns `None`
+/// for sources no decoder handles.
+pub fn reencode(bytes: &[u8], profile: ImageProfile) -> Option<(Vec<u8>, &'static str)> {
+    if looks_like_svg(bytes) {
+        let raster = rasterize_svg(bytes, profile)?;
+        return reencode(&raster, profile);
+    }
+    let mut reader = ImageReader::new(Cursor::new(bytes))
+        .with_guessed_format()
+        .map_err(|e| tracing::debug!("unrecognized image: {e}"))
+        .ok()?;
+    let format = reader.format();
+    let mut limits = Limits::default();
+    limits.max_image_width = Some(MAX_DECODED_DIMENSION_PX);
+    limits.max_image_height = Some(MAX_DECODED_DIMENSION_PX);
+    limits.max_alloc = Some(MAX_DECODE_ALLOC_BYTES);
+    reader.limits(limits);
+    let decoded = reader
+        .decode()
+        .map_err(|e| tracing::debug!("undecodable image: {e}"))
+        .ok()?;
+
+    let (w, h) = decoded.dimensions();
+    if w < MIN_DIMENSION_PX || h < MIN_DIMENSION_PX {
+        tracing::debug!(w, h, "skipping decorative image");
+        return None;
+    }
+
+    let has_alpha = decoded.color().has_alpha();
+    let flattened = if has_alpha {
+        flatten_to_white(&decoded)
+    } else {
+        decoded
+    };
+
+    let resized = if w > profile.max_width || h > profile.max_height {
+        flattened.resize(
+            profile.max_width,
+            profile.max_height,
+            image::imageops::FilterType::Lanczos3,
+        )
+    } else {
+        flattened
+    };
+
+    // Keep line art (PNG source, few distinct tones) lossless; everything else
+    // becomes JPEG, which is far smaller for photographs (§3.10).
+    let keep_png = format == Some(ImageFormat::Png) && is_line_art(&resized);
+
+    let mut out = Cursor::new(Vec::new());
+    // NB: encode the concrete buffer, not the `DynamicImage` — the latter always
+    // reports RGBA pixels, which would silently re-colorize a grayscale image.
+    if profile.grayscale {
+        let gray = resized.to_luma8();
+        if keep_png {
+            DynamicImage::ImageLuma8(gray)
+                .write_to(&mut out, ImageFormat::Png)
+                .ok()?;
+            return Some((out.into_inner(), "image/png"));
+        }
+        let mut enc =
+            image::codecs::jpeg::JpegEncoder::new_with_quality(&mut out, profile.jpeg_quality);
+        enc.encode_image(&gray).ok()?;
+        return Some((out.into_inner(), "image/jpeg"));
+    }
+
+    let rgb = resized.to_rgb8();
+    if keep_png {
+        DynamicImage::ImageRgb8(rgb)
+            .write_to(&mut out, ImageFormat::Png)
+            .ok()?;
+        return Some((out.into_inner(), "image/png"));
+    }
+    let mut enc =
+        image::codecs::jpeg::JpegEncoder::new_with_quality(&mut out, profile.jpeg_quality);
+    enc.encode_image(&rgb).ok()?;
+    Some((out.into_inner(), "image/jpeg"))
+}
+
+/// True when `bytes` are an SVG document (possibly behind an XML prolog or BOM).
+fn looks_like_svg(bytes: &[u8]) -> bool {
+    let head = &bytes[..bytes.len().min(1024)];
+    let text = String::from_utf8_lossy(head);
+    let text = text.trim_start_matches('\u{feff}').trim_start();
+    text.starts_with("<svg")
+        || (text.starts_with("<?xml") || text.starts_with("<!DOCTYPE svg")) && text.contains("<svg")
+}
+
+/// Rasterize an SVG to a PNG at the profile's target width (§3.10).
+///
+/// The profile's own resize pass then handles the height cap, so this only has
+/// to land in the right ballpark.
+fn rasterize_svg(bytes: &[u8], profile: ImageProfile) -> Option<Vec<u8>> {
+    let mut options = resvg::usvg::Options::default();
+    options.fontdb_mut().load_system_fonts();
+    let tree = resvg::usvg::Tree::from_data(bytes, &options)
+        .map_err(|e| tracing::debug!("svg did not parse: {e}"))
+        .ok()?;
+
+    // An `<svg>` that parses but draws nothing is not an image, it is a stray
+    // tag: rasterizing it would embed a blank rectangle.
+    if tree.root().children().is_empty() {
+        tracing::debug!("svg has nothing to draw");
+        return None;
+    }
+    let size = tree.size();
+    let (sw, sh) = (size.width(), size.height());
+    if !(sw.is_finite() && sh.is_finite()) || sw <= 0.0 || sh <= 0.0 {
+        return None;
+    }
+    // Judge "decorative" by the declared size, before scaling: a 16×16 icon is
+    // an icon however large we choose to draw it.
+    if sw < MIN_DIMENSION_PX as f32 || sh < MIN_DIMENSION_PX as f32 {
+        tracing::debug!(sw, sh, "skipping decorative svg");
+        return None;
+    }
+    // Vector art has no native resolution, so render straight at the edition's
+    // target width — upscaling a rasterized copy afterwards would only blur it.
+    let target_w = profile
+        .max_width
+        .max(SVG_FALLBACK_SIZE.min(profile.max_width));
+    let scale = (target_w as f32 / sw).min(profile.max_height as f32 / sh);
+    let scale = if scale.is_finite() && scale > 0.0 {
+        scale
+    } else {
+        1.0
+    };
+    let (w, h) = (
+        (sw * scale).round().max(1.0) as u32,
+        (sh * scale).round().max(1.0) as u32,
+    );
+    let mut pixmap = tiny_skia::Pixmap::new(w, h)?;
+    // E-ink has no transparency; render onto white so alpha never becomes black.
+    pixmap.fill(tiny_skia::Color::WHITE);
+    resvg::render(
+        &tree,
+        tiny_skia::Transform::from_scale(scale, scale),
+        &mut pixmap.as_mut(),
+    );
+    let rgba = image::RgbaImage::from_raw(w, h, pixmap.take_demultiplied())?;
+    let mut png = Cursor::new(Vec::new());
+    DynamicImage::ImageRgba8(rgba)
+        .write_to(&mut png, ImageFormat::Png)
+        .ok()?;
+    Some(png.into_inner())
+}
+
+/// Composite over an opaque white page — e-ink has no transparency (§3.10).
+fn flatten_to_white(img: &DynamicImage) -> DynamicImage {
+    let rgba = img.to_rgba8();
+    let mut rgb = image::RgbImage::new(rgba.width(), rgba.height());
+    for (x, y, px) in rgba.enumerate_pixels() {
+        let a = f32::from(px[3]) / 255.0;
+        let blend = |c: u8| {
+            ((f32::from(c) * a) + 255.0 * (1.0 - a))
+                .round()
+                .clamp(0.0, 255.0) as u8
+        };
+        rgb.put_pixel(x, y, image::Rgb([blend(px[0]), blend(px[1]), blend(px[2])]));
+    }
+    DynamicImage::ImageRgb8(rgb)
+}
+
+/// Cheap line-art test: few distinct colors (diagrams, logos, screenshots of text).
+fn is_line_art(img: &DynamicImage) -> bool {
+    const SAMPLE_LIMIT: usize = 20_000;
+    const DISTINCT_LIMIT: usize = 64;
+    let rgb = img.to_rgb8();
+    let mut distinct: Vec<[u8; 3]> = Vec::with_capacity(DISTINCT_LIMIT + 1);
+    for (i, px) in rgb.pixels().enumerate() {
+        if i >= SAMPLE_LIMIT {
+            break;
+        }
+        let c = [px[0], px[1], px[2]];
+        if !distinct.contains(&c) {
+            distinct.push(c);
+            if distinct.len() > DISTINCT_LIMIT {
+                return false;
+            }
+        }
+    }
+    true
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn reencode_resizes_grayscales_and_encodes() {
+        let mut img = image::RgbaImage::new(200, 100);
+        for (x, y, px) in img.enumerate_pixels_mut() {
+            *px = image::Rgba([(x % 256) as u8, (y % 256) as u8, 128, 255]);
+        }
+        let mut png = Cursor::new(Vec::new());
+        DynamicImage::ImageRgba8(img)
+            .write_to(&mut png, ImageFormat::Png)
+            .unwrap();
+        let raw = png.into_inner();
+
+        let (std_bytes, std_mime) = reencode(&raw, ImageProfile::STANDARD).unwrap();
+        assert_eq!(std_mime, "image/jpeg");
+        let decoded = image::load_from_memory(&std_bytes).unwrap();
+        assert_eq!(decoded.dimensions(), (200, 100), "no upscaling");
+
+        let (x4_bytes, _) = reencode(&raw, ImageProfile::X4).unwrap();
+        let x4 = image::load_from_memory(&x4_bytes).unwrap();
+        assert!(x4.width() <= 480 && x4.height() <= 800);
+        assert_eq!(x4.color(), image::ColorType::L8, "X4 is grayscale Luma8");
+    }
+
+    #[test]
+    fn reencode_skips_decorative_images_and_junk() {
+        let tiny = image::RgbaImage::new(8, 8);
+        let mut png = Cursor::new(Vec::new());
+        DynamicImage::ImageRgba8(tiny)
+            .write_to(&mut png, ImageFormat::Png)
+            .unwrap();
+        assert!(reencode(&png.into_inner(), ImageProfile::STANDARD).is_none());
+        // A bare `<svg>` tag with nothing to draw is markup, not a picture.
+        assert!(reencode(b"<svg>not an image</svg>", ImageProfile::STANDARD).is_none());
+        assert!(reencode(b"not an image at all", ImageProfile::STANDARD).is_none());
+    }
+
+    #[test]
+    fn reencode_rejects_excessive_source_dimensions() {
+        // A narrow image keeps the fixture cheap while exercising the strict
+        // source-dimension limit before any large decoded buffer is allocated.
+        let img = image::GrayImage::new(MAX_DECODED_DIMENSION_PX + 1, MIN_DIMENSION_PX);
+        let mut png = Cursor::new(Vec::new());
+        DynamicImage::ImageLuma8(img)
+            .write_to(&mut png, ImageFormat::Png)
+            .unwrap();
+
+        assert!(reencode(&png.into_inner(), ImageProfile::STANDARD).is_none());
+        assert_eq!(MAX_DECODE_ALLOC_BYTES, 512 * 1024 * 1024);
+    }
+
+    #[test]
+    fn svg_charts_are_rasterized_rather_than_dropped() {
+        let svg = br##"<svg xmlns="http://www.w3.org/2000/svg" width="400" height="300">
+            <rect x="10" y="10" width="380" height="280" fill="#3355bb"/>
+            <circle cx="200" cy="150" r="60" fill="#ffcc00"/>
+        </svg>"##;
+        let (bytes, mime) = reencode(svg, ImageProfile::STANDARD).expect("svg rasterizes");
+        assert_eq!(mime, "image/png", "flat colour art stays lossless");
+        let decoded = image::load_from_memory(&bytes).expect("decodable output");
+        // Drawn at the edition's target width, not at the SVG's nominal size.
+        assert_eq!(decoded.dimensions(), (1200, 900));
+        assert!(!decoded.color().has_alpha(), "rendered onto white");
+
+        // An XML prolog and a leading BOM must not hide the format.
+        let with_prolog = format!(
+            "\u{feff}<?xml version=\"1.0\"?>{}",
+            String::from_utf8_lossy(svg)
+        );
+        let (x4, _) = reencode(with_prolog.as_bytes(), ImageProfile::X4).expect("x4 rasterizes");
+        let x4 = image::load_from_memory(&x4).unwrap();
+        assert!(x4.width() <= 480 && x4.height() <= 800);
+
+        // A 16×16 icon is decorative however large we could draw it.
+        let icon = br#"<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
+            <rect width="16" height="16"/></svg>"#;
+        assert!(reencode(icon, ImageProfile::STANDARD).is_none());
+    }
+
+    #[test]
+    fn line_art_png_stays_png_and_is_flattened() {
+        let mut img = image::RgbaImage::new(120, 60);
+        for (x, _y, px) in img.enumerate_pixels_mut() {
+            *px = if x % 12 == 0 {
+                image::Rgba([0, 0, 0, 255])
+            } else {
+                image::Rgba([255, 255, 255, 0])
+            };
+        }
+        let mut png = Cursor::new(Vec::new());
+        DynamicImage::ImageRgba8(img)
+            .write_to(&mut png, ImageFormat::Png)
+            .unwrap();
+        let (bytes, mime) = reencode(&png.into_inner(), ImageProfile::STANDARD).unwrap();
+        assert_eq!(mime, "image/png");
+        let decoded = image::load_from_memory(&bytes).unwrap();
+        assert!(!decoded.color().has_alpha(), "transparency is flattened");
+        // Transparent pixels became white.
+        assert_eq!(
+            decoded.to_rgb8().get_pixel(1, 1),
+            &image::Rgb([255, 255, 255])
+        );
+    }
+}

src/images/fetch.rs

@@ -0,0 +1,180 @@
+//! Downloading an issue's images and packing them into EPUB assets.
+
+use std::time::Duration;
+
+use futures::StreamExt;
+
+use crate::types::{Edition, ImageAsset, Pick};
+
+use super::encode::{ImageProfile, reencode};
+use super::refs::{ImgRef, extract_img_refs};
+
+/// Per-image download timeout (§3.10).
+pub const DOWNLOAD_TIMEOUT_SECS: u64 = 10;
+/// Per-image compressed download cap (§3.10).
+///
+/// Source screenshots and lossless artwork can be much larger than the asset we
+/// ultimately embed. Decoded dimensions and allocations are capped separately
+/// by the encoder, and the original bytes are discarded after re-encoding.
+pub const MAX_IMAGE_BYTES: usize = 100 * 1024 * 1024;
+/// Concurrent downloads (§3.10).
+pub const CONCURRENCY: usize = 8;
+/// Whole-issue asset budget (§3.10).
+pub const ISSUE_ASSET_BUDGET_BYTES: usize = 25 * 1024 * 1024;
+
+fn exceeds_download_cap(bytes: usize) -> bool {
+    bytes > MAX_IMAGE_BYTES
+}
+
+/// Download one image, honoring the timeout and size cap (§3.10).
+pub async fn download(http: &reqwest::Client, url: &str) -> Option<Vec<u8>> {
+    let resp = http
+        .get(url)
+        // Some CDNs answer `Accept: */*` with an HTML interstitial (§3.10).
+        .header(reqwest::header::ACCEPT, "image/*,*/*;q=0.8")
+        .timeout(Duration::from_secs(DOWNLOAD_TIMEOUT_SECS))
+        .send()
+        .await
+        .map_err(|e| tracing::debug!(url, "image download failed: {e}"))
+        .ok()?;
+    if !resp.status().is_success() {
+        tracing::debug!(url, status = %resp.status(), "image download rejected");
+        return None;
+    }
+    if let Some(len) = resp.content_length()
+        && usize::try_from(len).map_or(true, exceeds_download_cap)
+    {
+        tracing::debug!(url, len, "image exceeds the size cap");
+        return None;
+    }
+    let mut resp = resp;
+    let mut buf: Vec<u8> = Vec::new();
+    loop {
+        match resp.chunk().await {
+            Ok(Some(chunk)) => {
+                if exceeds_download_cap(buf.len().saturating_add(chunk.len())) {
+                    tracing::debug!(url, "image exceeds the size cap mid-stream");
+                    return None;
+                }
+                buf.extend_from_slice(&chunk);
+            }
+            Ok(None) => break,
+            Err(e) => {
+                tracing::debug!(url, "image download interrupted: {e}");
+                return None;
+            }
+        }
+    }
+    if buf.is_empty() { None } else { Some(buf) }
+}
+
+/// Everything needed to fetch one image, in deterministic issue order.
+#[derive(Debug, Clone)]
+struct PendingImage {
+    id: String,
+    url: String,
+    alt: String,
+    caption: Option<String>,
+}
+
+fn pending_for_pick(pick: &Pick) -> Vec<PendingImage> {
+    let entry_id = pick.article.best_entry_id;
+    let mut refs = extract_img_refs(&pick.article.content_html);
+    if refs.is_empty() {
+        refs = pick
+            .article
+            .image_urls
+            .iter()
+            .map(|u| ImgRef {
+                src: u.clone(),
+                alt: String::new(),
+                caption: None,
+            })
+            .collect();
+    }
+    refs.into_iter()
+        .filter(|r| r.src.starts_with("http://") || r.src.starts_with("https://"))
+        .enumerate()
+        .map(|(i, r)| PendingImage {
+            id: format!("img-{entry_id}-{i}"),
+            url: r.src,
+            alt: r.alt,
+            caption: r.caption,
+        })
+        .collect()
+}
+
+/// Download and re-encode every image referenced by the lineup for one edition,
+/// respecting [`ISSUE_ASSET_BUDGET_BYTES`] (§3.10).
+pub async fn collect_for_issue(
+    http: &reqwest::Client,
+    picks: &[Pick],
+    edition: Edition,
+) -> Vec<ImageAsset> {
+    let profile = ImageProfile::for_edition(edition);
+    let pending: Vec<PendingImage> = picks.iter().flat_map(pending_for_pick).collect();
+    if pending.is_empty() {
+        return Vec::new();
+    }
+    tracing::info!(count = pending.len(), ?edition, "downloading issue images");
+
+    let results: Vec<Option<(PendingImage, Vec<u8>, &'static str)>> =
+        futures::stream::iter(pending.into_iter().map(|p| {
+            let http = http.clone();
+            async move {
+                let raw = download(&http, &p.url).await?;
+                let (bytes, mime) = tokio::task::spawn_blocking(move || reencode(&raw, profile))
+                    .await
+                    .ok()
+                    .flatten()?;
+                Some((p, bytes, mime))
+            }
+        }))
+        .buffered(CONCURRENCY)
+        .collect()
+        .await;
+
+    let mut assets = Vec::new();
+    let mut budget_used = 0usize;
+    let mut skipped = 0usize;
+    for result in results.into_iter().flatten() {
+        let (pending, bytes, mime) = result;
+        if budget_used + bytes.len() > ISSUE_ASSET_BUDGET_BYTES {
+            skipped += 1;
+            continue;
+        }
+        budget_used += bytes.len();
+        let ext = if mime == "image/png" { "png" } else { "jpg" };
+        assets.push(ImageAsset {
+            href: format!("images/{}.{ext}", pending.id),
+            id: pending.id,
+            mime: mime.to_string(),
+            data: bytes,
+            alt: pending.alt,
+            caption: pending.caption,
+            source_url: pending.url,
+        });
+    }
+    if skipped > 0 {
+        tracing::warn!(skipped, budget_used, "issue image budget exhausted");
+    }
+    tracing::info!(
+        embedded = assets.len(),
+        bytes = budget_used,
+        "issue images ready"
+    );
+    assets
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn compressed_download_cap_is_large_but_bounded() {
+        assert_eq!(MAX_IMAGE_BYTES, 100 * 1024 * 1024);
+        assert!(!exceeds_download_cap(MAX_IMAGE_BYTES));
+        assert!(exceeds_download_cap(MAX_IMAGE_BYTES + 1));
+        assert!(!exceeds_download_cap(5_790_082));
+    }
+}

src/images/mod.rs

@@ -0,0 +1,25 @@
+//! Article images, end to end (spec §3.3 extraction, §3.10 "Images").
+//!
+//! The stages run in this order, and each submodule owns one of them:
+//!
+//! | stage | module | when |
+//! |---|---|---|
+//! | make a page's `<img>` elements usable | [`normalize`] | extraction, before readability |
+//! | find what an article references | [`refs`] | extraction and issue build |
+//! | download and re-encode per edition | [`fetch`], [`encode`] | issue build |
+//! | point the markup at the embedded files | [`embed`] | chapter rendering |
+//!
+//! Nothing here ever fails a run: an image that cannot be fetched, decoded or
+//! resolved is dropped, and the article is rendered without it (notes §3).
+
+pub mod embed;
+pub mod encode;
+pub mod fetch;
+pub mod normalize;
+pub mod refs;
+
+pub use embed::rewrite_img_srcs;
+pub use encode::{ImageProfile, MIN_DIMENSION_PX, reencode};
+pub use fetch::{ISSUE_ASSET_BUDGET_BYTES, collect_for_issue, download};
+pub use normalize::{normalize_img_tags, prepare_for_readability, unwrap_image_wrappers};
+pub use refs::{ImgRef, collect_image_urls, extract_img_refs};

src/images/normalize.rs

@@ -0,0 +1,516 @@
+//! Making a page's `<img>` elements usable before anything else touches them.
+//!
+//! Publishers ship images in a dozen incompatible shapes: lazy placeholders,
+//! `srcset` lists, `<picture>` sources, URLs parked in `data-*` attributes, and
+//! `src` values that are not URLs at all (unfilled templates, JSON blobs, whole
+//! `srcset` strings). On top of that, readability actively damages images while
+//! it works — it deletes `<button>` subtrees, taking lightbox images with them,
+//! and its own lazy-image heuristic overwrites a working `src` with any
+//! attribute whose value happens to contain `.jpg`.
+//!
+//! [`prepare_for_readability`] runs before readability and leaves every `<img>`
+//! as a plain `src`/`alt`/`title` triple, which both fixes the input and denies
+//! readability the raw material for its substitution. [`normalize_img_tags`] is
+//! also applied to feed content, which carries the same markup.
+//!
+//! Candidates are judged by *shape*, never by publisher: a string with braces,
+//! whitespace or quotes in it cannot resolve, whoever wrote it.
+
+use crate::html::{html_to_text, parse_attrs, tag_end, tag_name, truncate_utf8};
+
+/// Elements that readability deletes outright, and which a page may nevertheless
+/// have wrapped around an image (lightbox triggers, mostly).
+const IMAGE_WRAPPER_TAGS: &[&str] = &["button", "form", "fieldset", "object"];
+
+/// Attributes lazy-loading libraries use for the real image URL.
+///
+/// These outrank `src`, because a page only sets them when `src` is a stand-in:
+/// a transparent GIF, a blurred thumbnail, an inline SVG spacer. Attributes that
+/// merely *look* image-ish (`data-template`, `data-attrs`, `data-orig-file`) are
+/// deliberately absent — those hold templates and metadata, and preferring them
+/// is exactly the mistake readability's own heuristic makes.
+const LAZY_SRC_ATTRS: &[&str] = &[
+    "data-src",
+    "data-lazy-src",
+    "data-original",
+    "data-runner-src",
+    "data-full-src",
+    "data-hi-res-src",
+    "data-image-src",
+];
+
+/// Widest `srcset` candidate we will pick; above this we are downloading pixels
+/// the re-encoder immediately throws away.
+const MAX_SRCSET_WIDTH: u32 = 2000;
+
+/// Make a fetched page safe to hand to readability (§3.3).
+///
+/// Two passes, both about images: unwrap the elements that would take an image
+/// with them when readability deletes them, then reduce every `<img>` to a plain
+/// `src`/`alt`/`title` triple. The second pass is what stops readability's own
+/// lazy-image heuristic from replacing a working `src` — with no `srcset`,
+/// `loading` or `data-*` attributes left on the element, it has nothing to
+/// substitute and leaves the image alone.
+pub fn prepare_for_readability(html: &str) -> String {
+    normalize_img_tags(&unwrap_image_wrappers(html))
+}
+
+/// Replace image-only `<button>`/`<form>`/`<fieldset>`/`<object>` wrappers with
+/// their contents (§3.3).
+///
+/// A `<button>` holding nothing but an image is a lightbox trigger, not a
+/// control: the image is the content. Wrappers that also carry text are left
+/// alone, because those really are interface.
+pub fn unwrap_image_wrappers(html: &str) -> String {
+    let mut out = String::with_capacity(html.len());
+    let mut cursor = 0usize;
+    while let Some(rel) = html[cursor..].find('<') {
+        let start = cursor + rel;
+        out.push_str(&html[cursor..start]);
+        let Some(end) = tag_end(html, start) else {
+            out.push_str(&html[start..]);
+            return out;
+        };
+        let raw = &html[start..end];
+        let inner = raw.trim_start_matches('<').trim_end_matches('>');
+        let name = tag_name(inner);
+
+        if IMAGE_WRAPPER_TAGS.contains(&name.as_str())
+            && !inner.trim_end().ends_with('/')
+            && let Some((content, after)) = element_content(html, end, &name)
+            && content.contains("<img")
+            && html_to_text(content).trim().is_empty()
+        {
+            out.push_str(&unwrap_image_wrappers(content));
+            cursor = after;
+            continue;
+        }
+
+        out.push_str(raw);
+        cursor = end;
+    }
+    out.push_str(&html[cursor..]);
+    out
+}
+
+/// The content of an element whose open tag ended at `body_start`, plus the
+/// offset just past its close tag. `None` when the element is never closed.
+fn element_content<'a>(html: &'a str, body_start: usize, name: &str) -> Option<(&'a str, usize)> {
+    let open = format!("<{name}");
+    let close = format!("</{name}");
+    let mut depth = 1usize;
+    let mut cursor = body_start;
+    while let Some(rel) = html[cursor..].find('<') {
+        let start = cursor + rel;
+        let end = tag_end(html, start)?;
+        let tag = &html[start..end];
+        let lower = tag.to_ascii_lowercase();
+        if lower.starts_with(&close) {
+            depth -= 1;
+            if depth == 0 {
+                return Some((&html[body_start..start], end));
+            }
+        } else if lower.starts_with(&open)
+            && !lower[open.len()..].starts_with(|c: char| c.is_alphanumeric() || c == '-')
+            && !tag.trim_end().ends_with("/>")
+        {
+            depth += 1;
+        }
+        cursor = end;
+    }
+    None
+}
+
+/// Reduce every `<img>` to `<img src alt title>` with a usable URL (§3.3).
+///
+/// The `src` a page ships is not automatically the one to use: it can be a lazy
+/// placeholder (`data:image/svg+xml,…`), an unresolved template
+/// (`…/resize/{width}/…`), a JSON blob a framework parked there, or an entire
+/// `srcset` string. Candidates are tried in order and the first plausible one
+/// wins; an image with no plausible candidate is dropped, because a broken
+/// `<img>` only becomes clutter further down the pipeline.
+pub fn normalize_img_tags(html: &str) -> String {
+    let mut out = String::with_capacity(html.len());
+    let mut cursor = 0usize;
+    // `<picture>` puts the real candidates on sibling `<source>` elements.
+    let mut picture_srcset: Option<String> = None;
+
+    while let Some(rel) = html[cursor..].find('<') {
+        let start = cursor + rel;
+        out.push_str(&html[cursor..start]);
+        let Some(end) = tag_end(html, start) else {
+            out.push_str(&html[start..]);
+            return out;
+        };
+        let raw = &html[start..end];
+        let inner = raw
+            .trim_start_matches('<')
+            .trim_end_matches('>')
+            .trim_end_matches('/');
+        let name = tag_name(inner);
+
+        match name.as_str() {
+            "picture" => {
+                picture_srcset = None;
+                out.push_str(raw);
+            }
+            "source" => {
+                let attrs = parse_attrs(inner);
+                if picture_srcset.is_none()
+                    && let Some(set) =
+                        attr(&attrs, "srcset").or_else(|| attr(&attrs, "data-srcset"))
+                {
+                    picture_srcset = Some(set.to_string());
+                }
+                out.push_str(raw);
+            }
+            "img" => {
+                let attrs = parse_attrs(inner);
+                if let Some(src) = best_img_src(&attrs, picture_srcset.as_deref()) {
+                    out.push_str("<img src=\"");
+                    out.push_str(&escape_attr(&src));
+                    out.push('"');
+                    for key in ["alt", "title"] {
+                        if let Some(v) = attr(&attrs, key) {
+                            out.push(' ');
+                            out.push_str(key);
+                            out.push_str("=\"");
+                            out.push_str(&escape_attr(v));
+                            out.push('"');
+                        }
+                    }
+                    out.push_str("/>");
+                } else {
+                    tracing::debug!(tag = %truncate_utf8(raw, 120), "dropping unusable img");
+                }
+            }
+            _ => out.push_str(raw),
+        }
+        if name == "picture" && inner.starts_with('/') {
+            picture_srcset = None;
+        }
+        cursor = end;
+    }
+    out.push_str(&html[cursor..]);
+    out
+}
+
+fn attr<'a>(attrs: &'a [(String, String)], name: &str) -> Option<&'a str> {
+    attrs
+        .iter()
+        .find(|(k, _)| k == name)
+        .map(|(_, v)| v.trim())
+        .filter(|v| !v.is_empty())
+}
+
+/// Pick the best URL for one `<img>` from everything the element carries.
+fn best_img_src(attrs: &[(String, String)], picture_srcset: Option<&str>) -> Option<String> {
+    for key in LAZY_SRC_ATTRS {
+        if let Some(v) = attr(attrs, key).filter(|s| plausible_url(s)) {
+            return Some(v.to_string());
+        }
+    }
+    if let Some(src) = attr(attrs, "src").filter(|s| plausible_url(s)) {
+        return Some(src.to_string());
+    }
+    if let Some(from_set) = attr(attrs, "srcset").and_then(best_from_srcset) {
+        return Some(from_set);
+    }
+    if let Some(from_set) = attr(attrs, "data-srcset").and_then(best_from_srcset) {
+        return Some(from_set);
+    }
+    picture_srcset.and_then(best_from_srcset)
+}
+
+/// The widest candidate in a `srcset` that is still worth downloading.
+///
+/// Parsed by whitespace rather than by comma: the URLs of several image CDNs
+/// contain commas of their own, and splitting on those shreds them.
+fn best_from_srcset(srcset: &str) -> Option<String> {
+    let mut best: Option<(u32, String)> = None;
+    let mut smallest: Option<(u32, String)> = None;
+    let mut pending: Option<String> = None;
+
+    let mut consider = |url: String, width: u32| {
+        if width <= MAX_SRCSET_WIDTH && best.as_ref().is_none_or(|(w, _)| width > *w) {
+            best = Some((width, url.clone()));
+        }
+        if smallest.as_ref().is_none_or(|(w, _)| width < *w) {
+            smallest = Some((width, url));
+        }
+    };
+
+    for token in srcset.split_whitespace() {
+        let token = token.trim_end_matches(',');
+        if token.is_empty() {
+            continue;
+        }
+        match parse_descriptor(token) {
+            Some(width) => {
+                if let Some(url) = pending.take() {
+                    consider(url, width);
+                }
+            }
+            None => {
+                // A URL with no descriptor of its own still counts, at width 1x.
+                if let Some(url) = pending.replace(token.to_string()) {
+                    consider(url, 1);
+                }
+            }
+        }
+    }
+    if let Some(url) = pending.take() {
+        consider(url, 1);
+    }
+
+    best.or(smallest)
+        .map(|(_, url)| url)
+        .filter(|u| plausible_url(u))
+}
+
+/// `800w` → 800, `2x` → a synthetic width so density candidates sort sensibly.
+fn parse_descriptor(token: &str) -> Option<u32> {
+    let (value, unit) = token.split_at(token.len().checked_sub(1)?);
+    match unit {
+        "w" => value.parse::<u32>().ok(),
+        "x" => value
+            .parse::<f32>()
+            .ok()
+            .map(|d| (d * 1000.0).round().clamp(1.0, f32::from(u16::MAX)) as u32),
+        _ => None,
+    }
+}
+
+/// Whether a string can serve as an image URL at all.
+///
+/// This is deliberately about shape, not about the host: whitespace, braces and
+/// quotes mean we are looking at a `srcset` blob, an unfilled URL template or a
+/// serialized object, none of which will ever resolve.
+fn plausible_url(candidate: &str) -> bool {
+    let candidate = candidate.trim();
+    if candidate.is_empty() || candidate.len() > 2048 {
+        return false;
+    }
+    if candidate.starts_with("data:") || candidate.starts_with("about:") {
+        return false;
+    }
+    if candidate
+        .chars()
+        .any(|c| c.is_whitespace() || matches!(c, '{' | '}' | '"' | '\'' | '<' | '>' | '\\'))
+    {
+        return false;
+    }
+    // `%20` is a space that survived encoding — same blob, different spelling.
+    !candidate.contains("%20")
+}
+
+fn escape_attr(s: &str) -> String {
+    s.replace('&', "&amp;")
+        .replace('<', "&lt;")
+        .replace('>', "&gt;")
+        .replace('"', "&quot;")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use scraper::{Html, Selector};
+
+    /// The `src` values that survive normalization, read back with a real parser.
+    fn src_of(html: &str) -> Vec<String> {
+        Html::parse_fragment(html)
+            .select(&Selector::parse("img").unwrap())
+            .filter_map(|e| e.value().attr("src").map(str::to_string))
+            .collect()
+    }
+
+    #[test]
+    fn lazy_placeholder_src_gives_way_to_the_real_url() {
+        // IEEE Spectrum: an inline SVG spacer with the URL parked on data-runner-src.
+        let html = r#"<img alt="A well head" lazy-loadable="true"
+            src="data:image/svg+xml,%3Csvg%20xmlns=%27http://www.w3.org/2000/svg%27%3E%3C/svg%3E"
+            data-runner-src="https://spectrum.ieee.org/media-library/well.jpg?id=675&amp;width=980"/>"#;
+        assert_eq!(
+            src_of(&normalize_img_tags(html)),
+            ["https://spectrum.ieee.org/media-library/well.jpg?id=675&width=980"]
+        );
+    }
+
+    #[test]
+    fn unresolved_url_templates_fall_through_to_a_real_candidate() {
+        // NPR: readability copies data-template over a perfectly good src.
+        let html = r#"<img alt="Meghan Cliffel"
+            src="https://npr.brightspotcdn.com/resize/{width}/quality/{quality}/x.jpg"
+            srcset="https://npr.brightspotcdn.com/resize/1100/quality/50/x.jpg 1100w"/>"#;
+        assert_eq!(
+            src_of(&normalize_img_tags(html)),
+            ["https://npr.brightspotcdn.com/resize/1100/quality/50/x.jpg"]
+        );
+        // With nothing usable anywhere, the image goes rather than becoming a
+        // request for a picture that says "Image".
+        let only_template =
+            r#"<img alt="x" src="https://cdn.dev/resize/{width}/quality/{quality}/x.jpg"/>"#;
+        assert!(src_of(&normalize_img_tags(only_template)).is_empty());
+    }
+
+    #[test]
+    fn a_srcset_blob_parked_in_src_is_rejected_and_reparsed() {
+        // dfarq: readability copies the entire srcset string into src.
+        let blob = "https://i0.wp.com/x.jpg?resize=300%2C158&ssl=1 300w, \
+                    https://i0.wp.com/x.jpg?resize=1024%2C540&ssl=1 1024w, \
+                    https://i0.wp.com/x.jpg?w=3000&ssl=1 3000w";
+        let html = format!(r#"<img alt="printer" src="{blob}" srcset="{blob}"/>"#);
+        // The widest candidate under the download ceiling wins; 3000w does not.
+        assert_eq!(
+            src_of(&normalize_img_tags(&html)),
+            ["https://i0.wp.com/x.jpg?resize=1024%2C540&ssl=1"]
+        );
+    }
+
+    #[test]
+    fn a_json_blob_parked_in_src_is_rejected() {
+        // Substack: readability copies data-attrs (JSON) over src.
+        let html = r#"<img alt="" src="{&quot;src&quot;:&quot;https://s3.dev/a.jpeg&quot;,&quot;width&quot;:1000}"
+            srcset="https://substackcdn.com/image/fetch/$s_!y1,w_1456,c_limit/a.jpeg 1456w"/>"#;
+        assert_eq!(
+            src_of(&normalize_img_tags(html)),
+            ["https://substackcdn.com/image/fetch/$s_!y1,w_1456,c_limit/a.jpeg"]
+        );
+    }
+
+    #[test]
+    fn picture_sources_back_up_an_empty_img() {
+        let html = r#"<picture>
+            <source srcset="https://cdn.dev/a.avif 800w" type="image/avif"/>
+            <source srcset="https://cdn.dev/a.webp 800w" type="image/webp"/>
+            <img alt="A photo"/>
+        </picture>"#;
+        assert_eq!(
+            src_of(&normalize_img_tags(html)),
+            ["https://cdn.dev/a.avif"]
+        );
+    }
+
+    #[test]
+    fn a_good_src_is_never_traded_for_a_metadata_attribute() {
+        // `data-template`, `data-attrs` and friends hold templates and JSON, not
+        // URLs — trading a working src for one of those is the original sin.
+        let html = r#"<img alt="A photo" loading="lazy" class="lazyload"
+            src="https://cdn.dev/real.jpg"
+            data-template="https://cdn.dev/{width}/real.jpg"
+            data-orig-file="https://cdn.dev/orig.jpg"/>"#;
+        assert_eq!(
+            src_of(&normalize_img_tags(html)),
+            ["https://cdn.dev/real.jpg"]
+        );
+    }
+
+    #[test]
+    fn a_lazy_loader_placeholder_loses_to_its_data_src() {
+        // iRunFar (a3-lazy-load): src is a shared 1×1 spacer that would download
+        // and re-encode perfectly happily, and be the wrong picture.
+        let html = r#"<img class="lazy lazy-hidden" alt="Brooks Cascadia 20"
+            src="//www.irunfar.com/wp-content/plugins/a3-lazy-load/assets/images/lazy_placeholder.gif"
+            data-src="https://s3.amazonaws.com/www.irunfar.com/uploads/Brooks-Cascadia-20.jpg"/>"#;
+        assert_eq!(
+            src_of(&normalize_img_tags(html)),
+            ["https://s3.amazonaws.com/www.irunfar.com/uploads/Brooks-Cascadia-20.jpg"]
+        );
+    }
+
+    #[test]
+    fn normalization_keeps_alt_and_title_and_drops_the_rest() {
+        let html = r#"<img src="/a.png" alt="An &amp; alt" title="T" class="x" width="900"
+            onerror="evil()" srcset="/b.png 2x"/>"#;
+        let out = normalize_img_tags(html);
+        assert!(out.contains(r#"alt="An &amp; alt""#), "{out}");
+        assert!(out.contains(r#"title="T""#), "{out}");
+        for gone in ["class=", "width=", "onerror", "srcset="] {
+            assert!(
+                !out.contains(gone),
+                "expected {gone} to be dropped from {out}"
+            );
+        }
+        // Relative URLs survive: sanitize_with_base absolutizes them later.
+        assert_eq!(src_of(&out), ["/a.png"]);
+    }
+
+    #[test]
+    fn lightbox_buttons_no_longer_take_their_image_with_them() {
+        // Nautilus: readability deletes <button> and everything inside it.
+        let html = r#"<figure class="wp-block-image">
+            <button type="button" aria-label="Enlarge image">
+                <img class="wp-image-1" src="https://cdn.dev/flower.png?w=710" alt=""/>
+            </button>
+            <figcaption>BEAUTIFUL DANGER: a belladonna flower.</figcaption>
+        </figure>"#;
+        let out = unwrap_image_wrappers(html);
+        assert!(!out.contains("<button"), "{out}");
+        assert!(!out.contains("</button>"), "{out}");
+        assert!(out.contains("flower.png"), "{out}");
+        assert!(out.contains("BEAUTIFUL DANGER"), "caption survives: {out}");
+    }
+
+    #[test]
+    fn buttons_that_are_really_buttons_are_left_alone() {
+        let html = r#"<button class="subscribe">Subscribe <img src="/icon.png" alt=""/></button>"#;
+        assert_eq!(unwrap_image_wrappers(html), html);
+        // And an image-free control is untouched too.
+        let plain = r#"<button>Share</button>"#;
+        assert_eq!(unwrap_image_wrappers(plain), plain);
+    }
+
+    #[test]
+    fn nested_and_unclosed_wrappers_do_not_derail_the_scan() {
+        let nested = r#"<button><button><img src="/a.png"/></button></button><p>after</p>"#;
+        let out = unwrap_image_wrappers(nested);
+        assert!(!out.contains("button"), "{out}");
+        assert!(out.contains("/a.png") && out.contains("after"), "{out}");
+        // An open tag that never closes is passed through rather than eating
+        // the rest of the document.
+        let unclosed = r#"<button><img src="/a.png"/><p>rest</p>"#;
+        assert!(unwrap_image_wrappers(unclosed).contains("rest"));
+    }
+
+    #[test]
+    fn srcset_descriptors_pick_the_widest_usable_candidate() {
+        assert_eq!(
+            best_from_srcset("/a.png 150w, /b.png 800w, /c.png 4000w").as_deref(),
+            Some("/b.png")
+        );
+        // Density descriptors work as an ordering too.
+        assert_eq!(
+            best_from_srcset("/a.png 1x, /b.png 2x").as_deref(),
+            Some("/b.png")
+        );
+        // A bare URL with no descriptor is still a candidate.
+        assert_eq!(best_from_srcset("/only.png").as_deref(), Some("/only.png"));
+        // Every candidate too wide: take the narrowest rather than nothing.
+        assert_eq!(
+            best_from_srcset("/big.png 3000w, /huge.png 5000w").as_deref(),
+            Some("/big.png")
+        );
+        assert_eq!(best_from_srcset("   ").as_deref(), None);
+    }
+
+    #[test]
+    fn plausibility_is_about_shape_not_host() {
+        for good in [
+            "https://cdn.dev/a.jpg?id=1&width=980",
+            "/_next/image?url=https%3A%2F%2Fx.dev%2Fa.png&w=3840&q=75",
+            "https://substackcdn.com/image/fetch/$s_!y1,w_1456/a.jpeg",
+        ] {
+            assert!(plausible_url(good), "{good} should be usable");
+        }
+        for bad in [
+            "",
+            "data:image/svg+xml,%3Csvg%3E%3C/svg%3E",
+            "https://cdn.dev/resize/{width}/a.jpg",
+            r#"{"src":"https://cdn.dev/a.jpg"}"#,
+            "https://cdn.dev/a.jpg 300w, https://cdn.dev/b.jpg 600w",
+            "https://cdn.dev/a.jpg%20300w",
+        ] {
+            assert!(!plausible_url(bad), "{bad} should be rejected");
+        }
+    }
+}

src/images/refs.rs

@@ -0,0 +1,156 @@
+//! Finding the images an article references.
+//!
+//! The input here is already-normalized markup (see [`super::normalize`]), so
+//! every `<img>` is expected to carry a plain, usable `src`.
+
+use std::collections::HashSet;
+
+use scraper::{Html, Selector};
+use url::Url;
+
+/// One `<img>` found in article markup, with the caption of its `<figure>` if any.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ImgRef {
+    pub src: String,
+    pub alt: String,
+    pub caption: Option<String>,
+}
+
+/// Collect `<img>` references (src, alt, enclosing figcaption) from article markup.
+pub fn extract_img_refs(html: &str) -> Vec<ImgRef> {
+    let doc = scraper::Html::parse_fragment(html);
+    let Ok(img_sel) = scraper::Selector::parse("img") else {
+        return Vec::new();
+    };
+    let cap_sel = scraper::Selector::parse("figcaption").ok();
+
+    let mut out = Vec::new();
+    let mut seen: Vec<String> = Vec::new();
+    for el in doc.select(&img_sel) {
+        let Some(src) = el.value().attr("src") else {
+            continue;
+        };
+        let src = src.trim();
+        if src.is_empty() || src.starts_with("data:") {
+            continue;
+        }
+        if seen.iter().any(|s| s == src) {
+            continue;
+        }
+        seen.push(src.to_string());
+        let alt = el
+            .value()
+            .attr("alt")
+            .unwrap_or_default()
+            .trim()
+            .to_string();
+        // Walk up to an enclosing <figure> and take its caption, if any.
+        let mut caption = None;
+        if let Some(cap_sel) = &cap_sel {
+            let mut cursor = el.parent();
+            while let Some(node) = cursor {
+                if let Some(elem) = scraper::ElementRef::wrap(node) {
+                    if elem.value().name() == "figure" {
+                        caption = elem.select(cap_sel).next().map(|c| {
+                            c.text()
+                                .collect::<String>()
+                                .split_whitespace()
+                                .collect::<Vec<_>>()
+                                .join(" ")
+                        });
+                        break;
+                    }
+                    cursor = elem.parent();
+                } else {
+                    break;
+                }
+            }
+        }
+        out.push(ImgRef {
+            src: src.to_string(),
+            alt,
+            caption: caption.filter(|c| !c.is_empty()),
+        });
+    }
+    out
+}
+
+/// Absolute image URLs referenced by `html`, resolved against `base_url` (§3.3).
+///
+/// Every image an article carries is kept: a photo essay with thirty pictures is
+/// a photo essay, and the issue-wide byte budget is the real backstop.
+pub fn collect_image_urls(html: &str, base_url: &str) -> Vec<String> {
+    let Ok(selector) = Selector::parse("img") else {
+        return Vec::new();
+    };
+    let base = Url::parse(base_url).ok();
+    let document = Html::parse_fragment(html);
+    let mut seen: HashSet<String> = HashSet::new();
+    let mut out: Vec<String> = Vec::new();
+    for element in document.select(&selector) {
+        let raw = element
+            .value()
+            .attr("src")
+            .or_else(|| element.value().attr("data-src"))
+            .map(str::trim)
+            .filter(|s| !s.is_empty());
+        let Some(raw) = raw else { continue };
+        let resolved = match Url::parse(raw) {
+            Ok(u) => Some(u),
+            Err(_) => base.as_ref().and_then(|b| b.join(raw).ok()),
+        };
+        let Some(url) = resolved.filter(|u| matches!(u.scheme(), "http" | "https")) else {
+            continue;
+        };
+        let url = url.to_string();
+        if seen.insert(url.clone()) {
+            out.push(url);
+        }
+    }
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn image_collection_resolves_and_keeps_every_image() {
+        let mut html = String::from(r#"<img src="/a.png"><img src="https://cdn.dev/b.png">"#);
+        html.push_str(r#"<img data-src="c.png"><img src="/a.png"><img src="data:image/png;x">"#);
+        for i in 0..20 {
+            html.push_str(&format!(r#"<img src="/n{i}.png">"#));
+        }
+        let urls = collect_image_urls(&html, "https://blog.dev/posts/one");
+        // Two named images, the data-src one, and all twenty of the rest: no cap.
+        assert_eq!(urls.len(), 23);
+        assert_eq!(urls[0], "https://blog.dev/a.png");
+        assert_eq!(urls[1], "https://cdn.dev/b.png");
+        assert_eq!(urls[2], "https://blog.dev/posts/c.png");
+        // Duplicates and data: URIs never appear.
+        assert_eq!(urls.iter().filter(|u| u.ends_with("/a.png")).count(), 1);
+        assert!(!urls.iter().any(|u| u.starts_with("data:")));
+        assert!(collect_image_urls("<p>none</p>", "https://blog.dev").is_empty());
+    }
+
+    // -----------------------------------------------------------------------
+    // Image normalization — each case is a page shape seen in a real issue.
+    // -----------------------------------------------------------------------
+
+    #[test]
+    fn extracts_img_refs_with_captions() {
+        let html = r#"<p>hi</p>
+            <figure><img src="https://e.g/a.png" alt="A diagram"/>
+            <figcaption>Figure 1:  the thing</figcaption></figure>
+            <img src="https://e.g/b.jpg"/>
+            <img src="data:image/png;base64,zz"/>
+            <img src="https://e.g/a.png" alt="dupe"/>"#;
+        let refs = extract_img_refs(html);
+        assert_eq!(refs.len(), 2);
+        assert_eq!(refs[0].src, "https://e.g/a.png");
+        assert_eq!(refs[0].alt, "A diagram");
+        assert_eq!(refs[0].caption.as_deref(), Some("Figure 1: the thing"));
+        assert_eq!(refs[1].src, "https://e.g/b.jpg");
+        assert!(refs[1].caption.is_none());
+    }
+}

src/lib.rs

@@ -20,7 +20,10 @@ pub mod db;
 pub mod dedupe;
 pub mod epub;
 pub mod extract;
+pub mod html;
 pub mod http;
+pub mod images;
+pub mod lock;
 pub mod miniflux;
 pub mod pipeline;
 pub mod publish;

src/lock.rs

@@ -0,0 +1,168 @@
+//! One writer at a time (plan §5): an advisory `flock(LOCK_EX | LOCK_NB)` on
+//! `<database_path>.lock`, taken in `main` for `generate`, `profile rebuild`,
+//! `features backfill` and `backfill-social`. No table, no TTL: the kernel
+//! releases the lock when the holder exits, however it exits.
+
+use std::fs::{File, OpenOptions};
+use std::io::{self, Write as _};
+use std::os::fd::AsRawFd;
+use std::path::{Path, PathBuf};
+
+/// Held for as long as the value lives; dropping it (or dying) releases it.
+#[derive(Debug)]
+pub struct RunLock {
+    _file: File,
+    pub path: PathBuf,
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum LockError {
+    /// Another process holds the lock; `holder` names its command when known.
+    #[error("{holder} is already running")]
+    Held { holder: String, path: PathBuf },
+    #[error("lock file {path}: {source}")]
+    Io {
+        path: PathBuf,
+        #[source]
+        source: io::Error,
+    },
+}
+
+/// `<database_path>.lock`, next to the database.
+pub fn lock_path(database_path: &Path) -> PathBuf {
+    let mut name = database_path.file_name().unwrap_or_default().to_os_string();
+    name.push(".lock");
+    database_path.with_file_name(name)
+}
+
+/// Take the lock for `command`, writing the command's name into the file so a
+/// second invocation can say who is holding it.
+pub fn acquire(database_path: &Path, command: &str) -> Result<RunLock, LockError> {
+    let path = lock_path(database_path);
+    let io_err = |source: io::Error| LockError::Io {
+        path: path.clone(),
+        source,
+    };
+    if let Some(parent) = path.parent().filter(|p| !p.as_os_str().is_empty()) {
+        std::fs::create_dir_all(parent).map_err(io_err)?;
+    }
+    let mut file = OpenOptions::new()
+        .read(true)
+        .write(true)
+        .create(true)
+        .truncate(false)
+        .open(&path)
+        .map_err(io_err)?;
+    // SAFETY: `file` owns a valid open descriptor for the duration of the call.
+    let rc = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) };
+    if rc != 0 {
+        let source = io::Error::last_os_error();
+        if source.kind() == io::ErrorKind::WouldBlock {
+            let holder = std::fs::read_to_string(&path)
+                .ok()
+                .map(|s| s.trim().to_string())
+                .filter(|s| !s.is_empty())
+                .unwrap_or_else(|| "generate".to_string());
+            return Err(LockError::Held { holder, path });
+        }
+        return Err(io_err(source));
+    }
+    // Best effort: the name is a courtesy for the error message, never load-bearing.
+    let _ = file.set_len(0);
+    let _ = writeln!(file, "{command}");
+    Ok(RunLock { _file: file, path })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::process::{Command, Stdio};
+    use std::time::{Duration, Instant};
+
+    const HELPER_ENV: &str = "DAILY_EPUB_LOCK_TEST_HOLDER";
+
+    #[test]
+    fn lock_path_sits_next_to_the_database() {
+        assert_eq!(
+            lock_path(Path::new("/var/lib/daily-epub/daily-epub.db")),
+            PathBuf::from("/var/lib/daily-epub/daily-epub.db.lock")
+        );
+        assert_eq!(lock_path(Path::new("x.db")), PathBuf::from("x.db.lock"));
+    }
+
+    #[test]
+    fn second_acquire_in_the_same_process_loses_and_names_the_holder() {
+        let dir = tempfile::tempdir().unwrap();
+        let db = dir.path().join("nested").join("daily-epub.db");
+        let first = acquire(&db, "generate").expect("first lock");
+        assert!(first.path.exists(), "the lock file is created on demand");
+        match acquire(&db, "profile rebuild") {
+            Err(LockError::Held { holder, .. }) => {
+                assert_eq!(holder, "generate");
+                assert_eq!(
+                    LockError::Held {
+                        holder,
+                        path: first.path.clone()
+                    }
+                    .to_string(),
+                    "generate is already running"
+                );
+            }
+            other => panic!("expected the lock to be held, got {other:?}"),
+        }
+        drop(first);
+        let again = acquire(&db, "backfill-social").expect("released on drop");
+        assert_eq!(
+            std::fs::read_to_string(&again.path).unwrap().trim(),
+            "backfill-social"
+        );
+    }
+
+    /// Not a test of its own: when `HELPER_ENV` names a database path, this
+    /// body takes the lock and holds it until it is killed. The parent below
+    /// spawns the test binary with that variable set.
+    #[test]
+    fn lock_holder_helper() {
+        let Ok(db) = std::env::var(HELPER_ENV) else {
+            return;
+        };
+        let _held = acquire(Path::new(&db), "helper").expect("helper takes the lock");
+        std::thread::sleep(Duration::from_secs(30));
+    }
+
+    #[test]
+    fn a_killed_holder_frees_the_lock_for_the_next_process() {
+        let dir = tempfile::tempdir().unwrap();
+        let db = dir.path().join("daily-epub.db");
+        let mut child = Command::new(std::env::current_exe().unwrap())
+            .args(["--exact", "lock::tests::lock_holder_helper", "--nocapture"])
+            .env(HELPER_ENV, &db)
+            .stdout(Stdio::null())
+            .stderr(Stdio::null())
+            .spawn()
+            .expect("spawn the holder process");
+
+        // Wait (well under a second) until the child has written its name.
+        let path = lock_path(&db);
+        let deadline = Instant::now() + Duration::from_millis(900);
+        while std::fs::read_to_string(&path)
+            .map(|s| s.trim() != "helper")
+            .unwrap_or(true)
+        {
+            assert!(Instant::now() < deadline, "the helper never took the lock");
+            std::thread::sleep(Duration::from_millis(10));
+        }
+        match acquire(&db, "generate") {
+            Err(LockError::Held { holder, .. }) => assert_eq!(holder, "helper"),
+            other => panic!("another process holds the lock, got {other:?}"),
+        }
+
+        child.kill().expect("kill the holder");
+        child.wait().expect("reap the holder");
+        let lock = acquire(&db, "generate").expect("the kernel released the dead holder's lock");
+        assert_eq!(
+            std::fs::read_to_string(&lock.path).unwrap().trim(),
+            "generate"
+        );
+    }
+}

src/main.rs

@@ -3,17 +3,21 @@
 //! Everything of substance lives in the library (`src/lib.rs`); this binary only
 //! parses flags, loads config, opens the database and dispatches.
 
+use std::io::Write as _;
 use std::path::PathBuf;
 
 use anyhow::{Context, Result};
-use clap::{Parser, Subcommand};
+use clap::{Parser, Subcommand, ValueEnum};
 use tracing_subscriber::EnvFilter;
 
 use daily_epub::config::Config;
+use daily_epub::curate::embedding::{self, BACKFILL_CONFIRM_TOKENS};
+use daily_epub::curate::telemetry;
 use daily_epub::db::Db;
 use daily_epub::pipeline::{self, GenerateOptions, GenerateOutcome};
-use daily_epub::report::RunReport;
+use daily_epub::report::{RunReport, VOYAGE_PROVIDER};
-use daily_epub::{curate, http, server, social};
+use daily_epub::types::{ArticleId, RatingEvent, Vote};
+use daily_epub::{curate, http, lock, server, social};
 
 /// A personalized daily newspaper, delivered as an EPUB.
 #[derive(Debug, Parser)]
@@ -36,11 +40,30 @@ enum Command {
     /// Taste-profile maintenance.
     #[command(subcommand)]
     Profile(ProfileCommand),
+    /// Inspect and edit explicit article verdicts.
+    #[command(subcommand)]
+    Ratings(RatingsCommand),
+    /// Why an article was (not) in the paper, from persisted run telemetry.
+    Explain(ExplainArgs),
+    /// The weekly numbers: issues, ratings, retriever yield, cost, timing.
+    Stats(StatsArgs),
+    /// Embedding cache and telemetry maintenance.
+    #[command(subcommand)]
+    Features(FeaturesCommand),
     /// Re-poll social scores for recent entries.
     BackfillSocial(BackfillSocialArgs),
     /// Database maintenance.
     #[command(subcommand)]
     Db(DbCommand),
+    /// Inspect the resolved configuration.
+    #[command(subcommand)]
+    Config(ConfigCommand),
+}
+
+#[derive(Debug, Subcommand)]
+enum ConfigCommand {
+    /// Load and validate the config as `generate` would, then print one fact per line.
+    Check,
 }
 
 #[derive(Debug, clap::Args)]
@@ -57,9 +80,15 @@ struct GenerateArgs {
     /// Cap the lineup size (overrides `target_article_count`).
     #[arg(long, value_name = "N")]
     max_articles: Option<usize>,
-    /// Skip every LLM call: prefilter order selects, excerpts stand in for summaries.
+    /// Skip every LLM call: cheap-signal admission, excerpt summaries.
     #[arg(long)]
     skip_llm: bool,
+    /// Use cached embeddings only: zero Voyage calls.
+    #[arg(long)]
+    skip_embeddings: bool,
+    /// Ignore reusable triage/deep assessments and ask the bulk model again.
+    #[arg(long)]
+    rescore: bool,
 }
 
 impl From<&GenerateArgs> for GenerateOptions {
@@ -70,6 +99,8 @@ impl From<&GenerateArgs> for GenerateOptions {
             out: args.out.clone(),
             max_articles: args.max_articles,
             skip_llm: args.skip_llm,
+            skip_embeddings: args.skip_embeddings,
+            rescore: args.rescore,
         }
     }
 }
@@ -80,6 +111,145 @@ enum ProfileCommand {
     Rebuild,
 }
 
+#[derive(Debug, Subcommand)]
+enum RatingsCommand {
+    /// List current explicit ratings, newest first.
+    List(RatingsListArgs),
+    /// Set or correct an article's explicit rating.
+    Set(RatingsSetArgs),
+    /// Clear an article from the learned rating set.
+    Clear(RatingsClearArgs),
+}
+
+#[derive(Debug, Clone, Copy, ValueEnum)]
+enum RatingListLabel {
+    Loved,
+    Good,
+    Down,
+    Cleared,
+}
+
+impl RatingListLabel {
+    fn event_label(self) -> &'static str {
+        match self {
+            Self::Loved => "loved",
+            Self::Good => "good",
+            Self::Down => "not_for_me",
+            Self::Cleared => "cleared",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, ValueEnum)]
+enum RatingSetLabel {
+    Loved,
+    Good,
+    Down,
+}
+
+impl RatingSetLabel {
+    fn vote(self) -> Vote {
+        match self {
+            Self::Loved => Vote::Loved,
+            Self::Good => Vote::Good,
+            Self::Down => Vote::NotForMe,
+        }
+    }
+}
+
+#[derive(Debug, clap::Args)]
+struct RatingsListArgs {
+    /// How many days of verdicts to list.
+    #[arg(long, default_value_t = 90)]
+    days: i64,
+    /// Only verdicts with this label.
+    #[arg(long, value_enum)]
+    label: Option<RatingListLabel>,
+}
+
+#[derive(Debug, clap::Args)]
+struct RatingsSetArgs {
+    /// Article id, as printed by `ratings list` or `explain`.
+    #[arg(long, required_unless_present = "url", conflicts_with = "url")]
+    article: Option<ArticleId>,
+    /// Article URL; canonicalized before lookup.
+    #[arg(long, required_unless_present = "article", conflicts_with = "article")]
+    url: Option<String>,
+    /// The verdict to record.
+    #[arg(long, value_enum)]
+    label: RatingSetLabel,
+    /// Free-text note shown to the weekly profile rebuild.
+    #[arg(long)]
+    note: Option<String>,
+}
+
+#[derive(Debug, clap::Args)]
+struct RatingsClearArgs {
+    /// Article id, as printed by `ratings list` or `explain`.
+    #[arg(long, required_unless_present = "url", conflicts_with = "url")]
+    article: Option<ArticleId>,
+    /// Article URL; canonicalized before lookup.
+    #[arg(long, required_unless_present = "article", conflicts_with = "article")]
+    url: Option<String>,
+}
+
+/// `explain --date D (--article ID | --url URL) [--run-id N]` or
+/// `explain --date D --near-misses [N]` (plan §15.2).
+#[derive(Debug, clap::Args)]
+struct ExplainArgs {
+    /// Issue date whose run to read.
+    #[arg(long, value_name = "YYYY-MM-DD")]
+    date: String,
+    /// Article id, as printed by `ratings list` or `explain --near-misses`.
+    #[arg(
+        long,
+        required_unless_present_any = ["url", "near_misses"],
+        conflicts_with_all = ["url", "near_misses"]
+    )]
+    article: Option<ArticleId>,
+    /// Article URL; canonicalized before lookup.
+    #[arg(long, conflicts_with = "near_misses")]
+    url: Option<String>,
+    /// A specific run of that date instead of the latest non-dry one.
+    #[arg(long, value_name = "N")]
+    run_id: Option<i64>,
+    /// The top N articles that were considered but not selected (default 10).
+    #[arg(long, value_name = "N", num_args = 0..=1, default_missing_value = "10")]
+    near_misses: Option<usize>,
+}
+
+/// `stats [--days 14]` (plan §15.3).
+#[derive(Debug, clap::Args)]
+struct StatsArgs {
+    /// How many days back to summarize.
+    #[arg(long, default_value_t = 14)]
+    days: i64,
+}
+
+#[derive(Debug, Subcommand)]
+enum FeaturesCommand {
+    /// Embed rated and published articles, then interests, into the cache.
+    Backfill(BackfillArgs),
+    /// Drop stale embeddings, old candidate telemetry and old assessments per the retention config.
+    Prune,
+}
+
+#[derive(Debug, clap::Args)]
+struct BackfillArgs {
+    /// Window for published (and, with --all, other) articles.
+    #[arg(long, default_value_t = 30)]
+    days: i64,
+    /// Only the rated set.
+    #[arg(long, conflicts_with = "all")]
+    rated_only: bool,
+    /// Also every other article first seen inside the window.
+    #[arg(long)]
+    all: bool,
+    /// Skip the confirmation prompt above the token threshold.
+    #[arg(long)]
+    yes: bool,
+}
+
 #[derive(Debug, clap::Args)]
 struct BackfillSocialArgs {
     /// How many days back to re-poll.
@@ -100,6 +270,14 @@ async fn main() -> Result<()> {
     let config = Config::load(cli.config.as_deref()).context("loading configuration")?;
     tracing::debug!(?config.database_path, "configuration loaded");
 
+    // One writer at a time (§5); read-only commands never wait on it.
+    let _lock = match lock_holder(&cli.command) {
+        Some(name) => {
+            Some(lock::acquire(&config.database_path, name).map_err(|e| anyhow::anyhow!("{e}"))?)
+        }
+        None => None,
+    };
+
     match cli.command {
         Command::Generate(args) => {
             let db = Db::open_and_migrate(&config.database_path).await?;
@@ -114,6 +292,23 @@ async fn main() -> Result<()> {
             let db = Db::open_and_migrate(&config.database_path).await?;
             cmd_profile_rebuild(&config, &db).await?;
         }
+        Command::Ratings(command) => {
+            let db = Db::open_and_migrate(&config.database_path).await?;
+            cmd_ratings(&config, &db, command).await?;
+        }
+        Command::Explain(args) => {
+            let db = Db::open_and_migrate(&config.database_path).await?;
+            cmd_explain(&db, args).await?;
+        }
+        Command::Stats(args) => {
+            let db = Db::open_and_migrate(&config.database_path).await?;
+            let text = telemetry::stats(&db, args.days, jiff::Timestamp::now()).await?;
+            print!("{text}");
+        }
+        Command::Features(command) => {
+            let db = Db::open_and_migrate(&config.database_path).await?;
+            cmd_features(&config, &db, command).await?;
+        }
         Command::BackfillSocial(args) => {
             let db = Db::open_and_migrate(&config.database_path).await?;
             cmd_backfill_social(&db, args.days).await?;
@@ -123,10 +318,37 @@ async fn main() -> Result<()> {
             db.migrate().await?;
             println!("migrations up to date: {}", config.database_path.display());
         }
+        Command::Config(ConfigCommand::Check) => {
+            // Reaching here means `Config::load` already validated it; a bad
+            // config exited non-zero above. Nothing is opened, nothing locked.
+            let path = Config::resolve_path(cli.config.as_deref());
+            for line in config.check_report(path.as_deref()) {
+                println!("{line}");
+            }
+        }
     }
     Ok(())
 }
 
+/// The commands that write the database and provider budgets and so hold the
+/// run lock (§5): `generate`, `profile rebuild`, `features backfill`,
+/// `backfill-social`. Everything else is read-only or its own writer.
+fn lock_holder(command: &Command) -> Option<&'static str> {
+    match command {
+        Command::Generate(_) => Some("generate"),
+        Command::Profile(ProfileCommand::Rebuild) => Some("profile rebuild"),
+        Command::Features(FeaturesCommand::Backfill(_)) => Some("features backfill"),
+        Command::BackfillSocial(_) => Some("backfill-social"),
+        Command::Serve
+        | Command::Ratings(_)
+        | Command::Explain(_)
+        | Command::Stats(_)
+        | Command::Features(FeaturesCommand::Prune)
+        | Command::Db(_)
+        | Command::Config(_) => None,
+    }
+}
+
 /// `RUST_LOG`-driven tracing, defaulting to `info` (crate table "logging").
 fn init_tracing() {
     let filter = EnvFilter::try_from_default_env()
@@ -188,13 +410,36 @@ fn print_report(report: &RunReport) {
         report.counts.duplicates_merged,
         report.counts.entries_dropped,
     );
+    // The same four lines the run logged (§15.4).
+    for line in report.info_block() {
+        println!("{line}");
+    }
     println!(
-        "tokens: {} input · {} cached · {} output = ${:.4}",
+        "tokens: {} input · {} cache read · {} cache write · {} output · {} voyage = ${:.4}",
         report.usage.input_tokens,
         report.usage.cached_tokens,
+        report.usage.cache_write_tokens,
         report.usage.output_tokens,
+        report.voyage_tokens,
         report.cost_usd,
     );
+    for (provider, usage) in &report.provider_costs {
+        if provider == VOYAGE_PROVIDER {
+            continue; // embedding tokens are printed on their own line below
+        }
+        println!(
+            "  {provider}: {} input · {} cache read · {} cache write · {} output = ${:.4}",
+            usage.usage.input_tokens,
+            usage.usage.cached_tokens,
+            usage.usage.cache_write_tokens,
+            usage.usage.output_tokens,
+            usage.cost_usd,
+        );
+    }
+    println!(
+        "  voyage: {} tokens = ${:.4}",
+        report.voyage_tokens, report.voyage_cost_usd
+    );
     for warning in &report.warnings {
         println!("warning: {warning}");
     }
@@ -234,20 +479,248 @@ fn print_lineup(issue: &daily_epub::types::Issue) {
 // Other subcommands
 // ---------------------------------------------------------------------------
 
+/// `profile rebuild` runs on the editor when configured, else bulk (§14.3).
 async fn cmd_profile_rebuild(config: &Config, db: &Db) -> Result<()> {
-    let meter = curate::llm::UsageMeter::new(&config.deepseek, config.max_daily_usd);
+    use curate::llm::{Llms, provider_meters};
-    let profile = curate::profile::load_or_build(db, &config.interests_opml).await?;
+    let profile = curate::profile::load_or_build(
-    let llm = curate::llm::LlmClient::new(&config.deepseek, profile.text, meter)?;
+        db,
-    let rebuilt = curate::profile::rebuild(db, &llm, &config.interests_opml).await?;
+        &config.interests_opml,
-    let feeds = curate::profile::rebuild_feed_priors(db).await?;
+        &config.profile_path,
+        config.curation.feedback.verdicts_in_prompt,
+    )
+    .await?;
+    let llms = Llms::from_config(config, profile.text, &provider_meters(config));
+    let Some(llm) = llms.editor_or_bulk() else {
+        let keys = config
+            .referenced_providers()
+            .iter()
+            .map(|(name, _)| daily_epub::config::ProviderConfig::api_key_env_var(name))
+            .collect::<Vec<_>>();
+        anyhow::bail!(
+            "no LLM provider is available; assign [llm] roles and set {}",
+            if keys.is_empty() {
+                "a provider key".to_string()
+            } else {
+                keys.join(" or ")
+            }
+        );
+    };
+    tracing::info!(provider = llm.provider(), model = %llm.model, "rebuilding the profile");
+    let rebuilt = curate::profile::rebuild(
+        db,
+        llm,
+        &config.interests_opml,
+        &config.profile_path,
+        config.curation.feedback.verdicts_in_prompt,
+    )
+    .await?;
     println!(
-        "taste profile rebuilt (version {}, {} chars); {feeds} feed priors refreshed",
+        "taste profile rebuilt (version {}, {} chars)",
         rebuilt.version,
         rebuilt.text.len()
     );
     Ok(())
 }
 
+async fn resolve_rating_article(
+    db: &Db,
+    article: Option<ArticleId>,
+    url: Option<&str>,
+) -> Result<ArticleId> {
+    let article_id = match (article, url) {
+        (Some(article_id), None) => article_id,
+        (None, Some(url)) => {
+            let canonical = daily_epub::dedupe::canonical_url(url)
+                .with_context(|| format!("invalid article URL {url:?}"))?;
+            db.article_id_for_url(&canonical)
+                .await?
+                .with_context(|| format!("no article found for {canonical}"))?
+        }
+        _ => anyhow::bail!("provide exactly one of --article or --url"),
+    };
+    if db.get_article(article_id).await?.is_none() {
+        anyhow::bail!("article {article_id} was not found");
+    }
+    Ok(article_id)
+}
+
+async fn append_cli_event(
+    config: &Config,
+    db: &Db,
+    article_id: ArticleId,
+    vote: Option<Vote>,
+    note: Option<String>,
+) -> Result<i64> {
+    let (label, value) = match vote {
+        Some(Vote::Loved) => ("loved", Vote::Loved.value(&config.curation.feedback)),
+        Some(Vote::Good) => ("good", Vote::Good.value(&config.curation.feedback)),
+        Some(Vote::NotForMe) => (
+            "not_for_me",
+            Vote::NotForMe.value(&config.curation.feedback),
+        ),
+        None => ("cleared", 0.0),
+    };
+    let event = RatingEvent {
+        id: 0,
+        article_id,
+        issue_date: db.latest_issue_date_for_article(article_id).await?,
+        kind: "explicit".into(),
+        source: "cli".into(),
+        label: label.into(),
+        value,
+        note,
+        event_at: jiff::Timestamp::now(),
+    };
+    Ok(db.append_rating_event(&event).await?)
+}
+
+async fn cmd_ratings(config: &Config, db: &Db, command: RatingsCommand) -> Result<()> {
+    match command {
+        RatingsCommand::List(args) => {
+            let ratings = db.current_ratings_including_cleared(args.days).await?;
+            let mut shown = 0usize;
+            for rating in ratings {
+                if args
+                    .label
+                    .is_some_and(|label| rating.label != label.event_label())
+                {
+                    continue;
+                }
+                let note = rating
+                    .note
+                    .as_deref()
+                    .map(|note| format!(" · note: {note}"))
+                    .unwrap_or_default();
+                println!(
+                    "{} · article {} · {} · {} — {}{}",
+                    rating.event_at,
+                    rating.article_id,
+                    rating.label,
+                    rating.title,
+                    rating.feed_title,
+                    note
+                );
+                shown += 1;
+            }
+            println!("{shown} current rating(s)");
+        }
+        RatingsCommand::Set(args) => {
+            let article_id = resolve_rating_article(db, args.article, args.url.as_deref()).await?;
+            let vote = args.label.vote();
+            let event_id = append_cli_event(config, db, article_id, Some(vote), args.note).await?;
+            println!(
+                "recorded {} for article {article_id} (event {event_id})",
+                vote.as_str()
+            );
+        }
+        RatingsCommand::Clear(args) => {
+            let article_id = resolve_rating_article(db, args.article, args.url.as_deref()).await?;
+            let event_id = append_cli_event(config, db, article_id, None, None).await?;
+            println!("cleared article {article_id} (event {event_id})");
+        }
+    }
+    Ok(())
+}
+
+async fn cmd_explain(db: &Db, args: ExplainArgs) -> Result<()> {
+    let date: jiff::civil::Date = args
+        .date
+        .parse()
+        .with_context(|| format!("invalid --date {:?}, expected YYYY-MM-DD", args.date))?;
+    let text = if let Some(limit) = args.near_misses {
+        telemetry::explain_near_misses(db, date, args.run_id, limit).await?
+    } else {
+        let target = match (args.article, args.url) {
+            (Some(id), _) => telemetry::ExplainTarget::Article(id),
+            (None, Some(url)) => telemetry::ExplainTarget::Url(url),
+            (None, None) => anyhow::bail!("provide --article, --url or --near-misses"),
+        };
+        telemetry::explain(db, date, args.run_id, &target).await?
+    };
+    print!("{text}");
+    Ok(())
+}
+
+async fn cmd_features(config: &Config, db: &Db, command: FeaturesCommand) -> Result<()> {
+    match command {
+        FeaturesCommand::Backfill(args) => {
+            if !config.voyage.enabled {
+                anyhow::bail!("voyage.enabled is false; nothing to backfill");
+            }
+            let service = embedding::EmbeddingService::real(db.clone(), config.voyage.clone())
+                .context("building the Voyage client")?;
+            let opts = embedding::BackfillOptions {
+                days: args.days,
+                rated_only: args.rated_only,
+                all: args.all,
+            };
+            let plan = embedding::plan_backfill(db, config, &service, &opts).await?;
+            println!(
+                "backfill: {} articles ({} learned, {} other) + {} interests to embed, {} already cached",
+                plan.article_count(),
+                plan.learned.len(),
+                plan.others.len(),
+                plan.interests.len(),
+                plan.cached
+            );
+            if plan.is_empty() {
+                println!("cache is warm; nothing to do");
+                return Ok(());
+            }
+            println!(
+                "estimate: ~{} tokens ≈ ${:.4} with {} at ${:.2}/M",
+                plan.estimated_tokens,
+                plan.estimated_cost_usd(),
+                config.voyage.model,
+                embedding::VOYAGE_PRICE_PER_MTOK
+            );
+            if plan.estimated_tokens > BACKFILL_CONFIRM_TOKENS
+                && !args.yes
+                && !confirm("continue?")?
+            {
+                println!("aborted");
+                return Ok(());
+            }
+            let outcome = embedding::run_backfill(&service, &plan).await?;
+            println!(
+                "embedded {} articles and {} interests · {} tokens · ${:.4}",
+                outcome.articles_embedded,
+                outcome.interests_embedded,
+                outcome.tokens,
+                outcome.cost_usd
+            );
+        }
+        FeaturesCommand::Prune => {
+            let ranking = &config.curation.ranking;
+            let pruned = telemetry::prune(
+                db,
+                ranking.embedding_retention_days,
+                ranking.telemetry_retention_days,
+                jiff::Timestamp::now(),
+            )
+            .await?;
+            println!(
+                "pruned {} embeddings older than {} days, {} candidate rows and {} assessments older than {} days",
+                pruned.embeddings,
+                ranking.embedding_retention_days,
+                pruned.telemetry,
+                pruned.assessments,
+                ranking.telemetry_retention_days
+            );
+        }
+    }
+    Ok(())
+}
+
+/// A y/N question on stdin; anything but a leading `y` is a no.
+fn confirm(question: &str) -> Result<bool> {
+    print!("{question} [y/N] ");
+    std::io::stdout().flush()?;
+    let mut answer = String::new();
+    std::io::stdin().read_line(&mut answer)?;
+    Ok(answer.trim().to_lowercase().starts_with('y'))
+}
+
 async fn cmd_backfill_social(db: &Db, days: u32) -> Result<()> {
     let http = http::build_client(http::DEFAULT_TIMEOUT)?;
     let enricher = social::SocialEnricher::new(http, db.clone());
@@ -279,6 +752,8 @@ mod tests {
             "--max-articles",
             "6",
             "--skip-llm",
+            "--skip-embeddings",
+            "--rescore",
         ])
         .unwrap();
         match cli.command {
@@ -288,10 +763,12 @@ mod tests {
                 assert_eq!(a.out, Some(PathBuf::from("./out")));
                 assert_eq!(a.max_articles, Some(6));
                 assert!(a.skip_llm);
+                assert!(a.skip_embeddings);
+                assert!(a.rescore);
 
                 let opts = GenerateOptions::from(&a);
                 assert_eq!(opts.date.as_deref(), Some("2026-08-15"));
-                assert!(opts.dry_run && opts.skip_llm);
+                assert!(opts.dry_run && opts.skip_llm && opts.skip_embeddings && opts.rescore);
                 assert_eq!(opts.max_articles, Some(6));
             }
             other => panic!("expected generate, got {other:?}"),
@@ -309,6 +786,33 @@ mod tests {
                 .command,
             Command::Profile(ProfileCommand::Rebuild)
         ));
+        assert!(matches!(
+            Cli::try_parse_from([
+                "daily-epub",
+                "ratings",
+                "set",
+                "--article",
+                "42",
+                "--label",
+                "good"
+            ])
+            .unwrap()
+            .command,
+            Command::Ratings(RatingsCommand::Set(_))
+        ));
+        assert!(matches!(
+            Cli::try_parse_from([
+                "daily-epub",
+                "ratings",
+                "clear",
+                "--url",
+                "https://example.com"
+            ])
+            .unwrap()
+            .command,
+            Command::Ratings(RatingsCommand::Clear(_))
+        ));
+
         assert!(matches!(
             Cli::try_parse_from(["daily-epub", "backfill-social", "--days", "14"])
                 .unwrap()
@@ -325,4 +829,253 @@ mod tests {
         let cli = Cli::try_parse_from(["daily-epub", "--config", "/tmp/x.toml", "serve"]).unwrap();
         assert_eq!(cli.config, Some(PathBuf::from("/tmp/x.toml")));
     }
+
+    #[test]
+    fn only_the_writing_commands_take_the_lock() {
+        let parse = |args: &[&str]| {
+            Cli::try_parse_from(std::iter::once("daily-epub").chain(args.iter().copied()))
+                .unwrap()
+                .command
+        };
+        assert_eq!(lock_holder(&parse(&["generate"])), Some("generate"));
+        assert_eq!(
+            lock_holder(&parse(&["profile", "rebuild"])),
+            Some("profile rebuild")
+        );
+        assert_eq!(
+            lock_holder(&parse(&["features", "backfill"])),
+            Some("features backfill")
+        );
+        assert_eq!(
+            lock_holder(&parse(&["backfill-social"])),
+            Some("backfill-social")
+        );
+        for args in [
+            vec!["serve"],
+            vec!["explain", "--date", "2026-09-02", "--near-misses"],
+            vec!["stats"],
+            vec!["ratings", "list"],
+            vec!["db", "migrate"],
+            vec!["features", "prune"],
+            vec!["config", "check"],
+        ] {
+            assert_eq!(lock_holder(&parse(&args)), None, "{args:?}");
+        }
+    }
+
+    #[test]
+    fn parses_config_check() {
+        assert!(matches!(
+            Cli::try_parse_from(["daily-epub", "--config", "/etc/x.toml", "config", "check"])
+                .unwrap()
+                .command,
+            Command::Config(ConfigCommand::Check)
+        ));
+        assert!(Cli::try_parse_from(["daily-epub", "config"]).is_err());
+    }
+
+    #[test]
+    fn parses_stats() {
+        match Cli::try_parse_from(["daily-epub", "stats"])
+            .unwrap()
+            .command
+        {
+            Command::Stats(args) => assert_eq!(args.days, 14),
+            other => panic!("expected stats, got {other:?}"),
+        }
+        match Cli::try_parse_from(["daily-epub", "stats", "--days", "7"])
+            .unwrap()
+            .command
+        {
+            Command::Stats(args) => assert_eq!(args.days, 7),
+            other => panic!("expected stats, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn parses_explain_and_features() {
+        match Cli::try_parse_from([
+            "daily-epub",
+            "explain",
+            "--date",
+            "2026-09-02",
+            "--article",
+            "42",
+            "--run-id",
+            "7",
+        ])
+        .unwrap()
+        .command
+        {
+            Command::Explain(args) => {
+                assert_eq!(args.date, "2026-09-02");
+                assert_eq!(args.article, Some(42));
+                assert_eq!(args.run_id, Some(7));
+                assert_eq!(args.near_misses, None);
+            }
+            other => panic!("expected explain, got {other:?}"),
+        }
+        match Cli::try_parse_from([
+            "daily-epub",
+            "explain",
+            "--date",
+            "2026-09-02",
+            "--url",
+            "https://example.com/post",
+        ])
+        .unwrap()
+        .command
+        {
+            Command::Explain(args) => {
+                assert_eq!(args.url.as_deref(), Some("https://example.com/post"))
+            }
+            other => panic!("expected explain, got {other:?}"),
+        }
+        match Cli::try_parse_from([
+            "daily-epub",
+            "explain",
+            "--date",
+            "2026-09-02",
+            "--near-misses",
+        ])
+        .unwrap()
+        .command
+        {
+            Command::Explain(args) => assert_eq!(args.near_misses, Some(10)),
+            other => panic!("expected explain, got {other:?}"),
+        }
+        match Cli::try_parse_from([
+            "daily-epub",
+            "explain",
+            "--date",
+            "2026-09-02",
+            "--near-misses",
+            "3",
+        ])
+        .unwrap()
+        .command
+        {
+            Command::Explain(args) => assert_eq!(args.near_misses, Some(3)),
+            other => panic!("expected explain, got {other:?}"),
+        }
+        assert!(Cli::try_parse_from(["daily-epub", "explain", "--date", "2026-09-02"]).is_err());
+        assert!(
+            Cli::try_parse_from([
+                "daily-epub",
+                "explain",
+                "--date",
+                "2026-09-02",
+                "--article",
+                "1",
+                "--near-misses"
+            ])
+            .is_err()
+        );
+
+        match Cli::try_parse_from([
+            "daily-epub",
+            "features",
+            "backfill",
+            "--days",
+            "60",
+            "--all",
+            "--yes",
+        ])
+        .unwrap()
+        .command
+        {
+            Command::Features(FeaturesCommand::Backfill(args)) => {
+                assert_eq!(args.days, 60);
+                assert!(args.all && args.yes && !args.rated_only);
+            }
+            other => panic!("expected features backfill, got {other:?}"),
+        }
+        match Cli::try_parse_from(["daily-epub", "features", "backfill"])
+            .unwrap()
+            .command
+        {
+            Command::Features(FeaturesCommand::Backfill(args)) => assert_eq!(args.days, 30),
+            other => panic!("expected features backfill, got {other:?}"),
+        }
+        assert!(
+            Cli::try_parse_from([
+                "daily-epub",
+                "features",
+                "backfill",
+                "--rated-only",
+                "--all"
+            ])
+            .is_err()
+        );
+        assert!(matches!(
+            Cli::try_parse_from(["daily-epub", "features", "prune"])
+                .unwrap()
+                .command,
+            Command::Features(FeaturesCommand::Prune)
+        ));
+    }
+
+    #[tokio::test]
+    async fn cli_set_and_clear_append_cli_events_with_latest_issue_date() {
+        use sqlx::Row as _;
+
+        let dir = tempfile::tempdir().unwrap();
+        let db = Db::open_and_migrate(&dir.path().join("ratings.db"))
+            .await
+            .unwrap();
+        sqlx::query(
+            "INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
+             (42, 'https://example.com/article', 'Article', '2026-08-15T00:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .unwrap();
+        sqlx::query(
+            "INSERT INTO issues (date, issue_number, generated_at) VALUES
+             ('2026-08-14', 1, '2026-08-14T12:00:00Z'),
+             ('2026-08-15', 2, '2026-08-15T12:00:00Z')",
+        )
+        .execute(db.pool())
+        .await
+        .unwrap();
+        sqlx::query(
+            "INSERT INTO issue_articles (issue_date, article_id, section) VALUES
+             ('2026-08-14', 42, 'Top Stories'),
+             ('2026-08-15', 42, 'Top Stories')",
+        )
+        .execute(db.pool())
+        .await
+        .unwrap();
+
+        let config = Config::default();
+        append_cli_event(
+            &config,
+            &db,
+            42,
+            Some(Vote::Good),
+            Some("useful note".into()),
+        )
+        .await
+        .unwrap();
+        append_cli_event(&config, &db, 42, None, None)
+            .await
+            .unwrap();
+
+        let rows = sqlx::query(
+            "SELECT issue_date, source, label, value, note FROM rating_events ORDER BY id",
+        )
+        .fetch_all(db.pool())
+        .await
+        .unwrap();
+        assert_eq!(rows.len(), 2);
+        assert_eq!(rows[0].get::<String, _>("source"), "cli");
+        assert_eq!(rows[0].get::<String, _>("issue_date"), "2026-08-15");
+        assert_eq!(rows[0].get::<String, _>("label"), "good");
+        assert_eq!(rows[0].get::<f64, _>("value"), 0.35);
+        assert_eq!(rows[0].get::<String, _>("note"), "useful note");
+        assert_eq!(rows[1].get::<String, _>("source"), "cli");
+        assert_eq!(rows[1].get::<String, _>("label"), "cleared");
+        assert_eq!(rows[1].get::<f64, _>("value"), 0.0);
+        assert!(db.current_ratings(36500).await.unwrap().is_empty());
+    }
 }

src/publish.rs

@@ -752,6 +752,7 @@ mod tests {
             editorial: Editorial::default(),
             world_briefing: None,
             colophon: Colophon::default(),
+            behind: Default::default(),
         }
     }
 

src/report.rs

@@ -22,7 +22,7 @@ pub enum RunStatus {
     /// Everything completed.
     Ok,
     /// The issue was produced but a best-effort stage failed (social, XTC,
-    /// world briefing, images) or the cost guardrail tripped (§3.6).
+    /// world briefing, images) or a provider budget tripped (§5).
     Degraded,
     /// No issue was produced.
     Failed,
@@ -49,7 +49,7 @@ impl fmt::Display for RunStatus {
 }
 
 /// Per-stage article counts as the pipeline narrows the day's feed volume (§2).
-#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
 pub struct StageCounts {
     /// Entries returned by Miniflux inside the lookback window (§3.1).
     pub entries_fetched: i64,
@@ -67,11 +67,52 @@ pub struct StageCounts {
     pub excerpt_only: i64,
     /// Social lookups that returned a hit (§3.4).
     pub social_hits: i64,
-    /// Articles surviving the heuristic pre-filter (§3.5).
+    /// Articles passing hygiene and eligible for personalized signals.
+    pub eligible: i64,
+    /// Eligible articles with a valid embedding.
+    pub embedded: i64,
+    /// Current rated articles with a valid embedding.
+    pub rated_with_embeddings: i64,
+    /// The neighbour signal's gate ramp, 0–1 (§9.2); 0 means the signal is absent.
+    #[serde(default)]
+    pub knn_gate: f64,
+    /// The feed-affinity gate ramp, 0–1 (§9.3).
+    #[serde(default)]
+    pub feed_gate: f64,
+    /// Explicit verdicts rendered into the system prompt (§8.4).
+    #[serde(default)]
+    pub verdicts_in_prompt: i64,
+    /// Articles with a reusable or newly produced triage assessment.
+    pub triaged: i64,
+    /// Triage assessments served from cached rows.
+    #[serde(default)]
+    pub triage_reused: i64,
+    /// Pool articles without a triage because a provider rejected them
+    /// (this run's unrecovered rejections plus fresh `provider_rejected` rows).
+    #[serde(default)]
+    pub triage_rejected: i64,
+    /// Articles admitted to close reading.
+    pub admitted: i64,
+    /// First admitting retriever counts.
+    pub admitted_by: BTreeMap<String, i64>,
+    pub exploration_admitted: i64,
+    pub exploration_selected: i64,
+    /// Deep assessments, whether reused or newly produced.
+    pub assessed: i64,
+    /// Deep assessments served from cached rows.
+    #[serde(default)]
+    pub deep_reused: i64,
+    /// Admitted articles without a deep assessment because a provider
+    /// rejected them.
+    #[serde(default)]
+    pub deep_rejected: i64,
+    /// Candidates shown to the editor after diversification.
+    pub shortlisted: i64,
+    /// Leader clusters formed over the deep set.
+    pub clusters: i64,
+    /// Admitted deep-set count retained for the colophon and runs table.
     pub candidates: i64,
-    /// Articles scored by the LLM (§3.6 stage A).
+    /// Articles in the final lineup (§13).
-    pub llm_scored: i64,
-    /// Articles in the final lineup (§3.6 stage B).
     pub selected: i64,
     /// Discussion chapters rendered (§3.7).
     pub discussions: i64,
@@ -79,6 +120,10 @@ pub struct StageCounts {
     pub images_embedded: i64,
 }
 
+/// Key under which Voyage usage sits in `provider_costs` (§7.6). Its token count
+/// is embedding input, so it stays out of the LLM `usage` aggregate.
+pub const VOYAGE_PROVIDER: &str = "voyage";
+
 /// Wall-clock milliseconds per pipeline stage.
 #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
 pub struct StageTimings(pub BTreeMap<String, i64>);
@@ -93,6 +138,14 @@ impl StageTimings {
     }
 }
 
+/// Usage and computed cost for one provider in this run (§5).
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct ProviderUsage {
+    #[serde(flatten)]
+    pub usage: TokenUsage,
+    pub cost_usd: f64,
+}
+
 /// The full summary of one `generate` invocation (§3.13).
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct RunReport {
@@ -101,7 +154,15 @@ pub struct RunReport {
     pub finished_at: Option<Timestamp>,
     pub status: RunStatus,
     pub counts: StageCounts,
+    /// Aggregate usage retained for the legacy `runs` columns.
     pub usage: TokenUsage,
+    /// LLM provider-keyed usage and cost written to `runs.provider_costs_json`.
+    pub provider_costs: BTreeMap<String, ProviderUsage>,
+    /// Resolved curation/editorial/model settings for this run.
+    pub config_json: serde_json::Value,
+    /// Voyage document/query tokens and cost for this run.
+    pub voyage_tokens: i64,
+    pub voyage_cost_usd: f64,
     pub cost_usd: f64,
     pub timings: StageTimings,
     /// Ingest window actually used, RFC3339 (§3.1).
@@ -124,6 +185,10 @@ impl RunReport {
             status: RunStatus::Running,
             counts: StageCounts::default(),
             usage: TokenUsage::default(),
+            provider_costs: BTreeMap::new(),
+            config_json: serde_json::Value::Null,
+            voyage_tokens: 0,
+            voyage_cost_usd: 0.0,
             cost_usd: 0.0,
             timings: StageTimings::default(),
             window_start: None,
@@ -146,16 +211,24 @@ impl RunReport {
         self.error = Some(err.to_string());
     }
 
-    /// Stamp the end time, compute cost from [`TokenUsage`] and settle the status.
+    /// Stamp the end time, total provider costs (LLM providers plus Voyage) and
-    pub fn finish(
+    /// settle the status.
-        &mut self,
+    ///
-        finished_at: Timestamp,
+    /// Voyage is counted once: through its `provider_costs` entry when the
-        price_input: f64,
+    /// pipeline recorded one, else through `voyage_cost_usd`.
-        price_cached: f64,
+    pub fn finish(&mut self, finished_at: Timestamp) {
-        price_output: f64,
-    ) {
         self.finished_at = Some(finished_at);
-        self.cost_usd = self.usage.cost_usd(price_input, price_cached, price_output);
+        self.usage = TokenUsage::default();
+        self.cost_usd = 0.0;
+        for (provider, usage) in &self.provider_costs {
+            self.cost_usd += usage.cost_usd;
+            if provider != VOYAGE_PROVIDER {
+                self.usage.add(usage.usage);
+            }
+        }
+        if !self.provider_costs.contains_key(VOYAGE_PROVIDER) {
+            self.cost_usd += self.voyage_cost_usd;
+        }
         if self.status == RunStatus::Running {
             self.status = if self.warnings.is_empty() {
                 RunStatus::Ok
@@ -182,11 +255,12 @@ impl RunReport {
     /// Compact human-readable summary printed at the end of `generate`.
     pub fn summary_line(&self) -> String {
         format!(
-            "{} [{}] {} entries → {} articles → {} candidates → {} selected · ${:.4} · {}s",
+            "{} [{}] {} entries → {} articles → {} eligible → {} candidates → {} selected · ${:.4} · {}s",
             self.date,
             self.status,
             self.counts.entries_fetched,
             self.counts.articles,
+            self.counts.eligible,
             self.counts.candidates,
             self.counts.selected,
             self.cost_usd,
@@ -194,6 +268,82 @@ impl RunReport {
         )
     }
 
+    /// "23m12s" / "48s" for the log block and `stats`.
+    pub fn format_duration(secs: i64) -> String {
+        let secs = secs.max(0);
+        if secs >= 60 {
+            format!("{}m{:02}s", secs / 60, secs % 60)
+        } else {
+            format!("{secs}s")
+        }
+    }
+
+    /// The four-line info block of §15.4 (`curation:`, `admission:`,
+    /// `preference:`, `providers:`), logged once per run and printed by the CLI.
+    pub fn info_block(&self) -> [String; 4] {
+        let c = &self.counts;
+        let admitted = |name: &str| c.admitted_by.get(name).copied().unwrap_or(0);
+        let gate = |value: f64| {
+            if value > 0.0 {
+                format!("{value:.2}")
+            } else {
+                "off".to_string()
+            }
+        };
+        let providers = self
+            .provider_costs
+            .iter()
+            .map(|(provider, usage)| format!("{provider} ${:.2}", usage.cost_usd))
+            .collect::<Vec<_>>()
+            .join(" · ");
+        let providers = if providers.is_empty() {
+            "none".to_string()
+        } else {
+            providers
+        };
+        let rejected = |count: i64| {
+            if count > 0 {
+                format!(" · {count} rejected")
+            } else {
+                String::new()
+            }
+        };
+        [
+            format!(
+                "curation: {} considered → {} eligible → {} triaged{} → {} assessed{} → {} shortlisted → {} selected",
+                c.articles,
+                c.eligible,
+                c.triaged,
+                rejected(c.triage_rejected),
+                c.assessed,
+                rejected(c.deep_rejected),
+                c.shortlisted,
+                c.selected
+            ),
+            format!(
+                "admission: triage {} · interest {} · knn {} · exploration {} · blend {} · auto {}",
+                admitted("triage"),
+                admitted("interest"),
+                admitted("knn"),
+                admitted("exploration"),
+                admitted("blend"),
+                admitted("auto_include"),
+            ),
+            format!(
+                "preference: {} rated w/ embeddings → knn {} · feed {} · {} verdicts in prompt",
+                c.rated_with_embeddings,
+                gate(c.knn_gate),
+                gate(c.feed_gate),
+                c.verdicts_in_prompt
+            ),
+            format!(
+                "providers: {providers} · total ${:.2} · {}",
+                self.cost_usd,
+                Self::format_duration(self.duration_secs().unwrap_or(0))
+            ),
+        ]
+    }
+
     /// Feeds ordered by entry count, descending — the M1 dry-run breakdown.
     pub fn top_feeds(&self, limit: usize) -> Vec<(&str, i64)> {
         let mut v: Vec<(&str, i64)> = self
@@ -215,25 +365,126 @@ mod tests {
         s.parse().unwrap()
     }
 
+    fn usage(input: i64, cached: i64, cache_write: i64, output: i64) -> TokenUsage {
+        TokenUsage {
+            input_tokens: input,
+            cached_tokens: cached,
+            cache_write_tokens: cache_write,
+            output_tokens: output,
+        }
+    }
+
     #[test]
-    fn finish_computes_cost_and_status() {
+    fn finish_totals_provider_costs_and_settles_status() {
         let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
-        r.usage.add(TokenUsage {
+        r.provider_costs.insert(
-            input_tokens: 1_000_000,
+            "deepseek".into(),
-            cached_tokens: 1_000_000,
+            ProviderUsage {
-            output_tokens: 1_000_000,
+                usage: usage(1_000_000, 1_000_000, 0, 1_000_000),
-        });
+                cost_usd: 0.4228,
-        r.finish(ts("2026-08-15T05:36:00Z"), 0.14, 0.0028, 0.28);
+            },
+        );
+        r.provider_costs.insert(
+            "anthropic".into(),
+            ProviderUsage {
+                usage: usage(100, 3_000, 2_000, 800),
+                cost_usd: 0.05,
+            },
+        );
+        r.voyage_tokens = 250_000;
+        r.voyage_cost_usd = 0.005;
+        r.finish(ts("2026-08-15T05:36:00Z"));
         assert_eq!(r.status, RunStatus::Ok);
-        assert!((r.cost_usd - 0.4228).abs() < 1e-9);
+        // LLM providers plus Voyage; Voyage tokens stay out of the LLM aggregate.
+        assert!((r.cost_usd - 0.4778).abs() < 1e-9);
+        // The legacy aggregate columns are the sum across providers.
+        assert_eq!(r.usage, usage(1_000_100, 1_003_000, 2_000, 1_000_800));
         assert_eq!(r.duration_secs(), Some(360));
+
+        // Once the pipeline records Voyage as a provider (§7.6) it is counted
+        // there, not twice, and its tokens still stay out of the LLM aggregate.
+        r.provider_costs.insert(
+            VOYAGE_PROVIDER.into(),
+            ProviderUsage {
+                usage: usage(250_000, 0, 0, 0),
+                cost_usd: 0.005,
+            },
+        );
+        r.finish(ts("2026-08-15T05:36:00Z"));
+        assert!((r.cost_usd - 0.4778).abs() < 1e-9);
+        assert_eq!(r.usage, usage(1_000_100, 1_003_000, 2_000, 1_000_800));
+    }
+
+    #[test]
+    fn info_block_has_the_four_lines_of_the_plan() {
+        let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
+        r.counts.articles = 412;
+        r.counts.eligible = 398;
+        r.counts.triaged = 398;
+        r.counts.assessed = 120;
+        r.counts.shortlisted = 60;
+        r.counts.selected = 17;
+        r.counts.admitted_by = BTreeMap::from([
+            ("triage".to_string(), 60),
+            ("interest".to_string(), 20),
+            ("knn".to_string(), 12),
+            ("exploration".to_string(), 5),
+            ("blend".to_string(), 23),
+        ]);
+        r.counts.rated_with_embeddings = 14;
+        r.counts.knn_gate = 0.35;
+        r.counts.verdicts_in_prompt = 41;
+        for (provider, cost) in [("deepseek", 0.11), ("anthropic", 0.62), ("voyage", 0.02)] {
+            r.provider_costs.insert(
+                provider.into(),
+                ProviderUsage {
+                    usage: usage(1, 0, 0, 1),
+                    cost_usd: cost,
+                },
+            );
+        }
+        r.finish(ts("2026-08-15T05:53:12Z"));
+        let [curation, admission, preference, providers] = r.info_block();
+        assert_eq!(
+            curation,
+            "curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected"
+        );
+        assert_eq!(
+            admission,
+            "admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0"
+        );
+        assert_eq!(
+            preference,
+            "preference: 14 rated w/ embeddings → knn 0.35 · feed off · 41 verdicts in prompt"
+        );
+        assert_eq!(
+            providers,
+            "providers: anthropic $0.62 · deepseek $0.11 · voyage $0.02 · total $0.75 · 23m12s"
+        );
+        assert_eq!(RunReport::format_duration(48), "48s");
+        assert_eq!(RunReport::format_duration(3600), "60m00s");
+
+        // Provider rejections show up only when there were any.
+        r.counts.triaged = 395;
+        r.counts.triage_rejected = 3;
+        r.counts.deep_rejected = 0;
+        assert_eq!(
+            r.info_block()[0],
+            "curation: 412 considered → 398 eligible → 395 triaged · 3 rejected → 120 assessed → 60 shortlisted → 17 selected"
+        );
+        r.counts.assessed = 119;
+        r.counts.deep_rejected = 1;
+        assert_eq!(
+            r.info_block()[0],
+            "curation: 412 considered → 398 eligible → 395 triaged · 3 rejected → 119 assessed · 1 rejected → 60 shortlisted → 17 selected"
+        );
     }
 
     #[test]
     fn warnings_degrade_the_run() {
         let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
         r.warn("xtc converter missing");
-        r.finish(ts("2026-08-15T05:31:00Z"), 0.14, 0.0028, 0.28);
+        r.finish(ts("2026-08-15T05:31:00Z"));
         assert_eq!(r.status, RunStatus::Degraded);
         assert_eq!(r.warnings.len(), 1);
     }
@@ -245,12 +496,27 @@ mod tests {
         r.per_feed_counts.insert("Hacker News".into(), 30);
         r.per_feed_counts.insert("Lobsters".into(), 12);
         r.timings.record("ingest", 1500);
-        r.finish(ts("2026-08-15T05:31:00Z"), 0.14, 0.0028, 0.28);
+        r.config_json = serde_json::json!({"models": {"editor": "claude-opus-5"}});
+        r.provider_costs.insert(
+            "anthropic".into(),
+            ProviderUsage {
+                usage: usage(1, 2, 3, 4),
+                cost_usd: 0.01,
+            },
+        );
+        r.finish(ts("2026-08-15T05:31:00Z"));
         let json = r.to_json();
         let back: RunReport = serde_json::from_str(&json).unwrap();
         assert_eq!(back, r);
         assert_eq!(back.top_feeds(1), vec![("Hacker News", 30)]);
         assert_eq!(back.timings.total_ms(), 1500);
         assert!(back.summary_line().contains("412 entries"));
+        // `ProviderUsage` flattens the token counts next to the cost.
+        let value: serde_json::Value = serde_json::from_str(&json).unwrap();
+        assert_eq!(
+            value["provider_costs"]["anthropic"]["cache_write_tokens"],
+            3
+        );
+        assert_eq!(value["provider_costs"]["anthropic"]["cost_usd"], 0.01);
     }
 }

src/server.rs

@@ -6,7 +6,7 @@
 //! Routes (§3.12):
 //! | route | behaviour |
 //! |---|---|
-//! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC, upsert rating, rebuild feed priors |
+//! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC and append a rating event |
 //! | `GET /opds/daily.xml` (also `/opds`, `/opds/`) | OPDS 1.2 acquisition feed over `publish.epub_dir` |
 //! | `GET /files/epub/{name}` | EPUB download — what the feed's acquisition links point at |
 //! | `GET /files/xtc/{name}` | XTC artifact download, unlisted (no path traversal) |
@@ -15,7 +15,7 @@
 //!
 //! `/opds/*` and `/files/*` sit behind optional Basic auth (`server.basic_auth_*`).
 //!
-//! The EPUB article footer (§3.10) mints its 👍/👎 links with the very same
+//! The EPUB article footer (§3.10) mints its three verdict links with the very same
 //! [`rating_url`] this module verifies with — both re-export [`crate::auth`],
 //! which pins the shared test vector (`secret = "test-secret"`, `2026-08-15`,
 //! article `42`, `up` → `3b314cf7e6d8f50f`). An issue generated while
@@ -38,7 +38,7 @@ use tower_http::trace::TraceLayer;
 
 use crate::config::Config;
 use crate::db::Db;
-use crate::types::{ArticleId, Rating, Vote};
+use crate::types::{ArticleId, RatingEvent, Vote};
 
 /// Characters of the hex HMAC kept in rating links (§3.9).
 pub const TOKEN_LEN: usize = crate::auth::TOKEN_LEN;
@@ -244,46 +244,47 @@ async fn handle_rating(
         }
     };
 
-    let rating = Rating {
+    let label = match vote {
-        issue_date: date,
+        Vote::Loved => "loved",
+        Vote::Good => "good",
+        Vote::NotForMe => "not_for_me",
+    };
+    let event = RatingEvent {
+        id: 0,
+        issue_date: Some(date),
         article_id,
-        vote,
+        kind: "explicit".into(),
-        rated_at: Timestamp::now(),
+        source: "epub".into(),
+        label: label.into(),
+        value: vote.value(&state.config.curation.feedback),
+        note: None,
+        event_at: Timestamp::now(),
     };
-    let changed = match state.db.upsert_rating(&rating).await {
+    if let Err(error) = state.db.append_rating_event(&event).await {
-        Ok(changed) => changed,
+        tracing::error!(%error, article_id, "recording the rating failed");
-        Err(e) => {
+        return page(StatusCode::INTERNAL_SERVER_ERROR, "Database error.", None);
-            tracing::error!(error = %e, article_id, "recording the rating failed");
-            return page(StatusCode::INTERNAL_SERVER_ERROR, "Database error.", None);
-        }
-    };
-    if changed && let Err(e) = crate::curate::profile::rebuild_feed_priors(&state.db).await {
-        // The vote is stored; a stale prior only affects the next run's ranking.
-        tracing::error!(error = %e, "refreshing feed priors failed");
     }
     tracing::info!(
         %date,
         article_id,
         feed_id = article.feed_id,
         vote = vote.as_str(),
-        changed,
         title = %article.title,
-        "recorded rating"
+        "recorded rating event"
     );
 
-    let glyph = match vote {
+    let message = match vote {
-        Vote::Up => "👍",
+        Vote::Loved => "Recorded: Loved it — thanks.",
-        Vote::Down => "👎",
+        Vote::Good => "Recorded: Good — thanks.",
+        Vote::NotForMe => "Recorded: Not for me — thanks.",
     };
-    let message = if changed {
+    confirmation_page(
-        format!("Recorded {glyph} — thanks!")
-    } else {
-        format!("Already recorded {glyph} — thanks!")
-    };
-    page(
         StatusCode::OK,
-        &message,
+        message,
-        Some(&format!("{date} · article {article_id}")),
+        &state.config,
+        date,
+        article_id,
+        vote,
     )
 }
 
@@ -457,7 +458,7 @@ fn check_basic_auth(config: &Config, headers: &HeaderMap) -> Option<Response> {
 /// A self-contained response page — no external CSS, well under 1 KB, legible on
 /// a 6" e-ink browser (§3.9).
 fn page(status: StatusCode, message: &str, note: Option<&str>) -> Response {
-    let body = page_html(message, note);
+    let body = page_html(message, note, None);
     (
         status,
         [
@@ -469,8 +470,52 @@ fn page(status: StatusCode, message: &str, note: Option<&str>) -> Response {
         .into_response()
 }
 
-/// The page markup itself: no stylesheet, no script, no images (§3.9).
+fn confirmation_page(
-fn page_html(message: &str, note: Option<&str>) -> String {
+    status: StatusCode,
+    message: &str,
+    config: &Config,
+    date: Date,
+    article_id: ArticleId,
+    selected: Vote,
+) -> Response {
+    let Some(secret) = config.server.hmac_secret.as_deref() else {
+        return page(status, message, None);
+    };
+    let choices = [
+        (Vote::Loved, "Loved it"),
+        (Vote::Good, "Good"),
+        (Vote::NotForMe, "Not for me"),
+    ]
+    .into_iter()
+    .filter(|(vote, _)| *vote != selected)
+    .map(|(vote, label)| {
+        let url = rating_url(&config.server.public_url, secret, date, article_id, vote);
+        format!(
+            "<a href=\"{}\">[ {} ]</a>",
+            escape_attr(&url),
+            escape(label)
+        )
+    })
+    .collect::<Vec<_>>()
+    .join(" &nbsp; ");
+    let body = page_html(
+        message,
+        None,
+        Some(&format!("<p><small>Change it: {choices}</small></p>")),
+    );
+    (
+        status,
+        [
+            (header::CONTENT_TYPE, "text/html; charset=utf-8"),
+            (header::CACHE_CONTROL, "no-store"),
+        ],
+        body,
+    )
+        .into_response()
+}
+
+/// The page markup itself: no external stylesheet, script, or images (§6.1).
+fn page_html(message: &str, note: Option<&str>, extra_html: Option<&str>) -> String {
     let note = note
         .map(|n| format!("<p><small>{}</small></p>", escape(n)))
         .unwrap_or_default();
@@ -478,11 +523,12 @@ fn page_html(message: &str, note: Option<&str>) -> String {
         "<!doctype html><html lang=\"en\"><meta charset=\"utf-8\">\
 <meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\
 <title>The Daily EPUB</title>\
-<style>body{{margin:3em auto;max-width:16em;padding:0 1em;text-align:center;\
+<style>body{{margin:3em auto;max-width:18em;padding:0 1em;text-align:center;\
 font:1.3em/1.5 Georgia,serif}}small{{font-size:.65em}}</style>\
-<p>{}</p>{}",
+<p>{}</p>{}{}",
         escape(message),
-        note
+        note,
+        extra_html.unwrap_or_default()
     )
 }
 
@@ -492,6 +538,10 @@ fn escape(s: &str) -> String {
         .replace('>', "&gt;")
 }
 
+fn escape_attr(s: &str) -> String {
+    escape(s).replace('\"', "&quot;")
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -512,29 +562,33 @@ mod tests {
 
     #[test]
     fn token_matches_the_shared_test_vector() {
-        assert_eq!(
+        let loved = rating_token(VECTOR_SECRET, date(), 42, Vote::Loved);
-            rating_token(VECTOR_SECRET, date(), 42, Vote::Up),
+        assert_eq!(loved, "cece96767d6c5f8a");
-            VECTOR_TOKEN_UP
+        assert_eq!(loved.len(), 16);
-        );
-        assert_eq!(rating_token(VECTOR_SECRET, date(), 42, Vote::Up).len(), 16);
         // Down differs from up, and both verify.
-        let down = rating_token(VECTOR_SECRET, date(), 42, Vote::Down);
+        let down = rating_token(VECTOR_SECRET, date(), 42, Vote::NotForMe);
         assert_ne!(down, VECTOR_TOKEN_UP);
         assert!(verify_token(
             VECTOR_SECRET,
             date(),
             42,
-            Vote::Up,
+            Vote::Loved,
             VECTOR_TOKEN_UP
         ));
-        assert!(verify_token(VECTOR_SECRET, date(), 42, Vote::Down, &down));
+        assert!(verify_token(
+            VECTOR_SECRET,
+            date(),
+            42,
+            Vote::NotForMe,
+            &down
+        ));
     }
 
     /// The links the EPUB footer embeds must verify here — this is the whole
     /// feedback loop in one assertion (§3.9).
     #[test]
     fn epub_footer_links_verify_against_this_server() {
-        for (id, vote) in [(42, Vote::Up), (1234, Vote::Down)] {
+        for (id, vote) in [(42, Vote::Loved), (1234, Vote::NotForMe)] {
             let from_epub = crate::epub::build::rating_url(
                 "https://daily.hallada.net",
                 VECTOR_SECRET,
@@ -556,23 +610,23 @@ mod tests {
 
     #[test]
     fn token_verification_rejects_tampering() {
-        let t = rating_token(VECTOR_SECRET, date(), 42, Vote::Up);
+        let t = rating_token(VECTOR_SECRET, date(), 42, Vote::Loved);
-        assert!(!verify_token(VECTOR_SECRET, date(), 42, Vote::Down, &t));
+        assert!(!verify_token(VECTOR_SECRET, date(), 42, Vote::NotForMe, &t));
-        assert!(!verify_token(VECTOR_SECRET, date(), 43, Vote::Up, &t));
+        assert!(!verify_token(VECTOR_SECRET, date(), 43, Vote::Loved, &t));
-        assert!(!verify_token("other-secret", date(), 42, Vote::Up, &t));
+        assert!(!verify_token("other-secret", date(), 42, Vote::Loved, &t));
         assert!(!verify_token(
             VECTOR_SECRET,
             "2026-08-16".parse().unwrap(),
             42,
-            Vote::Up,
+            Vote::Loved,
             &t
         ));
-        assert!(!verify_token(VECTOR_SECRET, date(), 42, Vote::Up, ""));
+        assert!(!verify_token(VECTOR_SECRET, date(), 42, Vote::Loved, ""));
         assert!(!verify_token(
             VECTOR_SECRET,
             date(),
             42,
-            Vote::Up,
+            Vote::Loved,
             &format!("{t}00")
         ));
     }
@@ -585,9 +639,12 @@ mod tests {
                 VECTOR_SECRET,
                 date(),
                 42,
-                Vote::Up
+                Vote::Loved
             ),
-            format!("https://daily.hallada.net/r/2026-08-15/42/up?t={VECTOR_TOKEN_UP}")
+            format!(
+                "https://daily.hallada.net/r/2026-08-15/42/loved?t={}",
+                rating_token(VECTOR_SECRET, date(), 42, Vote::Loved)
+            )
         );
     }
 
@@ -626,16 +683,20 @@ mod tests {
 
     #[test]
     fn the_confirmation_page_is_tiny_and_self_contained() {
-        let html = page_html("Recorded 👍 — thanks!", Some("2026-08-15 · article 42"));
+        let html = page_html(
+            "Recorded: Loved it — thanks.",
+            Some("2026-08-15 · article 42"),
+            None,
+        );
         assert!(html.len() < 1024, "page is {} bytes", html.len());
         assert!(!html.contains("<link"), "no external stylesheet");
         assert!(!html.contains("<script"), "no script");
-        assert!(html.contains("Recorded 👍"));
+        assert!(html.contains("Recorded: Loved it"));
         assert_eq!(
             page(StatusCode::FORBIDDEN, "Invalid link.", None).status(),
             StatusCode::FORBIDDEN
         );
-        assert!(page_html("<b>x</b>", None).contains("&lt;b&gt;"));
+        assert!(page_html("<b>x</b>", None, None).contains("&lt;b&gt;"));
     }
 
     // -----------------------------------------------------------------
@@ -810,47 +871,43 @@ mod tests {
     }
 
     #[tokio::test]
-    async fn rating_happy_path_is_idempotent_and_updates_priors() {
+    async fn rating_taps_append_events_and_latest_correction_wins() {
         let server = TestServer::start(false).await;
         let id = server.seed_article().await;
-        let url = rating_url(&server.base, VECTOR_SECRET, date(), id, Vote::Up);
+        let loved = rating_url(&server.base, VECTOR_SECRET, date(), id, Vote::Loved);
 
-        let res = client().get(&url).send().await.unwrap();
+        let response = client().get(&loved).send().await.unwrap();
-        assert_eq!(res.status(), 200);
+        assert_eq!(response.status(), 200);
-        let body = res.text().await.unwrap();
+        let body = response.text().await.unwrap();
-        assert!(body.contains("Recorded"), "{body}");
+        assert!(body.contains("Recorded: Loved it — thanks."), "{body}");
-        assert!(!body.contains("Already"), "{body}");
+        assert!(body.contains("[ Good ]"), "{body}");
+        assert!(body.contains("[ Not for me ]"), "{body}");
         assert!(
-            body.len() < 1024,
+            body.len() < 2048,
             "confirmation page is {} bytes",
             body.len()
         );
 
-        // Same tap again: still 200, but reported as already recorded.
+        // Every tap is history, even a repeated one.
-        let body = client()
+        assert_eq!(client().get(&loved).send().await.unwrap().status(), 200);
-            .get(&url)
+        let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM rating_events")
-            .send()
+            .fetch_one(server.db.pool())
-            .await
-            .unwrap()
-            .text()
             .await
             .unwrap();
-        assert!(body.contains("Already recorded"), "{body}");
+        assert_eq!(count, 2);
 
-        let ratings = server.db.ratings_with_feed().await.unwrap();
+        // A correction appends and becomes the current verdict.
-        assert_eq!(ratings, vec![(7, Vote::Up)]);
+        let down = rating_url(&server.base, VECTOR_SECRET, date(), id, Vote::NotForMe);
-        let priors = server.db.feed_priors().await.unwrap();
-        assert_eq!(priors.len(), 1);
-        assert_eq!(
-            (priors[0].feed_id, priors[0].upvotes, priors[0].downvotes),
-            (7, 1, 0)
-        );
-
-        // Flipping the vote rewrites the prior rather than double-counting.
-        let down = rating_url(&server.base, VECTOR_SECRET, date(), id, Vote::Down);
         assert_eq!(client().get(&down).send().await.unwrap().status(), 200);
-        let priors = server.db.feed_priors().await.unwrap();
+        let current = server.db.current_ratings(36500).await.unwrap();
-        assert_eq!((priors[0].upvotes, priors[0].downvotes), (0, 1));
+        assert_eq!(current.len(), 1);
+        assert_eq!(current[0].label, "not_for_me");
+        let sources: Vec<String> =
+            sqlx::query_scalar("SELECT source FROM rating_events ORDER BY id")
+                .fetch_all(server.db.pool())
+                .await
+                .unwrap();
+        assert_eq!(sources, ["epub", "epub", "epub"]);
     }
 
     #[tokio::test]
@@ -864,17 +921,21 @@ mod tests {
         assert_eq!(client().get(&missing).send().await.unwrap().status(), 403);
 
         // A valid token for an article that does not exist.
-        let unknown = rating_url(&server.base, VECTOR_SECRET, date(), 9999, Vote::Up);
+        let unknown = rating_url(&server.base, VECTOR_SECRET, date(), 9999, Vote::Loved);
         assert_eq!(client().get(&unknown).send().await.unwrap().status(), 404);
 
         // Malformed date / vote.
-        let token = rating_token(VECTOR_SECRET, date(), id, Vote::Up);
+        let token = rating_token(VECTOR_SECRET, date(), id, Vote::Loved);
         let bad_date = format!("{}/r/not-a-date/{id}/up?t={token}", server.base);
         assert_eq!(client().get(&bad_date).send().await.unwrap().status(), 400);
         let bad_vote = format!("{}/r/2026-08-15/{id}/sideways?t={token}", server.base);
         assert_eq!(client().get(&bad_vote).send().await.unwrap().status(), 400);
 
-        assert!(server.db.ratings_with_feed().await.unwrap().is_empty());
+        let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM rating_events")
+            .fetch_one(server.db.pool())
+            .await
+            .unwrap();
+        assert_eq!(count, 0);
     }
 
     #[tokio::test]

src/types.rs

@@ -246,46 +246,83 @@ pub fn composite_social_score(refs: &[SocialRef]) -> f64 {
 }
 
 // ---------------------------------------------------------------------------
-// Curation (§3.5, §3.6)
+// Curation (plan §10–§13)
 // ---------------------------------------------------------------------------
 
-/// DeepSeek stage-A output for one article (§3.6).
+/// DeepSeek's close read of one article (§12.1).
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct LlmScore {
+pub struct Deep {
-    /// 0–10.
+    /// Editorial quality on the article's own terms, clamped to 0–10.
-    pub score: f64,
+    pub quality: f64,
-    pub category: String,
+    /// Fit for this reader, clamped to 0–10.
-    /// ≤ 20 words.
+    pub fit: f64,
+    pub category: Option<String>,
     pub rationale: String,
-    #[serde(default)]
+    pub paywalled_guess: bool,
-    pub is_paywalled_guess: bool,
+    pub facets: Facets,
+    pub model: String,
+    pub prompt_version: i64,
+    pub assessed_at: Timestamp,
 }
 
-/// An article carrying every ranking signal computed so far (§3.5, §3.6).
+/// Personalized first-pass judgment cached in `article_assessments` (§10).
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct ScoredArticle {
+pub struct Triage {
-    pub article: Article,
+    /// Reader interest, clamped to 0–10.
-    /// Heuristic pre-filter score, 0–100 (§3.5).
+    pub interest: f64,
-    pub prefilter_score: f64,
+    pub kind: String,
-    /// Cached [`composite_social_score`] for the article.
+    /// At most twelve words in a conforming response.
-    pub social_score: f64,
+    pub why: String,
-    /// Beta-smoothed per-feed upvote rate applied by the pre-filter (§3.9).
+    pub model: String,
-    pub feed_prior: f64,
+    pub prompt_version: i64,
-    /// `None` until stage A has run (or when `--skip-llm`).
+    pub assessed_at: Timestamp,
-    pub llm: Option<LlmScore>,
-    /// From `curation.always_include_feeds`: may be scored but never dropped (§3.5).
-    pub auto_include: bool,
 }
 
-impl ScoredArticle {
+/// Cached LLM judgments accumulated for an article (§7.3).
-    /// Ranking key for stage B: LLM score weighted with social proof and priors (§3.6).
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
-    pub fn combined_score(&self) -> f64 {
+pub struct Assessment {
-        let llm = self.llm.as_ref().map(|l| l.score).unwrap_or(0.0);
+    pub triage: Option<Triage>,
-        llm * 10.0 + self.social_score * 4.0 + self.feed_prior * 10.0 + self.prefilter_score * 0.1
+    pub deep: Option<Deep>,
+}
+
+/// An article carrying the state of the personalized curation pipeline (§18).
+#[derive(Debug, Clone, PartialEq)]
+pub struct Candidate {
+    pub article: Article,
+    pub auto_include: bool,
+    pub exploration: bool,
+    pub signals: crate::curate::signals::Signals,
+    pub assessment: Assessment,
+    pub utility: Option<f64>,
+    pub rank_utility: Option<i64>,
+    pub cluster: Option<i64>,
+    pub cluster_rank: Option<i64>,
+    pub admitted_by: Vec<String>,
+    pub stage: String,
+    pub excluded_reason: Option<String>,
+}
+
+impl Candidate {
+    pub fn new(article: Article, auto_include: bool) -> Self {
+        let signals = crate::curate::signals::Signals::baseline(&article);
+        Self {
+            article,
+            auto_include,
+            exploration: false,
+            signals,
+            assessment: Assessment::default(),
+            utility: None,
+            rank_utility: None,
+            cluster: None,
+            cluster_rank: None,
+            admitted_by: Vec::new(),
+            stage: "eligible".into(),
+            excluded_reason: None,
+        }
     }
 }
 
-/// One selected article with its section placement (§3.6 stage B).
+/// One selected article with its section placement (§13).
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct Pick {
     pub article: Article,
@@ -294,20 +331,23 @@ pub struct Pick {
     /// Order within the section, ascending.
     pub position: i64,
     pub is_lead: bool,
+    /// Editor-written reason, at most 14 words (§13).
+    pub why: Option<String>,
     /// Newspaper-abstract summary from stage C; `None` until editorial runs.
     pub summary: Option<String>,
-    pub llm: Option<LlmScore>,
+    pub llm: Option<Deep>,
     /// Rendered comment chapter, when the article had social refs (§3.7).
     pub discussion: Option<Discussion>,
 }
 
-/// The day's final lineup: 15–25 picks grouped into sections (§3.6 stage B).
+/// The day's final lineup grouped into sections (§13): no minimum size,
+/// `curation.max_article_count` (or `--max-articles`) as the ceiling.
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct Lineup {
     pub date: Date,
     /// Sorted by (section order, position).
     pub picks: Vec<Pick>,
-    /// Section names in issue order; empty sections are omitted (§3.6).
+    /// Section names in issue order; empty sections are omitted.
     pub section_order: Vec<String>,
 }
 
@@ -328,24 +368,26 @@ impl Lineup {
     }
 }
 
-/// Stage-C editorial output (§3.6).
+/// Editorial output: the Brief and the per-article summaries (§14).
 #[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
 pub struct Editorial {
-    /// "From the Editor", 250–400 words, already sanitized XHTML.
+    /// The Brief (§14.2), 120–200 words, already sanitized XHTML.
     pub front_page_html: String,
-    /// Section name → 2–3 sentence intro.
+    /// Article id → 2–3 sentence newspaper abstract (§14.1).
-    pub section_intros: BTreeMap<String, String>,
-    /// Article id → 2–3 sentence newspaper abstract.
     pub summaries: BTreeMap<ArticleId, String>,
 }
 
-/// The taste profile that forms the DeepSeek system prompt (§3.6, `kv`).
+/// The reader profile that forms the system prompt shared by every LLM call
+/// (§8.4); the current text lives in `kv`.
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct TasteProfile {
     /// Full ~600-word prompt document.
     pub text: String,
     pub version: i64,
     pub built_at: Timestamp,
+    /// Explicit verdicts rendered into the prompt's "Recent verdicts" block (§8.4).
+    #[serde(default)]
+    pub verdicts: usize,
 }
 
 // ---------------------------------------------------------------------------
@@ -525,12 +567,24 @@ pub struct Issue {
     pub world_briefing: Option<WorldBriefing>,
     /// Colophon facts: models used, token cost, feed counts (§3.10).
     pub colophon: Colophon,
+    /// The "Behind the paper" chapter's facts (§15.1); filled by the pipeline.
+    #[serde(default)]
+    pub behind: BehindThePaper,
+}
+
+/// Resolved model names printed in the colophon (§15.1).
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct Models {
+    pub bulk: String,
+    pub editor: String,
+    pub summaries: String,
 }
 
 /// Back-matter facts printed in the colophon chapter (§3.10).
 #[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
 pub struct Colophon {
-    pub model: String,
+    pub provider_costs: BTreeMap<String, f64>,
+    pub models: Models,
     pub entries_fetched: i64,
     pub feeds_seen: i64,
     pub candidates: i64,
@@ -538,6 +592,49 @@ pub struct Colophon {
     pub generator_version: String,
 }
 
+/// One of the highest-utility articles that did not make the paper (§15.1).
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct NearMiss {
+    pub article_id: ArticleId,
+    pub title: String,
+    pub feed_title: String,
+    pub quality: Option<f64>,
+    pub fit: Option<f64>,
+    /// `candidate_runs.stage` reached.
+    pub stage: String,
+    /// `candidate_runs.excluded_reason`, when the row carries one.
+    pub reason: Option<String>,
+}
+
+/// Facts for the "Behind the paper" chapter (§15.1), derived from the run's
+/// report and its `candidate_runs` rows. Templates render it verbatim.
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+pub struct BehindThePaper {
+    /// Deduplicated articles the run looked at.
+    pub considered: i64,
+    pub feeds_seen: i64,
+    pub eligible: i64,
+    pub triaged: i64,
+    /// Deep-assessed ("read closely").
+    pub read_closely: i64,
+    pub shortlisted: i64,
+    pub selected: i64,
+    /// First admitting retriever → count (§11).
+    pub admitted_by: BTreeMap<String, i64>,
+    pub rated_with_embeddings: i64,
+    /// The neighbour signal's gate ramp, 0–1 (§9.2).
+    pub knn_gate: f64,
+    /// The feed-affinity gate ramp, 0–1 (§9.3).
+    pub feed_gate: f64,
+    pub near_misses: Vec<NearMiss>,
+    pub models: Models,
+    pub embedding_model: String,
+    /// Every provider, Voyage included.
+    pub cost_usd: f64,
+    /// Wall clock from the run's start to issue assembly.
+    pub generation_secs: i64,
+}
+
 /// A downloaded, re-encoded image embedded in an edition (§3.10 images).
 #[derive(Debug, Clone, PartialEq)]
 pub struct ImageAsset {
@@ -565,75 +662,101 @@ pub struct Artifact {
 // Feedback (§3.9)
 // ---------------------------------------------------------------------------
 
-/// 👍 / 👎 stored as `+1` / `-1` in `ratings.vote` (§3.9).
+/// Explicit reader verdict embedded in rating-link URLs (§6.1).
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
 pub enum Vote {
-    Up,
+    #[serde(rename = "loved", alias = "up")]
-    Down,
+    Loved,
+    #[serde(rename = "good")]
+    Good,
+    #[serde(rename = "down")]
+    NotForMe,
 }
 
 impl Vote {
-    pub fn as_i64(self) -> i64 {
+    /// Stable path segment used in rating links (§6.1).
-        match self {
-            Vote::Up => 1,
-            Vote::Down => -1,
-        }
-    }
-
-    /// Path segment used in rating links: `up` / `down` (§3.9).
     pub fn as_str(self) -> &'static str {
         match self {
-            Vote::Up => "up",
+            Vote::Loved => "loved",
-            Vote::Down => "down",
+            Vote::Good => "good",
+            Vote::NotForMe => "down",
         }
     }
 
     pub fn parse(s: &str) -> Option<Self> {
         match s {
-            "up" => Some(Vote::Up),
+            "loved" | "up" => Some(Vote::Loved),
-            "down" => Some(Vote::Down),
+            "good" => Some(Vote::Good),
+            "down" => Some(Vote::NotForMe),
             _ => None,
         }
     }
+
+    pub fn value(self, cfg: &crate::config::FeedbackConfig) -> f64 {
+        match self {
+            Vote::Loved => cfg.loved_value,
+            Vote::Good => cfg.good_value,
+            Vote::NotForMe => cfg.not_for_me_value,
+        }
+    }
 }
 
-/// A recorded reader vote (`ratings` table, §3.9).
+/// One append-only feedback event (`rating_events`, §6.2).
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-pub struct Rating {
+pub struct RatingEvent {
-    pub issue_date: Date,
+    pub id: i64,
     pub article_id: ArticleId,
-    pub vote: Vote,
+    pub issue_date: Option<Date>,
-    pub rated_at: Timestamp,
+    pub kind: String,
+    pub source: String,
+    pub label: String,
+    pub value: f64,
+    pub note: Option<String>,
+    pub event_at: Timestamp,
 }
 
-/// Beta-smoothed per-feed upvote rate used by the pre-filter (`feed_priors`, §3.9).
+/// Descriptive deep-assessment facets (§12.1); shown to the editor, the profile
-#[derive(Debug, Clone, Copy, Default, PartialEq, Serialize, Deserialize)]
+/// rebuild and `explain`, never a numeric ranking signal.
-pub struct FeedPrior {
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
-    pub feed_id: FeedId,
+pub struct Facets {
-    pub upvotes: i64,
+    pub format: Option<String>,
-    pub downvotes: i64,
+    pub depth: Option<String>,
-    pub included: i64,
+    pub evidence: Option<String>,
+    pub commerciality: Option<String>,
+    pub topic_group: Option<String>,
+    pub technicality: Option<String>,
+    pub locality: Option<String>,
+    pub specific_topics: Option<Vec<String>>,
 }
 
-impl FeedPrior {
+/// The current explicit verdict for an article, enriched for prompts (§6.2).
-    /// `(up + 1) / (up + down + 2)` — 0.5 with no evidence (§3.9).
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-    pub fn rate(&self) -> f64 {
+pub struct RatedArticle {
-        (self.upvotes + 1) as f64 / (self.upvotes + self.downvotes + 2) as f64
+    pub article_id: ArticleId,
-    }
+    pub issue_date: Option<Date>,
+    pub title: String,
+    pub feed_title: String,
+    pub summary: Option<String>,
+    pub facets: Option<Facets>,
+    pub note: Option<String>,
+    pub value: f64,
+    pub label: String,
+    pub event_at: Timestamp,
 }
 
 // ---------------------------------------------------------------------------
-// LLM accounting (§3.6 cost guardrail)
+// LLM accounting (§5 per-provider budgets)
 // ---------------------------------------------------------------------------
 
-/// Token counters accumulated across every DeepSeek call in a run (§3.6).
+/// Token counters accumulated across one provider's calls in a run (§5).
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
 pub struct TokenUsage {
     /// Cache-miss input tokens (billed at the full input rate).
     pub input_tokens: i64,
-    /// Prefix-cache hits (billed at the cached rate).
+    /// Prefix-cache reads (billed at the provider's cache-read rate).
     pub cached_tokens: i64,
+    /// Tokens written into a prompt cache (Anthropic only).
+    pub cache_write_tokens: i64,
     pub output_tokens: i64,
 }
 
@@ -641,13 +764,21 @@ impl TokenUsage {
     pub fn add(&mut self, other: TokenUsage) {
         self.input_tokens += other.input_tokens;
         self.cached_tokens += other.cached_tokens;
+        self.cache_write_tokens += other.cache_write_tokens;
         self.output_tokens += other.output_tokens;
     }
 
-    /// USD cost given the per-1M-token prices from `[deepseek]` config (§3.6).
+    /// USD cost given a provider's per-1M-token prices (§4.1–§4.2).
-    pub fn cost_usd(&self, price_input: f64, price_cached: f64, price_output: f64) -> f64 {
+    pub fn cost_usd(
+        &self,
+        price_input: f64,
+        price_cache_write: f64,
+        price_cache_read: f64,
+        price_output: f64,
+    ) -> f64 {
         (self.input_tokens as f64 * price_input
-            + self.cached_tokens as f64 * price_cached
+            + self.cache_write_tokens as f64 * price_cache_write
+            + self.cached_tokens as f64 * price_cache_read
             + self.output_tokens as f64 * price_output)
             / 1_000_000.0
     }
@@ -684,18 +815,6 @@ mod tests {
         assert_eq!(composite_social_score(&[]), 0.0);
     }
 
-    #[test]
-    fn feed_prior_is_beta_smoothed() {
-        assert_eq!(FeedPrior::default().rate(), 0.5);
-        let p = FeedPrior {
-            feed_id: 1,
-            upvotes: 3,
-            downvotes: 1,
-            included: 4,
-        };
-        assert!((p.rate() - 4.0 / 6.0).abs() < 1e-12);
-    }
-
     #[test]
     fn stats_line_and_reading_time() {
         assert_eq!(reading_minutes(0), 1);
@@ -730,8 +849,18 @@ mod tests {
 
     #[test]
     fn vote_and_social_source_round_trip() {
-        assert_eq!(Vote::parse("up"), Some(Vote::Up));
+        let feedback = crate::config::FeedbackConfig::default();
-        assert_eq!(Vote::Down.as_i64(), -1);
+        assert_eq!(Vote::parse("loved"), Some(Vote::Loved));
+        assert_eq!(Vote::parse("up"), Some(Vote::Loved));
+        assert_eq!(Vote::parse("good"), Some(Vote::Good));
+        assert_eq!(Vote::parse("down"), Some(Vote::NotForMe));
+        assert_eq!(Vote::Loved.as_str(), "loved");
+        assert_eq!(Vote::Good.as_str(), "good");
+        assert_eq!(Vote::NotForMe.as_str(), "down");
+        assert_eq!(Vote::Loved.value(&feedback), 1.0);
+        assert_eq!(Vote::Good.value(&feedback), 0.35);
+        assert_eq!(Vote::NotForMe.value(&feedback), -1.0);
+        assert_eq!(serde_json::to_string(&Vote::NotForMe).unwrap(), "\"down\"");
         assert_eq!(
             SocialSource::parse("lobsters"),
             Some(SocialSource::Lobsters)

src/world.rs

@@ -9,7 +9,7 @@ use serde::Deserialize;
 use url::Url;
 
 use crate::curate::llm::{LlmClient, LlmError};
-use crate::epub::images::text_escape;
+use crate::html::text_escape;
 use crate::types::{WorldBriefing, WorldBriefingSection, WorldEvent};
 
 type NodeRef<'a> = <scraper::ElementRef<'a> as std::ops::Deref>::Target;
@@ -160,11 +160,11 @@ fn dropped(element: &scraper::node::Element) -> bool {
 fn node_text(node: NodeRef<'_>, skip_lists: bool, out: &mut String) {
     match node.value() {
         Node::Text(text) => {
-            out.push_str(&text);
+            out.push_str(text);
             out.push(' ');
         }
         Node::Element(element) => {
-            if dropped(&element) || (skip_lists && matches!(element.name(), "ul" | "ol")) {
+            if dropped(element) || (skip_lists && matches!(element.name(), "ul" | "ol")) {
                 return;
             }
             for child in node.children() {
@@ -208,7 +208,7 @@ fn links_without_child_lists(element: ElementRef<'_>, base: &Url) -> Vec<String>
         let Node::Element(element) = node.value() else {
             return;
         };
-        if dropped(&element) || matches!(element.name(), "ul" | "ol") {
+        if dropped(element) || matches!(element.name(), "ul" | "ol") {
             return;
         }
         if element.name() == "a"
@@ -718,11 +718,14 @@ mod tests {
         backend: std::sync::Arc<crate::curate::llm::MockBackend>,
         limit: f64,
     ) -> LlmClient {
-        let config = crate::config::DeepseekConfig::default();
+        let config = crate::config::ProviderConfig::deepseek();
         LlmClient::with_backend(
             "mock",
             "World Briefing test".into(),
-            crate::curate::llm::UsageMeter::new(&config, limit),
+            crate::curate::llm::UsageMeter::with_prices(
+                crate::curate::llm::PriceTable::from(&config),
+                limit,
+            ),
             backend,
         )
     }

tests/config_check.rs

@@ -0,0 +1,69 @@
+//! `daily-epub config check` end to end: the built binary, a temp config, no
+//! environment. It must validate like `generate`, print the provider table
+//! with `key MISSING` warnings, exit 0, and exit non-zero on an invalid file —
+//! all without a database or the run lock.
+
+use std::path::Path;
+use std::process::Command;
+
+fn run(config_body: &str) -> (i32, String, String) {
+    let dir = tempfile::tempdir().expect("tempdir");
+    let path = dir.path().join("config.toml");
+    std::fs::write(&path, config_body).expect("write config");
+    let output = Command::new(env!("CARGO_BIN_EXE_daily-epub"))
+        .env_clear()
+        .env("PATH", std::env::var_os("PATH").unwrap_or_default())
+        .args(["--config"])
+        .arg(&path)
+        .args(["config", "check"])
+        .output()
+        .expect("run daily-epub");
+    (
+        output.status.code().unwrap_or(-1),
+        String::from_utf8_lossy(&output.stdout).into_owned(),
+        String::from_utf8_lossy(&output.stderr).into_owned(),
+    )
+}
+
+#[test]
+fn config_check_prints_the_facts_and_exits_zero_without_keys() {
+    let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml");
+    let body = std::fs::read_to_string(example).expect("example config");
+    let (code, stdout, stderr) = run(&body);
+    assert_eq!(code, 0, "stdout:\n{stdout}\nstderr:\n{stderr}");
+    for needle in [
+        "config: ",
+        "database_path: /var/lib/daily-epub/daily-epub.db",
+        "profile_path: data/profile.md",
+        "interests_opml: data/scour-interests.opml",
+        "llm.bulk: deepseek · openai · deepseek-v4-flash",
+        "key MISSING (set DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY)",
+        "llm.editor: anthropic · anthropic · claude-opus-5 · effort high · max_daily_usd $3.00",
+        "key MISSING (set DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY)",
+        "providers.gemini: unreferenced",
+        "voyage: voyage-4-lite · enabled · max_daily_usd $0.50 · key MISSING (set DAILY_EPUB_VOYAGE__API_KEY)",
+        "editorial.summary_model: editor",
+        "publish.epub_dir: /srv/bookorbit/libraries/daily-epub",
+        "publish.xtc_dir: /var/lib/daily-epub/xtc",
+    ] {
+        assert!(stdout.contains(needle), "missing {needle:?} in:\n{stdout}");
+    }
+    assert!(
+        stdout.lines().any(|line| line.starts_with("! ")),
+        "missing keys are flagged with a `!` prefix:\n{stdout}"
+    );
+    // No lock file, no database: the command is read-only.
+    assert!(!Path::new("/var/lib/daily-epub/daily-epub.db.lock").exists());
+}
+
+#[test]
+fn config_check_exits_non_zero_on_an_invalid_config() {
+    let (code, stdout, stderr) = run("[llm]\nbulk = \"nope\"\n");
+    assert_ne!(code, 0);
+    assert!(stdout.is_empty(), "{stdout}");
+    assert!(stderr.contains("providers.nope"), "{stderr}");
+
+    let (code, _, stderr) = run("[deepseek]\nmodel = \"x\"\n");
+    assert_ne!(code, 0);
+    assert!(stderr.contains("[providers.deepseek]"), "{stderr}");
+}

tests/e2e_pipeline.rs

@@ -2,14 +2,14 @@
 //! (spec §2, §5).
 //!
 //! ```text
-//! synthetic entries → dedupe → extract (offline) → persist → prefilter
+//! synthetic entries → dedupe → extract (offline) → persist → admission
 //!   → select → editorial → issue → both EPUB editions → publish → OPDS + rows
 //! ```
 //!
 //! Two passes over the same machinery:
 //!
 //! * [`skip_llm_pipeline_produces_a_published_issue`] takes the `--skip-llm`
-//!   route (prefilter order selects, feed excerpts stand in for summaries);
+//!   route (preliminary blend selects, feed excerpts stand in for summaries);
 //! * [`llm_pipeline_runs_against_a_mock_backend`] takes the DeepSeek route with
 //!   [`MockBackend`] standing in for the API, so stages A, B and C are all
 //!   exercised — prompts, parsers, budget accounting and all — offline.
@@ -18,18 +18,19 @@
 //! no article in the fixtures carries an image, so the EPUB builder's image
 //! downloader has nothing to fetch.
 
+use std::collections::{BTreeMap, HashMap};
 use std::path::Path;
 
 use jiff::Timestamp;
 use jiff::civil::Date;
 
 use daily_epub::config::{Config, PublishConfig, ServerConfig, XtcConfig};
-use daily_epub::curate::llm::{LlmClient, MockBackend, UsageMeter};
+use daily_epub::curate::llm::{LlmClient, Llms, MockBackend, UsageMeter};
-use daily_epub::curate::{Curator, editorial, prefilter};
+use daily_epub::curate::{Curator, admit, editorial, rank};
 use daily_epub::db::Db;
 use daily_epub::extract::Extractor;
 use daily_epub::types::{
-    Article, Colophon, Edition, Entry, Issue, Lineup, ScoredArticle, SourceKind, Vote,
+    Article, Candidate, Colophon, Edition, Entry, Issue, Lineup, Models, SourceKind, Vote,
 };
 use daily_epub::{auth, dedupe, epub, miniflux, pipeline, publish};
 
@@ -56,7 +57,6 @@ fn test_config(root: &Path) -> Config {
         database_path: root.join("db").join("daily-epub.db"),
         out_dir: root.join("out"),
         target_article_count: 6,
-        prefilter_keep: 20,
         world_briefing: false,
         publish: PublishConfig {
             epub_dir: root.join("bookorbit"),
@@ -283,7 +283,7 @@ async fn assemble_build_publish(
     )
     .expect("render chapters");
     let first = issue.lineup.picks[0].article.id;
-    let expected = auth::rating_url(&cfg.server.public_url, SECRET, date(), first, Vote::Up);
+    let expected = auth::rating_url(&cfg.server.public_url, SECRET, date(), first, Vote::Loved);
     let chapter = chapters
         .iter()
         .find(|c| c.id == format!("art-{}", issue.lineup.picks[0].article.best_entry_id))
@@ -298,8 +298,8 @@ async fn assemble_build_publish(
             SECRET,
             date(),
             first,
-            Vote::Up,
+            Vote::Loved,
-            &auth::rating_token(SECRET, date(), first, Vote::Up)
+            &auth::rating_token(SECRET, date(), first, Vote::Loved)
         ),
         "the server must accept the token the EPUB minted"
     );
@@ -400,24 +400,32 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
     let articles = ingest_dedupe_extract_persist(&db).await;
 
     // --- Stages 6–7 with no LLM at all (notes §6) ---
-    let curator = Curator::new(cfg.clone(), db.clone(), None);
+    let curator = Curator::new(cfg.clone(), db.clone(), Llms::default());
-    let candidates = curator
+    let mut personalized = articles
-        .prefilter(articles, date())
+        .into_iter()
-        .await
+        .map(|article| Candidate::new(article, false))
-        .expect("prefilter runs");
+        .collect::<Vec<_>>();
+    for candidate in &mut personalized {
+        candidate.signals.preliminary = candidate.signals.heuristic;
+    }
+    admit::admit(&mut personalized, date(), &cfg.curation.ranking);
+    let candidates = personalized
+        .into_iter()
+        .filter(|candidate| candidate.stage == "admitted")
+        .collect::<Vec<_>>();
     assert_eq!(candidates.len(), 5, "nothing is dropped at this volume");
-    assert!(
-        candidates
-            .windows(2)
-            .all(|w| w[0].prefilter_score >= w[1].prefilter_score),
-        "candidates come back in prefilter order"
-    );
     // The excerpt-only story is penalized (§3.5).
     let allocator = candidates
         .iter()
         .find(|c| c.article.excerpt_only)
         .expect("the allocator teaser survived");
-    assert!(allocator.prefilter_score < candidates[0].prefilter_score);
+    assert!(
+        allocator.signals.heuristic.unwrap_or_default()
+            < candidates
+                .iter()
+                .filter_map(|candidate| candidate.signals.heuristic)
+                .fold(f64::NEG_INFINITY, f64::max)
+    );
 
     let lineup = curator.select(candidates, date()).await.expect("select");
     assert_eq!(lineup.picks.len(), 5, "target 6, only 5 candidates exist");
@@ -432,7 +440,12 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
     );
 
     let colophon = Colophon {
-        model: "none (--skip-llm)".into(),
+        provider_costs: BTreeMap::new(),
+        models: Models {
+            bulk: "none".into(),
+            editor: "none".into(),
+            summaries: "none".into(),
+        },
         entries_fetched: 8,
         feeds_seen: 8,
         candidates: 5,
@@ -491,19 +504,28 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
         .expect("open db");
 
     let articles = ingest_dedupe_extract_persist(&db).await;
-    let ctx = prefilter::PrefilterContext::load(&db, date())
+    let mut personalized = articles
-        .await
+        .into_iter()
-        .expect("prefilter context");
+        .map(|article| Candidate::new(article, false))
-    let candidates: Vec<ScoredArticle> = prefilter::run(articles, &ctx, &cfg);
+        .collect::<Vec<_>>();
+    for candidate in &mut personalized {
+        candidate.signals.preliminary = candidate.signals.heuristic;
+    }
+    admit::admit(&mut personalized, date(), &cfg.curation.ranking);
+    let candidates = personalized
+        .into_iter()
+        .filter(|candidate| candidate.stage == "admitted")
+        .collect::<Vec<_>>();
     let ids: Vec<i64> = candidates.iter().map(|c| c.article.id).collect();
     assert_eq!(ids.len(), 5);
 
-    // --- Script DeepSeek: one stage-A batch, one stage-B call, five stage-C
+    // --- Script DeepSeek: one deep-assessment batch, one editor call, five
-    // summaries and one front page (§3.6). ---
+    // summaries and one brief. ---
     let backend = std::sync::Arc::new(MockBackend::new());
     let usage = daily_epub::types::TokenUsage {
         input_tokens: 1000,
         cached_tokens: 500,
+        cache_write_tokens: 0,
         output_tokens: 200,
     };
     let scores: Vec<String> = ids
@@ -511,8 +533,8 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
         .enumerate()
         .map(|(i, id)| {
             format!(
-                r#"{{"id": {id}, "score": {}, "category": "Tech & Engineering",
+                r#"{{"id": {id}, "quality": {}, "fit": 7, "category": "Tech & Engineering",
-                     "rationale": "solid systems writeup", "is_paywalled_guess": false}}"#,
+                     "rationale": "solid systems writeup", "paywalled_guess": false, "facets": {{"format":"analysis_essay"}}}}"#,
                 9 - i
             )
         })
@@ -544,31 +566,37 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
         );
     }
     backend.push(
-        r#"{"from_the_editor": "Today's issue leans on storage internals.\n\nRead on.",
+        r#"{"brief": "Today's issue leans on storage internals.\n\nRead on."}"#,
-            "section_intros": {"Top Stories": "The day in one place."}}"#,
         usage,
     );
 
-    let meter = UsageMeter::new(&cfg.deepseek, cfg.max_daily_usd);
+    let meter = UsageMeter::for_provider(&cfg.providers["deepseek"]);
     let llm = LlmClient::with_backend(
-        &cfg.deepseek.model,
+        &cfg.providers["deepseek"].model,
         "You are the editor of The Daily EPUB.".into(),
         meter.clone(),
         backend.clone(),
     );
-    let curator = Curator::new(cfg.clone(), db.clone(), Some(llm));
+    let curator = Curator::new(
+        cfg.clone(),
+        db.clone(),
+        Llms {
+            bulk: Some(llm),
+            editor: None,
+        },
+    );
 
     let mut candidates = candidates;
     curator
-        .score(&mut candidates, date())
+        .assess(&mut candidates, true, None, jiff::Timestamp::now())
         .await
-        .expect("stage A");
+        .expect("deep assessment");
     assert!(
-        candidates.iter().all(|c| c.llm.is_some()),
+        candidates.iter().all(|c| c.assessment.deep.is_some()),
-        "every candidate came back scored"
+        "every candidate came back assessed"
     );
 
-    let lineup = curator.select(candidates, date()).await.expect("stage B");
+    let lineup = curator.select(candidates, date()).await.expect("editor");
     assert_eq!(lineup.picks.len(), 5);
     assert_eq!(
         lineup.lead().map(|p| p.article.id),
@@ -590,12 +618,9 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
         "the model's summaries were used, not excerpts"
     );
     assert!(editorial_doc.front_page_html.contains("storage internals"));
-    assert_eq!(
+    assert!(
-        editorial_doc
+        lineup.picks.iter().all(|p| p.why.is_none()),
-            .section_intros
+        "the scripted editor gave no why lines"
-            .get("Top Stories")
-            .map(String::as_str),
-        Some("The day in one place.")
     );
 
     // Every scripted response was consumed, and the meter priced them (§3.6).
@@ -615,7 +640,12 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
 
     // And it all assembles, builds and publishes like the skip-llm route does.
     let colophon = Colophon {
-        model: cfg.deepseek.model.clone(),
+        provider_costs: BTreeMap::from([("deepseek".to_string(), meter.cost_usd())]),
+        models: Models {
+            bulk: cfg.providers["deepseek"].model.clone(),
+            editor: format!("{} (bulk fallback)", cfg.providers["deepseek"].model),
+            summaries: cfg.providers["deepseek"].model.clone(),
+        },
         entries_fetched: 8,
         feeds_seen: 8,
         candidates: 5,
@@ -625,6 +655,95 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
     let mut lineup = lineup;
     pipeline::apply_summaries(&mut lineup, &editorial_doc);
     let issue = assemble_build_publish(&db, &cfg, lineup, colophon).await;
-    assert_eq!(issue.colophon.model, cfg.deepseek.model);
+    assert_eq!(issue.colophon.models.bulk, cfg.providers["deepseek"].model);
     assert!(issue.colophon.cost_usd > 0.0);
 }
+
+#[tokio::test]
+async fn failing_deepseek_still_publishes_with_heuristic_fallbacks() {
+    let root = tempfile::tempdir().expect("tempdir");
+    let cfg = test_config(root.path());
+    let db = Db::open_and_migrate(&cfg.database_path)
+        .await
+        .expect("open db");
+    let articles = ingest_dedupe_extract_persist(&db).await;
+    let mut personalized = articles
+        .into_iter()
+        .map(|article| Candidate::new(article, false))
+        .collect::<Vec<_>>();
+    for candidate in &mut personalized {
+        candidate.signals.preliminary = candidate.signals.heuristic;
+    }
+    admit::admit(&mut personalized, date(), &cfg.curation.ranking);
+    let mut candidates = personalized
+        .into_iter()
+        .filter(|candidate| candidate.stage == "admitted")
+        .collect::<Vec<_>>();
+
+    let backend = std::sync::Arc::new(MockBackend::new());
+    let client = LlmClient::with_backend(
+        &cfg.providers["deepseek"].model,
+        "reader profile".into(),
+        UsageMeter::for_provider(&cfg.providers["deepseek"]),
+        backend.clone(),
+    );
+    let curator = Curator::new(
+        cfg.clone(),
+        db.clone(),
+        Llms {
+            bulk: Some(client),
+            editor: None,
+        },
+    );
+    curator
+        .assess(&mut candidates, true, None, jiff::Timestamp::now())
+        .await
+        .expect("failed batches degrade, not abort");
+    assert!(
+        candidates
+            .iter()
+            .all(|candidate| candidate.assessment.deep.is_none())
+    );
+    // Utility over the present signals, clustered as singletons without
+    // embeddings: every admitted article is shortlisted with a cluster id.
+    let ranked = rank::shortlist(&mut candidates, &HashMap::new(), &cfg.curation.ranking);
+    assert_eq!(ranked.shortlisted, candidates.len());
+    assert_eq!(ranked.clusters, candidates.len());
+    for candidate in &candidates {
+        assert_eq!(candidate.stage, "shortlisted");
+        assert!(candidate.utility.is_some() && candidate.cluster.is_some());
+        assert!(candidate.rank_utility.is_some());
+    }
+    let mut lineup = curator
+        .select(candidates, date())
+        .await
+        .expect("fallback lineup");
+    let editorial = curator
+        .editorial(&lineup)
+        .await
+        .expect("fallback editorial");
+    pipeline::apply_summaries(&mut lineup, &editorial);
+    assert!(!lineup.picks.is_empty());
+    assert!(backend.calls() > 0, "the failing backend was exercised");
+    let issue = assemble_build_publish(
+        &db,
+        &cfg,
+        lineup,
+        Colophon {
+            provider_costs: BTreeMap::new(),
+            models: Models {
+                bulk: cfg.providers["deepseek"].model.clone(),
+                editor: format!("{} (bulk fallback)", cfg.providers["deepseek"].model),
+                summaries: cfg.providers["deepseek"].model.clone(),
+            },
+            entries_fetched: 8,
+            feeds_seen: 8,
+            candidates: 5,
+            cost_usd: 0.0,
+            generator_version: format!("daily-epub {}", daily_epub::VERSION),
+        },
+    )
+    .await;
+    assert!(!issue.lineup.picks.is_empty());
+    assert_eq!(std::fs::read_dir(&cfg.publish.epub_dir).unwrap().count(), 2);
+}

tests/fixtures/claude_brief.json

@@ -0,0 +1,3 @@
+{
+  "brief": "The lead, \"Migrating 40TB off Postgres\", is the rare migration write-up that keeps its failures in: two aborted cutovers, the rollback that took longer than the move, and a bill at the end. Read it first while the coffee is hot; it rewards attention and it is long.\n\nThe local desk answers with \"The MBTA slow-zone dataset\", which finally puts the T's own numbers into a shape a rider can argue with, and the charts do more persuading than a year of press releases. \"A failover story you'd argue with\" rounds out the engineering pages with a Postgres HA design that disagrees with the lead on almost every point, which is exactly why the two belong in the same issue. The issue is shorter than usual because a thin Friday is a good excuse to finish the long one properly rather than skim six."
+}

tests/fixtures/deepseek_deep_batch.json

@@ -0,0 +1,76 @@
+{
+  "articles": [
+    {
+      "id": 101,
+      "quality": 8.5,
+      "fit": 7.0,
+      "category": "Tech & Engineering",
+      "rationale": "First-hand 40TB Postgres migration with failure timeline, numbers and a rollback plan",
+      "paywalled_guess": false,
+      "facets": {
+        "format": "first_hand_account",
+        "depth": "deep",
+        "evidence": "first_hand",
+        "commerciality": "none",
+        "topic_group": "software_engineering",
+        "technicality": "advanced",
+        "locality": "not_applicable",
+        "specific_topics": ["Postgres migration", "storage failover", "rollback planning"]
+      }
+    },
+    {
+      "id": 102,
+      "quality": 3.0,
+      "fit": 2.5,
+      "category": "AI & Machine Learning",
+      "rationale": "Model release announcement, benchmark table from the vendor, no independent evaluation",
+      "paywalled_guess": false,
+      "facets": {
+        "format": "announcement_roundup",
+        "depth": "brief",
+        "evidence": "speculative",
+        "commerciality": "promotional",
+        "topic_group": "ai_ml",
+        "technicality": "light",
+        "locality": "international",
+        "specific_topics": ["model launch"]
+      }
+    },
+    {
+      "id": 103,
+      "quality": 6.5,
+      "fit": 8.0,
+      "category": "Boston & Local",
+      "rationale": "MBTA slow-zone data pulled from the tracker and charted by line with original analysis",
+      "paywalled_guess": false,
+      "facets": {
+        "format": "analysis_essay",
+        "depth": "standard",
+        "evidence": "data_or_experiment",
+        "commerciality": "none",
+        "topic_group": "boston_new_england",
+        "technicality": "intermediate",
+        "locality": "boston_new_england",
+        "specific_topics": ["MBTA slow zones", "transit data"]
+      }
+    },
+    {
+      "id": 104,
+      "quality": 5.0,
+      "fit": 6.0,
+      "category": "Culture & Essays",
+      "rationale": "Promising essay on typesetting history; the body reads cut off after the second section",
+      "paywalled_guess": true,
+      "facets": {
+        "format": "analysis_essay",
+        "depth": "standard",
+        "evidence": "synthesis",
+        "commerciality": "none",
+        "topic_group": "books_writing",
+        "technicality": "nontechnical",
+        "locality": "not_applicable",
+        "specific_topics": ["typesetting", "print history"]
+      }
+    }
+  ]
+}

tests/fixtures/deepseek_deep_batch_messy.json

@@ -0,0 +1,67 @@
+{
+  "articles": [
+    {
+      "id": 201,
+      "quality": 7.0,
+      "fit": 6.5,
+      "category": "Science & Space",
+      "rationale": "careful write-up of an amateur radio occultation measurement",
+      "paywalled_guess": false,
+      "facets": {
+        "format": "first_hand_account",
+        "depth": "standard",
+        "evidence": "data_or_experiment",
+        "commerciality": "none",
+        "topic_group": "science_space",
+        "technicality": "intermediate",
+        "locality": "us",
+        "specific_topics": ["radio occultation"]
+      }
+    },
+    {
+      "id": "202",
+      "quality": "6",
+      "fit": "5.5",
+      "category": "Niche Corner",
+      "rationale": "mailing-list argument about tape drives, oddly gripping",
+      "paywalled_guess": "false",
+      "facets": {
+        "format": "discussion_thread",
+        "depth": "standard",
+        "evidence": "anecdote",
+        "commerciality": "none",
+        "topic_group": "retro_computing",
+        "technicality": "intermediate",
+        "locality": "not_applicable",
+        "specific_topics": ["LTO tape", "backups", "archival", "vendors", "pricing"]
+      }
+    },
+    {
+      "id": 203,
+      "quality": 4,
+      "fit": 4
+    },
+    {
+      "id": 204,
+      "quality": 12.5,
+      "fit": -1,
+      "category": "Sports",
+      "rationale": "model ignored the rubric ceiling and invented a section here",
+      "paywalled_guess": false,
+      "facets": "not an object"
+    },
+    {
+      "id": 205,
+      "quality": 8.0,
+      "category": "Top Stories",
+      "rationale": "fit is missing, so this item is unusable"
+    },
+    {
+      "quality": 9.0,
+      "fit": 9.0,
+      "category": "Top Stories",
+      "rationale": "no id at all, unusable"
+    },
+    "a bare string where an object belongs"
+  ]
+}

tests/fixtures/deepseek_front_page.json

@@ -1,8 +0,0 @@
-{
-  "from_the_editor": "Two of today's pieces are, underneath, the same story: what it costs to move data you no longer trust. The lead — a team hauling forty terabytes off Postgres, rollback plans and all — is the version with the invoices attached, and it earns the front page by refusing to tidy up its failures. Read it first, while the coffee is hot; it rewards attention and it is long.\n\nThe local desk supplies the counterpoint. Somebody has finally put the MBTA's slow-zone data into a shape a rider can argue with, and the charts do more persuading than a year of press releases. It is a short read and a satisfying one, and it pairs unreasonably well with the migration story: both are about institutions discovering what they actually have.\n\nThe rest of the issue is quieter than usual. That is not a complaint — a thin Friday is a good excuse to finish the long one properly rather than skimming six. If you only get through the lead today, you will not have missed much else.",
-  "section_intros": {
-    "Top Stories": "The day's most substantial piece: a full account of a forty-terabyte migration, with the failures left in. It is long, technical and unusually honest about what went wrong.",
-    "Boston & Local": "Transit data gets the treatment it deserves. A rider-built analysis of MBTA slow zones, with charts you can check yourself and a methodology section that holds up.",
-    "Niche Corner": "A section the model wrote an intro for even though nothing was placed in it today — a stray thread about tape-drive firmware and the people who still maintain it. The issue drops intros for sections that never ran."
-  }
-}

tests/fixtures/deepseek_lineup.json

@@ -1,10 +1,10 @@
 {
   "picks": [
-    { "id": 101, "section": "Top Stories", "position": 1, "lead_story": true },
+    { "id": 101, "section": "Top Stories", "position": 1, "lead_story": true, "why": "The migration post-mortem with the invoices still attached" },
-    { "id": 102, "section": "Tech & Engineering", "position": 1, "lead_story": false },
+    { "id": 102, "section": "Tech & Engineering", "position": 1, "lead_story": false, "why": "A failover story you'd argue with over coffee" },
-    { "id": 103, "section": "Tech & Engineering", "position": 2, "lead_story": false },
+    { "id": 103, "section": "Tech & Engineering", "position": 2, "lead_story": false, "why": "Rare first-hand detail on a tool you use daily" },
-    { "id": 104, "section": "AI & Machine Learning", "position": 1, "lead_story": false },
+    { "id": 104, "section": "AI & Machine Learning", "position": 1, "lead_story": false, "why": "The one benchmark piece this week that shows its work" },
-    { "id": 105, "section": "Boston & Local", "position": 1, "lead_story": false },
+    { "id": 105, "section": "Boston & Local", "position": 1, "lead_story": false, "why": "MBTA slow zones charted by a rider, not a press office" },
     { "id": 106, "section": "Culture & Essays", "position": 1 },
     { "section": "Niche Corner", "position": 2, "lead_story": false },
     "the model sometimes trails off like this"

tests/fixtures/deepseek_score_batch.json

@@ -1,32 +0,0 @@
-{
-  "articles": [
-    {
-      "id": 101,
-      "score": 8.5,
-      "category": "Tech & Engineering",
-      "rationale": "first-hand 40TB Postgres migration with numbers, failures and rollback plan",
-      "is_paywalled_guess": false
-    },
-    {
-      "id": 102,
-      "score": 3.0,
-      "category": "AI & Machine Learning",
-      "rationale": "model release announcement, no independent evaluation",
-      "is_paywalled_guess": false
-    },
-    {
-      "id": 103,
-      "score": 6.5,
-      "category": "Boston & Local",
-      "rationale": "MBTA slow-zone data analysis with original charts",
-      "is_paywalled_guess": false
-    },
-    {
-      "id": 104,
-      "score": 5.0,
-      "category": "Culture & Essays",
-      "rationale": "promising essay on typesetting, body appears truncated",
-      "is_paywalled_guess": true
-    }
-  ]
-}

tests/fixtures/deepseek_score_batch_messy.json

@@ -1,35 +0,0 @@
-{
-  "articles": [
-    {
-      "id": 201,
-      "score": 7.0,
-      "category": "Science & Space",
-      "rationale": "careful write-up of an amateur radio occultation measurement",
-      "is_paywalled_guess": false
-    },
-    {
-      "id": "202",
-      "score": "6",
-      "category": "Niche Corner",
-      "rationale": "mailing-list argument about tape drives, oddly gripping",
-      "is_paywalled_guess": "false"
-    },
-    {
-      "id": 203,
-      "score": 4
-    },
-    {
-      "id": 204,
-      "score": 12.5,
-      "category": "Top Stories",
-      "rationale": "model ignored the rubric ceiling here",
-      "is_paywalled_guess": false
-    },
-    {
-      "score": 9.0,
-      "category": "Top Stories",
-      "rationale": "no id at all, unusable"
-    },
-    "a bare string where an object belongs"
-  ]
-}

tests/fixtures/deepseek_triage_batch.json

@@ -0,0 +1,16 @@
+{
+  "articles": [
+    {
+      "id": 4821,
+      "interest": 7.5,
+      "kind": "first_hand",
+      "why": "Specific field notes from a difficult migration"
+    },
+    {
+      "id": 4822,
+      "interest": 3.0,
+      "kind": "news",
+      "why": "Competent but familiar daily industry coverage"
+    }
+  ]
+}

tests/m3_curation.rs

@@ -3,8 +3,8 @@
 //! The stage logic is unit-tested inside `src/curate/*`. What this file guards is
 //! the contract *between* the curation stages and everything around them:
 //!
-//! * the recorded DeepSeek fixtures still parse through the real
+//! * recorded DeepSeek fixtures still parse through the real
-//!   `score.rs` / `select.rs` / `editorial.rs` parsers into the structures the
+//!   `assess.rs` / `editor.rs` / `editorial.rs` parsers into the structures the
 //!   pipeline consumes, and the lenient parsers still cope with the messy one;
 //! * the shipped Scour OPML still yields the ~220 interests the taste profile is
 //!   assembled from (§3.6a);
@@ -17,11 +17,10 @@ use std::collections::BTreeSet;
 use std::path::{Path, PathBuf};
 use std::process::Command;
 
-use daily_epub::curate::editorial::FrontPageResponse;
+use daily_epub::curate::assess::parse_deep_response;
+use daily_epub::curate::editor::parse_selection_response;
+use daily_epub::curate::editorial::BriefResponse;
 use daily_epub::curate::profile;
-use daily_epub::curate::score::parse_score_response;
-use daily_epub::curate::select::parse_selection_response;
-use daily_epub::types::LlmScore;
 
 fn repo(rel: &str) -> PathBuf {
     Path::new(env!("CARGO_MANIFEST_DIR")).join(rel)
@@ -32,71 +31,88 @@ fn fixture(name: &str) -> String {
     std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("reading {}: {e}", path.display()))
 }
 
-/// Stage A responses must parse into `{id, score, category, rationale,
+/// Deep responses must parse into `{id, quality, fit, category, rationale,
-/// is_paywalled_guess}` per article (§3.6).
+/// paywalled_guess, facets}` per article (§12.1).
 #[test]
-fn stage_a_fixture_parses_into_scores() {
+fn deep_fixture_parses_into_assessments() {
-    let items = parse_score_response(&fixture("deepseek_score_batch.json"));
+    let sections = daily_epub::config::CurationConfig::default().sections;
+    let items = parse_deep_response(&fixture("deepseek_deep_batch.json"), &sections);
     assert!(items.len() >= 4, "fixture should cover a realistic batch");
 
     for item in &items {
         assert!(item.id > 0, "every item carries a positive article id");
+        assert!((0.0..=10.0).contains(&item.quality));
+        assert!((0.0..=10.0).contains(&item.fit));
+        assert!(item.category.is_some(), "every category is on the palette");
         assert!(
-            (0.0..=10.0).contains(&item.score),
+            item.rationale.split_whitespace().count() <= 25,
-            "score {} out of range",
+            "rationale must stay under 25 words: {:?}",
-            item.score
-        );
-        assert!(!item.category.is_empty());
-        assert!(
-            item.rationale.split_whitespace().count() <= 20,
-            "rationale must stay under 20 words: {:?}",
             item.rationale
         );
+        assert!(item.facets.format.is_some() && item.facets.topic_group.is_some());
     }
     assert!(
-        items.iter().any(|i| i.is_paywalled_guess),
+        items.iter().any(|item| item.paywalled_guess),
         "the fixture should exercise the paywall flag"
     );
+    // Quality and fit are judged separately: the fixture has an article whose
+    // fit exceeds its quality and one the other way round.
+    assert!(items.iter().any(|item| item.fit > item.quality));
+    assert!(items.iter().any(|item| item.quality > item.fit));
 
     // The batch spans the rubric rather than clustering at one score.
-    let scores: Vec<f64> = items.iter().map(|i| i.score).collect();
+    let qualities: Vec<f64> = items.iter().map(|i| i.quality).collect();
-    let spread = scores.iter().cloned().fold(f64::MIN, f64::max)
+    let spread = qualities.iter().cloned().fold(f64::MIN, f64::max)
-        - scores.iter().cloned().fold(f64::MAX, f64::min);
+        - qualities.iter().cloned().fold(f64::MAX, f64::min);
     assert!(spread >= 3.0, "fixture scores are too uniform to be useful");
-
-    // Every item converts into the shared curation type.
-    let converted: Vec<LlmScore> = items.into_iter().map(LlmScore::from).collect();
-    assert!(converted.iter().all(|s| (0.0..=10.0).contains(&s.score)));
 }
 
 /// The messy fixture must stay messy: it is what proves the parser is lenient
-/// (string ids, string scores, out-of-range scores, junk entries).
+/// (string ids and scores, out-of-range scores, unknown facet tokens, junk).
 #[test]
-fn stage_a_messy_fixture_is_salvaged_not_rejected() {
+fn deep_messy_fixture_is_salvaged_not_rejected() {
-    let raw = fixture("deepseek_score_batch_messy.json");
+    let sections = daily_epub::config::CurationConfig::default().sections;
-    // The hard cases are still present in the recording…
+    let raw = fixture("deepseek_deep_batch_messy.json");
     assert!(raw.contains("\"id\": \""), "needs a string id");
-    assert!(raw.contains("\"score\": \""), "needs a string score");
+    assert!(raw.contains("\"quality\": \""), "needs a string score");
+    assert!(
+        raw.contains("discussion_thread"),
+        "needs an unknown facet token"
+    );
 
-    // …and the real parser copes with all of them.
+    let items = parse_deep_response(&raw, &sections);
-    let items = parse_score_response(&raw);
     assert!(!items.is_empty(), "the parser salvaged nothing");
     assert!(
-        items.iter().all(|i| (0.0..=10.0).contains(&i.score)),
+        items
-        "out-of-range scores must be clamped: {:?}",
+            .iter()
-        items.iter().map(|i| i.score).collect::<Vec<_>>()
+            .all(|i| (0.0..=10.0).contains(&i.quality) && (0.0..=10.0).contains(&i.fit)),
+        "out-of-range scores must be clamped"
     );
     assert!(items.iter().all(|i| i.id > 0), "id-less items are skipped");
+    let messy = items
+        .iter()
+        .find(|i| i.id == 202)
+        .expect("string-valued item");
+    assert!(
+        messy.facets.format.is_none(),
+        "unknown facet tokens become None"
+    );
+    assert_eq!(messy.facets.depth.as_deref(), Some("standard"));
+    let off_palette = items.iter().find(|i| i.id == 204).expect("clamped item");
+    assert!(
+        off_palette.category.is_none(),
+        "invented sections become None"
+    );
 
-    // A response that is not JSON at all degrades to "no scores", never a panic.
+    // A response that is not JSON at all degrades to "no assessments", never a panic.
-    assert!(parse_score_response("I'm sorry, I can't do that.").is_empty());
+    assert!(parse_deep_response("I'm sorry, I can't do that.", &sections).is_empty());
-    assert!(parse_score_response("").is_empty());
+    assert!(parse_deep_response("", &sections).is_empty());
 }
 
-/// Stage B responses must carry `{id, section, position, lead_story}` with
+/// Editor responses must carry `{id, section, position, lead_story}` with
-/// exactly one lead, and use only palette section names (§3.6).
+/// exactly one lead, and use only palette section names (plan §13).
 #[test]
-fn stage_b_fixture_parses_into_a_lineup() {
+fn editor_fixture_parses_into_a_lineup() {
     // The palette from `CurationConfig::default()` (§3.14).
     let sections = daily_epub::config::CurationConfig::default().sections;
 
@@ -130,35 +146,41 @@ fn stage_b_fixture_parses_into_a_lineup() {
     );
 }
 
-/// Stage C's front-page response must deserialize into a 250–400 word editor's
+/// The Brief must deserialize into 120–200 words of plain prose that names at
-/// note plus per-section intros (§3.6).
+/// least three picks by title (§14.2). Section intros are gone.
 #[test]
-fn stage_c_fixture_parses_into_a_front_page() {
+fn stage_c_fixture_parses_into_the_brief() {
-    let response: FrontPageResponse = serde_json::from_str(&fixture("deepseek_front_page.json"))
+    let response: BriefResponse = serde_json::from_str(&fixture("claude_brief.json"))
-        .expect("the front-page fixture must match FrontPageResponse");
+        .expect("the brief fixture must match BriefResponse");
 
-    let words = response.from_the_editor.split_whitespace().count();
+    let words = response.brief.split_whitespace().count();
     assert!(
-        (150..=450).contains(&words),
+        (100..=220).contains(&words),
-        "From the Editor is {words} words; the prompt asks for 250-400"
+        "The Brief is {words} words; the prompt asks for 120-200"
     );
     assert!(
-        response.from_the_editor.contains("\n\n"),
+        !response.brief.contains("- ") && !response.brief.contains('#'),
-        "the prompt asks for 2-4 blank-line separated paragraphs"
+        "no bullets or headings in the brief"
     );
+    let titles = response.brief.matches('"').count() / 2;
     assert!(
-        !response.from_the_editor.contains("- "),
+        titles >= 3,
-        "no bullet lists on the front page"
+        "the brief names at least three picks; found {titles}"
     );
-
+    for banned in [
-    assert!(response.section_intros.len() >= 2);
+        "delve",
-    for (section, intro) in &response.section_intros {
+        "dive",
-        let words = intro.split_whitespace().count();
+        "explore",
+        "a mix of",
+        "something for everyone",
+    ] {
         assert!(
-            (10..=90).contains(&words),
+            !response.brief.to_lowercase().contains(banned),
-            "intro for {section} is {words} words; the prompt asks for 35-60"
+            "banned phrase {banned}"
         );
     }
+    let value: serde_json::Value = serde_json::from_str(&fixture("claude_brief.json")).unwrap();
+    assert!(value.get("section_intros").is_none());
 }
 
 /// The taste profile is seeded from this file; a broken export would silently
@@ -190,7 +212,18 @@ fn scour_opml_still_yields_the_interest_list() {
 
     // The assembled profile is what actually reaches DeepSeek as the system
     // prompt; it must mention the stated preferences and the interests (§3.6).
-    let document = profile::build(&interests, profile::NO_LEARNED_ADJUSTMENTS);
+    let profile_file = profile::load_profile(Path::new(concat!(
+        env!("CARGO_MANIFEST_DIR"),
+        "/data/profile.md"
+    )))
+    .expect("profile file");
+    let document = profile::build(
+        &profile_file.body,
+        &interests,
+        profile::NO_LEARNED_ADJUSTMENTS,
+        &[],
+        60,
+    );
     assert!(document.contains("Rust"));
     assert!(document.len() > 1000, "the profile is suspiciously short");
 }

tests/m4_epub.rs

@@ -89,6 +89,7 @@ fn standard_edition_is_a_well_formed_epub3_archive() {
         "OEBPS/sec-niche-corner.xhtml",
         "OEBPS/art-1002.xhtml",
         "OEBPS/world.xhtml",
+        "OEBPS/behind.xhtml",
         "OEBPS/colophon.xhtml",
     ] {
         assert!(contains_entry(&zip, entry), "missing {entry}");
@@ -186,11 +187,77 @@ fn chapter_ids_hrefs_and_toc_levels_are_stable() {
             ),
             ("art-1002".into(), "art-1002.xhtml".into(), 2),
             ("world".into(), "world.xhtml".into(), 1),
+            ("behind".into(), "behind.xhtml".into(), 1),
             ("colophon".into(), "colophon.xhtml".into(), 1),
         ]
     );
 }
 
+/// "Behind the paper" sits between the World Briefing and the colophon in both
+/// editions, carries the counts, the admission mix, the learned-signal state,
+/// the near misses and the models, and has no links (§15.1).
+#[test]
+fn behind_the_paper_renders_counts_and_near_misses_in_both_editions() {
+    let issue = fixtures::issue();
+    for edition in [Edition::Standard, Edition::X4] {
+        let (_dir, _, zip) = build_edition_to_bytes(&issue, edition);
+        let behind = read_entry(&zip, "OEBPS/behind.xhtml");
+        assert!(behind.contains("<h1>Behind the paper</h1>"), "{behind}");
+        assert!(
+            behind.contains(
+                "Considered 412 articles from 1,465 feeds \u{00b7} 398 eligible \u{00b7} 398 triaged \u{00b7} 120 read closely \u{00b7} 60 shortlisted \u{00b7} 2 selected."
+            ),
+            "{behind}"
+        );
+        assert!(
+            behind.contains(
+                "Admitted via: triage 60 \u{00b7} interests 20 \u{00b7} your ratings 12 \u{00b7} exploration 5 \u{00b7} blend 23."
+            ),
+            "{behind}"
+        );
+        assert!(
+            behind.contains(
+                "Learned signals: 14 rated articles with embeddings (neighbour signal at 35%); feed affinity off."
+            ),
+            "{behind}"
+        );
+        assert!(behind.contains("<h2>Near misses</h2>"), "{behind}");
+        assert!(
+            behind.contains(
+                "<li>The One That Got Away \u{2014} Example Feed \u{00b7} quality 8.0 \u{00b7} fit 6.5 \u{00b7} shortlisted, not selected</li>"
+            ),
+            "{behind}"
+        );
+        assert!(
+            behind.contains(
+                "<li>Never Read Closely \u{2014} Other Feed \u{00b7} triaged, not admitted</li>"
+            ),
+            "{behind}"
+        );
+        assert!(
+            behind.contains(
+                "Models: triage and assessment deepseek-v4-flash \u{00b7} editor and summaries claude-opus-5 \u{00b7} embeddings voyage-4-lite. Cost $0.81. Generation 23 min."
+            ),
+            "{behind}"
+        );
+        assert!(!behind.contains("<a "), "no links in {edition:?}: {behind}");
+        assert!(!behind.contains("&nbsp;"));
+        for tag in ["html", "head", "body", "div", "p", "ul", "li", "h1", "h2"] {
+            // `<li` would also match `<link`; count real opening tags only.
+            let opens = behind.matches(&format!("<{tag}>")).count()
+                + behind.matches(&format!("<{tag} ")).count();
+            let closes = behind.matches(&format!("</{tag}>")).count();
+            assert_eq!(opens, closes, "unbalanced <{tag}> in {edition:?}");
+        }
+
+        // Ordering: world → behind → colophon in the spine.
+        let opf = read_entry(&zip, "OEBPS/content.opf");
+        let position = |href: &str| opf.find(&format!("href=\"{href}\"")).expect(href);
+        assert!(position("world.xhtml") < position("behind.xhtml"));
+        assert!(position("behind.xhtml") < position("colophon.xhtml"));
+    }
+}
+
 #[test]
 fn every_chapter_is_parseable_xhtml() {
     let issue = fixtures::issue();
@@ -270,16 +337,20 @@ fn rating_links_carry_the_spec_token() {
     .expect("article chapter");
 
     let date = issue.meta.date;
-    let up = build::rating_token("integration-secret", date, 1, Vote::Up);
+    let loved = build::rating_token("integration-secret", date, 1, Vote::Loved);
-    let down = build::rating_token("integration-secret", date, 1, Vote::Down);
+    let good = build::rating_token("integration-secret", date, 1, Vote::Good);
-    assert_eq!(up.len(), 16);
+    let down = build::rating_token("integration-secret", date, 1, Vote::NotForMe);
-    assert_ne!(up, down);
+    assert_eq!(loved.len(), 16);
-    assert!(chapter.xhtml.contains(&format!(
+    assert_ne!(loved, good);
-        "https://daily.hallada.net/r/2026-08-15/1/up?t={up}"
+    assert_ne!(good, down);
-    )));
+    for (segment, token) in [("loved", loved), ("good", good), ("down", down)] {
-    assert!(chapter.xhtml.contains(&format!(
+        assert!(chapter.xhtml.contains(&format!(
-        "https://daily.hallada.net/r/2026-08-15/1/down?t={down}"
+            "https://daily.hallada.net/r/2026-08-15/1/{segment}?t={token}"
-    )));
+        )));
+    }
+    assert!(chapter.xhtml.contains("[ Loved it ]"));
+    assert!(chapter.xhtml.contains("[ Good ]"));
+    assert!(chapter.xhtml.contains("[ Not for me ]"));
 }
 
 #[test]
@@ -316,7 +387,14 @@ fn colophon_facts_are_x4_safe_distinct_paragraphs() {
     for edition in [Edition::Standard, Edition::X4] {
         let (_dir, _, zip) = build_edition_to_bytes(&issue, edition);
         let colophon = read_entry(&zip, "OEBPS/colophon.xhtml");
-        assert_eq!(colophon.matches("<p class=\"fact-line\">").count(), 9);
+        // Issue, generated, three model lines, entries, candidates, articles,
+        // words, two per-provider cost lines, the total, generator (§15.1).
+        assert_eq!(colophon.matches("<p class=\"fact-line\">").count(), 13);
+        assert!(colophon.contains("<strong>Editor model:</strong> claude-opus-5"));
+        assert!(colophon.contains("<strong>Bulk model:</strong> deepseek-v4-flash"));
+        assert!(colophon.contains("<strong>anthropic cost:</strong> $0.0500"));
+        assert!(colophon.contains("<strong>deepseek cost:</strong> $0.0231"));
+        assert!(colophon.contains("<strong>Total token cost:</strong>"));
         assert!(!colophon.contains("<dl"));
         assert!(!colophon.contains("<dt"));
         assert!(!colophon.contains("<dd"));