the-daily-epub/docs/plans/2026-09-02-curation-v2-progress.md

# Personalized Curation v2 — implementation progress and handoff

**Updated:** 2026-09-02 (end of session 2)
**Plan:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§21 is the step sequence)
**Branch:** `curation-v2` (branched from `main` at `a599745`; not merged, not pushed)
**Briefs:** `docs/plans/curation-v2-briefs/` — `00-preamble.md` + one `stepN.md` per step. Each
brief was handed to an implementation agent as `cat 00-preamble.md stepN.md`.

## Where things stand

| Step (§21) | Status | Commit |
|---|---|---|
| 1. Feedback and profile | **done**, reviewed (codex review: no actionable issues) | `3a9f4b9` |
| 2. Claude editor and editorial | **done** | `57efbb4` |
| 3. Embeddings, signals, telemetry | **done** (branch `curation-v2-step3`, merged) | `ea3b141` + merge commit |
| 4. Triage replaces the gate | **done** (Codex plugin, high effort), reviewed | `10f4afd` |
| 5. Deep assessment, utility, diversity | **done** (Codex started, cut off by quota; Claude agent finished), reviewed | `05a74a0` |
| 6. Paper telemetry, stats, lock | **done** (Claude agent), reviewed | `d261cd4` |
| 7. Cleanup + implementation notes | **done** (Claude agent), reviewed | `d403c51` |

**Post-plan addition (same day):** a provider-agnostic LLM registry — `[llm]` roles (`bulk`,
`editor`) over named `[providers.*]` entries of `kind = openai | anthropic`, keys from
`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY`, a `gemini` entry (Gemini 3.8 Flash via Google's
OpenAI-compatible endpoint) declared but unreferenced, and a `daily-epub config check` subcommand.
The old `[deepseek]`/`[anthropic]` tables, top-level `max_daily_usd` and the old key env vars
fail loudly. The server upgrade is written up in `docs/runbooks/curation-v2-migration.md`.

All seven plan steps are implemented. `cargo fmt --check`, `cargo clippy --all-targets`
(including `-W dead_code`) and `cargo test` (329 lib tests + 6 bin tests + the 7 integration suites) are green at HEAD.

Nothing has been run against the production database or the real providers. No local
`config.toml`, database, or API keys exist on the dev box, so verification so far is tests only.

## Decisions and deviations made while implementing

Steps 1–3 (session 1):

- **Legacy `up` links.** `Vote::parse("up")` → `Loved`, and `auth::verify_token` also accepts tokens
  signed over the literal `up` segment so already-published issues keep working. Keep both.
- **Same-date regeneration** no longer excludes its own picks (`published_before` uses issue dates
  strictly before the run date), per plan §8.1.
- **`VoyageConfig.api_key`** exists as a field (figment maps `DAILY_EPUB_VOYAGE__API_KEY` into it;
  `deny_unknown_fields` would otherwise reject the env var). Never document it in TOML.
- **`anthropic.max_concurrent_requests`** is validated but not consumed; summary concurrency is
  the constant `SUMMARY_CONCURRENCY = 4` in `editorial.rs`.
- `RATINGS_LOOKBACK_DAYS` in `profile/mod.rs` is effectively unbounded (36,500) for the prompt
  verdict block and the weekly rebuild; the knn/feed preference state uses
  `curation.ranking.rating_lookback_days` (180) as the plan says.
- Footer CSS: `.rating` has no `white-space: nowrap` (it would clip on narrow e-ink screens).

Step 4:

- Hygiene order is blocked → published_before → recently_rejected, all skipped for auto-includes.
- `admitted_by` records every retriever whose own top-N would have taken the article (first entry
  is the admitting one); exploration is flagged only when `exploration` admitted it.
- Under the pool cap (`triage_max`) the surplus gets `stage='eligible'`, `excluded_reason='not_admitted'`.
- Migration `0003_drop_scores.sql` drops `scores`; the churn rule reads `article_assessments`.

Step 5:

- `Candidate` is the only flow type; `ScoredArticle`, `combined_score`, `score.rs`, `select.rs` are gone.
- The editor omits the `facets:` line when every facet is unknown (plan renders it unconditionally).
- Articles cut at the shortlist keep the stage they reached (`admitted` if never deep-assessed,
  else `assessed`) with `excluded_reason` `cluster_suppressed` (hit the base cap) or `shortlist_cap`.
- `Curator::assess` is a no-op under `--skip-llm`; cached deep rows are still reused when DeepSeek
  is merely down.
- `--near-misses` and the Behind-the-paper list order by utility, falling back to the preliminary
  blend per row.
- Fixtures: `tests/fixtures/deepseek_deep_batch{,_messy}.json` replace the old score fixtures;
  `deepseek_triage_batch.json` added in step 4.

Step 6:

- `BehindThePaper` hangs off `Issue.behind` (filled after `build_issue`), not a `build_issue` argument.
  Generation time and cost shown in the chapter are measured at issue assembly (they exclude the
  EPUB build and publish, a few seconds); the cost includes Voyage.
- Voyage spend is recorded under `provider_costs["voyage"]`; `finish()` counts it once and keeps its
  tokens out of the LLM aggregate. Voyage spend is not preloaded into its meter for the UTC day.
- `src/lock.rs` uses `libc::flock` (`libc` was already in `Cargo.lock`; now a direct dependency).
  The holder's command name is written into the lock file for the "X is already running" message.
- The mid-run `admission:` line is `debug`; the four-line §15.4 block is logged once after
  `finish()` and printed by `print_report`. Timings `summaries` and `brief` replace `editorial`.

Step 7:

- `features prune` sweeps `article_embeddings`, `candidate_runs` and now `article_assessments`;
  `generate` runs it once after a published (non-dry) issue, best effort.
- The Brief chapter's TOC/title string is now "The Brief" (was still "From the Editor").
- Kept on purpose: `UsageMeter::new(&DeepseekConfig, ..)` (a convenience over `with_prices`, 12
  call sites); `StageCounts.candidates` (feeds `runs.candidates` and the colophon); the
  `prefilter_keep`/`score_batch_size` removed-key startup errors (operator guards); the SQL comment
  "filled in by step 4/5" inside the applied migration `0002` (sqlx checksums); six v1-era
  unreferenced `pub` items (`enrich_one`, `issues_before`, `today_in_tz`, `to_json_pretty`,
  `with_retry_policy`, `db::issues_before`) as pre-v2 API surface.
- `config::tests::shipped_example_config_matches_the_defaults_key_for_key` compares the example
  against `Config::default()` in both directions (numbers within 1e-6).

## Operator to-dos before the first real run

1. Set `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` in the systemd env file.
2. Set hard spend limits in the DeepSeek, Anthropic and Voyage dashboards (the meters are runaway
   guards, not accounting).
3. Copy `data/profile.md` to wherever `profile_path` points on the server (default is relative to
   `WorkingDirectory=/var/lib/daily-epub`, like `data/scour-interests.opml`).
4. Back up the DB, then `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`, 0003 drops `scores`).
5. `daily-epub features backfill --rated-only` then `--days 30` to warm the embedding cache.
6. `daily-epub generate --dry-run`, read the paper (including the new Behind-the-paper chapter),
   then `explain --date … --near-misses` and `stats`.
7. Watch the first few real runs' `providers:` log line against the ~$0.80/day estimate (plan §3).

## How the work was run (so the next session can repeat it)

- Orchestrator: Claude Code (this repo), one implementation agent per step, review + commit by the
  orchestrator after independent `cargo fmt/clippy/test`.
- **Codex via the `openai-codex` Claude Code plugin (v1.0.6) works on this host as of session 2**
  (the operator fixed bubblewrap). Launch:
  `node ~/.claude/plugins/cache/openai-codex/codex/1.0.6/scripts/codex-companion.mjs task --background --write --effort high "$(cat brief.md)"`,
  poll `… status <job-id> --json` (`.job.status`), read `… result <job-id>`. Inside its sandbox
  `bind()` on 127.0.0.1 is forbidden, so exactly ten pre-existing loopback tests fail there
  (`curate::llm::tests::anthropic_*` ×4, `extract::tests::relative_urls_resolve_against_the_url_we_landed_on`,
  `server::tests::*` ×5, plus the two `m7_server` integration tests); the brief must tell the agent
  to ignore those and the orchestrator runs the full suite outside. The old "host quirk" paragraph
  in `00-preamble.md` is obsolete.
- Codex usage budget is roughly one large step per ChatGPT usage window: in session 2 it finished
  step 4 (~26 min) and was cut off 21 minutes into step 5 ("try again at 7:44 PM"). The job status
  becomes `failed` with the partial work left in the tree; a Claude `general-purpose` agent with a
  "finish, don't restart" brief (known failures, scope audit checklist) completed it. Steps 6 and 7
  ran on Claude agents directly.
- Steps 4 → 5 → 6 → 7 were sequential, each reviewed against the plan sections named in its brief
  before committing.

## Next session: what remains

1. `git checkout curation-v2 && cargo test` (expect green). Follow
   `docs/runbooks/curation-v2-migration.md` on the server.
2. Optional: a Codex `review --background --scope branch --base main` pass over the whole branch.
3. `git merge --no-ff curation-v2` into `main`, build, deploy, and do the operator to-dos above.
4. After a week of real runs: read `stats`, tune `[curation.ranking]` from what `explain` shows,
   and revisit the deferred items in plan §23.