Curation v2 step 7: cleanup, prune paths, implementation notes

Dead code and stale v1 comments removed (clippy -W dead_code clean, the
three world.rs warnings fixed), the Brief chapter's TOC title renamed from
"From the Editor", features prune now also sweeps article_assessments and
generate runs the sweep once after publishing, the example config is
tested key-for-key against Config::default(), README commands match
--help, and docs/plans/2026-08-15-implementation-notes.md records the
Anthropic and Voyage facts, the new tables, the budget-day rule and the
lock.

Implemented by a Claude agent from docs/plans/curation-v2-briefs/step7.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
This commit is contained in:
2026-09-02 16:32:18 +00:00
co-authored by Claude Fable 5.1
parent d261cd485d
commit d403c51edf
18 changed files with 288 additions and 88 deletions
+77 -5
View File
@@ -1,6 +1,8 @@
# Implementation notes (shared brief for all implementation agents)
Authoritative spec: `docs/plans/2026-08-15-the-daily-epub.md`. Read it fully before writing code.
For curation (§3.5, §3.6 and §3.9 of that spec) the authority is now
`docs/plans/2026-09-02-personalized-curation-v2.md`; see "Curation v2" below.
This file records implementation-time decisions and verified external facts. Follow both.
## Verified external facts (2026-08-15)
@@ -68,6 +70,31 @@ This file records implementation-time decisions and verified external facts. Fol
walks back up to `MAX_LOOKBACK_DAYS` and the section is datelined with the day it actually
covers, not the masthead date.
## Verified external facts (2026-09-02, curation v2)
- **Anthropic Messages API** (verified 2026-09-02 against the bundled Claude API reference):
`POST https://api.anthropic.com/v1/messages` with headers `x-api-key`,
`anthropic-version: 2023-06-01`, `content-type: application/json` and
`anthropic-beta: server-side-fallback-2026-07-01`. Model id `claude-opus-5`; pricing
**$5.00 / M input, $25.00 / M output**, cache reads 0.1× input ($0.50/M), cache writes
1.25× ($6.25/M); the minimum cacheable prefix is 512 tokens. **No sampling parameters**
(`temperature`, `top_p`, `top_k` are a 400) and no `thinking` block — adaptive thinking is on
by default and depth is set with `output_config: {"effort": "high"}` (`low | medium | high |
xhigh | max`). The system prompt goes in `system: [{type: "text", text, cache_control:
{type: "ephemeral"}}]`; no assistant prefill, JSON is asked for in the prompt and parsed
tolerantly. `"fallbacks": "default"` (with the beta header) routes a request the safety
classifiers would refuse to a fallback model server-side; a response can still end with
`stop_reason: "refusal"` on HTTP 200, which the code treats as an error that degrades the
call to DeepSeek. Usage fields: `input_tokens` (uncached remainder),
`cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`. Timeout 300 s;
retry 429/5xx/network, never 400. Key only from `DAILY_EPUB_ANTHROPIC__API_KEY`.
- **Voyage AI embeddings** (verified 2026-09-02): `POST https://api.voyageai.com/v1/embeddings`
with `Authorization: Bearer <key>`; body `{input: [...], model: "voyage-4-lite", input_type:
"document" | "query", truncation: true, output_dimension: 512, output_dtype: "float"}`. Up
to 1,000 inputs and 1M tokens per request, 32k tokens per input. Vectors are
unit-normalized, so dot product = cosine. **$0.02 / M tokens** after a 200M-token free
allocation. Key only from `DAILY_EPUB_VOYAGE__API_KEY`.
## Cross-cutting implementation decisions
1. **sqlx usage**: use *runtime* queries (`sqlx::query(...).bind(...)`) and manual row mapping
@@ -80,12 +107,20 @@ This file records implementation-time decisions and verified external facts. Fol
Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
4. **HTTP**: one shared `reqwest::Client` (rustls, gzip, no cookies, 10s timeouts, UA
`the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`), passed by clone.
5. **LLM**: `async-openai` with custom base URL. All LLM calls go through `curate/llm.rs`
`LlmClient` which tracks token usage into a shared `UsageMeter` (input/cached/output tokens,
cost usd) and enforces `max_daily_usd`.
5. **LLM**: a hand-rolled `reqwest` client, not `async-openai` (the published crate exposes
neither `Client` nor `CreateChatCompletionRequest` at the pinned version). Every LLM call
goes through `curate/llm.rs`: `LlmClient { system_prompt, model, meter, backend, retry }`
over the `ChatBackend` trait, with `DeepseekBackend` (OpenAI-compatible chat completions,
`response_format: json_object`) and `AnthropicBackend` (Messages API, facts above). The
pipeline holds `Llms { bulk, editor }`; `editor_or_bulk()` degrades to DeepSeek when the
Claude client is missing or its meter is tripped. One `UsageMeter` per provider
(DeepSeek, Anthropic, Voyage) with its own price table and `max_daily_usd`. The system
prompt is sent first and byte-identical within a run so both providers' prefix caches hit.
6. **Testing**: unit tests inline per module; integration tests in `tests/` over fixture JSON in
`tests/fixtures/`. Never hit the network in tests. LLM stage mockable via `--skip-llm`
(prefilter order used for selection, feed excerpts as summaries).
`tests/fixtures/`. Never hit the network in tests: `MockBackend` (`ChatBackend`) and the
embedding mock (`EmbeddingBackend`) stand in for all three providers. `--skip-llm` makes
zero LLM calls (admission by cheap signals, `select_without_llm` by utility, excerpt
summaries); `--skip-embeddings` makes zero Voyage calls.
7. **Style**: rustfmt defaults, `cargo clippy` clean-ish, no `unwrap()` outside tests, tracing
spans per pipeline stage.
8. **File ownership**: waves of agents work in parallel on disjoint files. Do not edit files
@@ -99,3 +134,40 @@ This file records implementation-time decisions and verified external facts. Fol
`style-x4.css`). Askama 0.12+ configured via `askama.toml` if needed.
12. **Determinism**: chapter ids `art-{entry_id}`, stable filenames, issue regeneration for the
same date replaces prior rows/files (idempotent upsert everywhere).
## Curation v2 (2026-09-02)
The personalized ranker is specified in `docs/plans/2026-09-02-personalized-curation-v2.md`
(§0 settled decisions, §3 target pipeline, §19 configuration, §21 the seven landed steps);
`docs/plans/2026-09-02-curation-v2-progress.md` records per-step deviations. Facts an
implementer needs that are easy to get wrong:
- **Tables** (`migrations/0002_curation_v2.sql`, `0003_drop_scores.sql`; never edit
`0001_init.sql`): `rating_events` (append-only; the current verdict is the latest
`explicit` event), `article_embeddings` and `interest_embeddings` (f32 little-endian BLOBs,
`input_hash` = sha256 of the embedded text), `article_assessments` (`stage IN ('triage',
'deep')`, reused while `model` and `prompt_version` match and `assessed_at` is within
`assessment_reuse_days`; `--rescore` ignores the cache), `candidate_runs` (one row per
considered article per run, upserted with every column set on each stage transition),
`runs.config_json` / `runs.provider_costs_json`, `issue_articles.why`. `ratings`,
`feed_priors` and `scores` are dropped; `kv` keeps `ingest_watermark`, `taste_profile`,
`taste_profile_learned`, `profile_version`.
- **Feedback**: `Vote` is `loved | good | down` (`NotForMe`); the HMAC message stays
`{issue_date}/{article_id}/{vote}`. `Vote::parse("up")``Loved` and `auth::verify_token`
still accepts tokens signed over the literal `up` segment because published issues carry
those links. Keep both.
- **Budget day**: each provider's `UsageMeter` is preloaded with the spend of earlier runs on
the **UTC date of the run's `started_at`**, summed from `runs.provider_costs_json`
(`db::spend_for_date` by nominal issue date is gone). A tripped meter skips that provider's
remaining calls; the paper always publishes.
- **Lock**: `src/lock.rs` takes `libc::flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`
for `generate`, `profile rebuild`, `features backfill` and `backfill-social`; a second
writer exits with "<command> is already running". `serve`, `explain`, `stats`, `ratings`,
`features prune` and `db migrate` never take it.
- **Retention**: `telemetry::prune` removes `article_embeddings` of unrated, unpublished
articles older than `embedding_retention_days` (120) and `candidate_runs` rows plus
`article_assessments` older than `telemetry_retention_days` (180). `features prune` runs it
on demand; `generate` runs it once after publishing, best effort.
- **Keys**: `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` map onto
`AnthropicConfig.api_key` / `VoyageConfig.api_key` through figment; the fields exist only
for that mapping and are never documented in TOML, logged, or stored.