Curation v2 step 7: cleanup, prune paths, implementation notes
Dead code and stale v1 comments removed (clippy -W dead_code clean, the three world.rs warnings fixed), the Brief chapter's TOC title renamed from "From the Editor", features prune now also sweeps article_assessments and generate runs the sweep once after publishing, the example config is tested key-for-key against Config::default(), README commands match --help, and docs/plans/2026-08-15-implementation-notes.md records the Anthropic and Voyage facts, the new tables, the budget-day rule and the lock. Implemented by a Claude agent from docs/plans/curation-v2-briefs/step7.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
This commit is contained in:
@@ -1,6 +1,8 @@
|
||||
# Implementation notes (shared brief for all implementation agents)
|
||||
|
||||
Authoritative spec: `docs/plans/2026-08-15-the-daily-epub.md`. Read it fully before writing code.
|
||||
For curation (§3.5, §3.6 and §3.9 of that spec) the authority is now
|
||||
`docs/plans/2026-09-02-personalized-curation-v2.md`; see "Curation v2" below.
|
||||
This file records implementation-time decisions and verified external facts. Follow both.
|
||||
|
||||
## Verified external facts (2026-08-15)
|
||||
@@ -68,6 +70,31 @@ This file records implementation-time decisions and verified external facts. Fol
|
||||
walks back up to `MAX_LOOKBACK_DAYS` and the section is datelined with the day it actually
|
||||
covers, not the masthead date.
|
||||
|
||||
## Verified external facts (2026-09-02, curation v2)
|
||||
|
||||
- **Anthropic Messages API** (verified 2026-09-02 against the bundled Claude API reference):
|
||||
`POST https://api.anthropic.com/v1/messages` with headers `x-api-key`,
|
||||
`anthropic-version: 2023-06-01`, `content-type: application/json` and
|
||||
`anthropic-beta: server-side-fallback-2026-07-01`. Model id `claude-opus-5`; pricing
|
||||
**$5.00 / M input, $25.00 / M output**, cache reads 0.1× input ($0.50/M), cache writes
|
||||
1.25× ($6.25/M); the minimum cacheable prefix is 512 tokens. **No sampling parameters**
|
||||
(`temperature`, `top_p`, `top_k` are a 400) and no `thinking` block — adaptive thinking is on
|
||||
by default and depth is set with `output_config: {"effort": "high"}` (`low | medium | high |
|
||||
xhigh | max`). The system prompt goes in `system: [{type: "text", text, cache_control:
|
||||
{type: "ephemeral"}}]`; no assistant prefill, JSON is asked for in the prompt and parsed
|
||||
tolerantly. `"fallbacks": "default"` (with the beta header) routes a request the safety
|
||||
classifiers would refuse to a fallback model server-side; a response can still end with
|
||||
`stop_reason: "refusal"` on HTTP 200, which the code treats as an error that degrades the
|
||||
call to DeepSeek. Usage fields: `input_tokens` (uncached remainder),
|
||||
`cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`. Timeout 300 s;
|
||||
retry 429/5xx/network, never 400. Key only from `DAILY_EPUB_ANTHROPIC__API_KEY`.
|
||||
- **Voyage AI embeddings** (verified 2026-09-02): `POST https://api.voyageai.com/v1/embeddings`
|
||||
with `Authorization: Bearer <key>`; body `{input: [...], model: "voyage-4-lite", input_type:
|
||||
"document" | "query", truncation: true, output_dimension: 512, output_dtype: "float"}`. Up
|
||||
to 1,000 inputs and 1M tokens per request, 32k tokens per input. Vectors are
|
||||
unit-normalized, so dot product = cosine. **$0.02 / M tokens** after a 200M-token free
|
||||
allocation. Key only from `DAILY_EPUB_VOYAGE__API_KEY`.
|
||||
|
||||
## Cross-cutting implementation decisions
|
||||
|
||||
1. **sqlx usage**: use *runtime* queries (`sqlx::query(...).bind(...)`) and manual row mapping
|
||||
@@ -80,12 +107,20 @@ This file records implementation-time decisions and verified external facts. Fol
|
||||
Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
|
||||
4. **HTTP**: one shared `reqwest::Client` (rustls, gzip, no cookies, 10s timeouts, UA
|
||||
`the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`), passed by clone.
|
||||
5. **LLM**: `async-openai` with custom base URL. All LLM calls go through `curate/llm.rs`
|
||||
`LlmClient` which tracks token usage into a shared `UsageMeter` (input/cached/output tokens,
|
||||
cost usd) and enforces `max_daily_usd`.
|
||||
5. **LLM**: a hand-rolled `reqwest` client, not `async-openai` (the published crate exposes
|
||||
neither `Client` nor `CreateChatCompletionRequest` at the pinned version). Every LLM call
|
||||
goes through `curate/llm.rs`: `LlmClient { system_prompt, model, meter, backend, retry }`
|
||||
over the `ChatBackend` trait, with `DeepseekBackend` (OpenAI-compatible chat completions,
|
||||
`response_format: json_object`) and `AnthropicBackend` (Messages API, facts above). The
|
||||
pipeline holds `Llms { bulk, editor }`; `editor_or_bulk()` degrades to DeepSeek when the
|
||||
Claude client is missing or its meter is tripped. One `UsageMeter` per provider
|
||||
(DeepSeek, Anthropic, Voyage) with its own price table and `max_daily_usd`. The system
|
||||
prompt is sent first and byte-identical within a run so both providers' prefix caches hit.
|
||||
6. **Testing**: unit tests inline per module; integration tests in `tests/` over fixture JSON in
|
||||
`tests/fixtures/`. Never hit the network in tests. LLM stage mockable via `--skip-llm`
|
||||
(prefilter order used for selection, feed excerpts as summaries).
|
||||
`tests/fixtures/`. Never hit the network in tests: `MockBackend` (`ChatBackend`) and the
|
||||
embedding mock (`EmbeddingBackend`) stand in for all three providers. `--skip-llm` makes
|
||||
zero LLM calls (admission by cheap signals, `select_without_llm` by utility, excerpt
|
||||
summaries); `--skip-embeddings` makes zero Voyage calls.
|
||||
7. **Style**: rustfmt defaults, `cargo clippy` clean-ish, no `unwrap()` outside tests, tracing
|
||||
spans per pipeline stage.
|
||||
8. **File ownership**: waves of agents work in parallel on disjoint files. Do not edit files
|
||||
@@ -99,3 +134,40 @@ This file records implementation-time decisions and verified external facts. Fol
|
||||
`style-x4.css`). Askama 0.12+ configured via `askama.toml` if needed.
|
||||
12. **Determinism**: chapter ids `art-{entry_id}`, stable filenames, issue regeneration for the
|
||||
same date replaces prior rows/files (idempotent upsert everywhere).
|
||||
|
||||
## Curation v2 (2026-09-02)
|
||||
|
||||
The personalized ranker is specified in `docs/plans/2026-09-02-personalized-curation-v2.md`
|
||||
(§0 settled decisions, §3 target pipeline, §19 configuration, §21 the seven landed steps);
|
||||
`docs/plans/2026-09-02-curation-v2-progress.md` records per-step deviations. Facts an
|
||||
implementer needs that are easy to get wrong:
|
||||
|
||||
- **Tables** (`migrations/0002_curation_v2.sql`, `0003_drop_scores.sql`; never edit
|
||||
`0001_init.sql`): `rating_events` (append-only; the current verdict is the latest
|
||||
`explicit` event), `article_embeddings` and `interest_embeddings` (f32 little-endian BLOBs,
|
||||
`input_hash` = sha256 of the embedded text), `article_assessments` (`stage IN ('triage',
|
||||
'deep')`, reused while `model` and `prompt_version` match and `assessed_at` is within
|
||||
`assessment_reuse_days`; `--rescore` ignores the cache), `candidate_runs` (one row per
|
||||
considered article per run, upserted with every column set on each stage transition),
|
||||
`runs.config_json` / `runs.provider_costs_json`, `issue_articles.why`. `ratings`,
|
||||
`feed_priors` and `scores` are dropped; `kv` keeps `ingest_watermark`, `taste_profile`,
|
||||
`taste_profile_learned`, `profile_version`.
|
||||
- **Feedback**: `Vote` is `loved | good | down` (`NotForMe`); the HMAC message stays
|
||||
`{issue_date}/{article_id}/{vote}`. `Vote::parse("up")` → `Loved` and `auth::verify_token`
|
||||
still accepts tokens signed over the literal `up` segment because published issues carry
|
||||
those links. Keep both.
|
||||
- **Budget day**: each provider's `UsageMeter` is preloaded with the spend of earlier runs on
|
||||
the **UTC date of the run's `started_at`**, summed from `runs.provider_costs_json`
|
||||
(`db::spend_for_date` by nominal issue date is gone). A tripped meter skips that provider's
|
||||
remaining calls; the paper always publishes.
|
||||
- **Lock**: `src/lock.rs` takes `libc::flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`
|
||||
for `generate`, `profile rebuild`, `features backfill` and `backfill-social`; a second
|
||||
writer exits with "<command> is already running". `serve`, `explain`, `stats`, `ratings`,
|
||||
`features prune` and `db migrate` never take it.
|
||||
- **Retention**: `telemetry::prune` removes `article_embeddings` of unrated, unpublished
|
||||
articles older than `embedding_retention_days` (120) and `candidate_runs` rows plus
|
||||
`article_assessments` older than `telemetry_retention_days` (180). `features prune` runs it
|
||||
on demand; `generate` runs it once after publishing, best effort.
|
||||
- **Keys**: `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` map onto
|
||||
`AnthropicConfig.api_key` / `VoyageConfig.api_key` through figment; the fields exist only
|
||||
for that mapping and are never documented in TOML, logged, or stored.
|
||||
|
||||
Reference in New Issue
Block a user