diff --git a/docs/plans/2026-09-02-curation-v2-progress.md b/docs/plans/2026-09-02-curation-v2-progress.md new file mode 100644 index 0000000..5b589ad --- /dev/null +++ b/docs/plans/2026-09-02-curation-v2-progress.md @@ -0,0 +1,90 @@ +# Personalized Curation v2 — implementation progress and handoff + +**Updated:** 2026-09-02 (end of session 1) +**Plan:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§21 is the step sequence) +**Branch:** `curation-v2` (branched from `main` at `a599745`; not merged, not pushed) +**Briefs:** `docs/plans/curation-v2-briefs/` — `00-preamble.md` + one `stepN.md` per step. Each +brief was handed to an implementation agent as `cat 00-preamble.md stepN.md`. + +## Where things stand + +| Step (§21) | Status | Commit | +|---|---|---| +| 1. Feedback and profile | **done**, reviewed (codex review: no actionable issues) | `3a9f4b9` | +| 2. Claude editor and editorial | **done** | `57efbb4` | +| 3. Embeddings, signals, telemetry | **done** (branch `curation-v2-step3`, merged) | `ea3b141` + merge commit | +| 4. Triage replaces the gate | not started — brief ready: `step4.md` | | +| 5. Deep assessment, utility, diversity | not started — brief ready: `step5.md` | | +| 6. Paper telemetry, stats, lock | not started — brief ready: `step6.md` | | +| 7. Cleanup + implementation notes | not started — brief ready: `step7.md` | | + +`cargo fmt --check`, `cargo clippy --all-targets` (only the three pre-existing `src/world.rs` +`needless_borrow` warnings) and `cargo test` (294 lib tests + 6 integration suites) are green at HEAD. + +Nothing has been run against the production database or the real providers. No local +`config.toml`, database, or API keys exist on the dev box, so verification so far is tests only. + +## Decisions and deviations made while implementing (read before step 4) + +- **`scores` table not yet dropped.** Migration `0002_curation_v2.sql` created every new table, copied + `ratings` → `rating_events`, and dropped `ratings` and `feed_priors`, but kept `scores` because the + old Stage A scoring (`src/curate/score.rs`, `db::upsert_score`, `recently_low_scored_ids`) still + uses it. Step 4 adds `migrations/0003_drop_scores.sql` and moves the churn rule to + `article_assessments` (already in `step4.md`). +- **Old prefilter still gates.** Hygiene → embeddings → signals → preliminary blend now run for every + article, and `candidate_runs` rows are written with the stage vocabulary mapped onto the old flow + (`admitted_by` is `["prefilter"]`/`["auto"]` for now), but `prefilter::run` + `prefilter_keep` + still decide the deep set until step 4. +- **Legacy `up` links.** `Vote::parse("up")` → `Loved`, and `auth::verify_token` also accepts tokens + signed over the literal `up` segment so already-published issues keep working. +- **Same-date regeneration** no longer excludes its own picks (`published_before` uses issue dates + strictly before the run date), per plan §8.1. +- **`VoyageConfig.api_key`** exists as a field (figment maps `DAILY_EPUB_VOYAGE__API_KEY` into it; + `deny_unknown_fields` would otherwise reject the env var). Never document it in TOML. +- **`anthropic.max_concurrent_requests`** is validated but not consumed yet; summary concurrency is + the constant `SUMMARY_CONCURRENCY = 4` in `editorial.rs`. +- `UsageMeter::new(&DeepseekConfig, ..)` survives as a compat constructor over `with_prices`. +- `tests/fixtures/deepseek_front_page.json` was replaced by `tests/fixtures/claude_brief.json`. +- `RATINGS_LOOKBACK_DAYS` in `profile/mod.rs` is effectively unbounded (36,500) for the prompt + verdict block and the weekly rebuild; the knn/feed preference state uses + `curation.ranking.rating_lookback_days` (180) as the plan says. +- Footer CSS: `.rating` has no `white-space: nowrap` (it would clip on narrow e-ink screens). + +## Operator to-dos before the first real run + +1. Set `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` in the systemd env file. +2. Set hard spend limits in the DeepSeek, Anthropic and Voyage dashboards (the meters are runaway + guards, not accounting). +3. Copy `data/profile.md` to wherever `profile_path` points on the server (default is relative to + `WorkingDirectory=/var/lib/daily-epub`, like `data/scour-interests.opml`). +4. Run `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`; back up the DB first). +5. `daily-epub features backfill --rated-only` then `--days 30` to warm the embedding cache. +6. A `generate --dry-run` and read the paper; `explain --near-misses` once step 4 lands. + +## How the work was run (so the next session can repeat it) + +- Orchestrator: Claude Code (this repo), one implementation agent per step, review + commit by the + orchestrator after independent `cargo fmt/clippy/test`. +- Codex: `codex exec -C --sandbox workspace-write --add-dir ~/.cargo -c + sandbox_workspace_write.network_access=true -c model_reasoning_effort=high -o - < `, + detached with `setsid nohup`, exit code written to a file and watched with a monitor. The + `openai-codex` Claude Code plugin's `task` runs fail on this host (bubblewrap cannot create user + namespaces: `kernel.apparmor_restrict_unprivileged_userns = 1`); the CLI works if the brief tells + the agent to edit files via shell commands instead of the `apply_patch` tool (see the preamble). + The plugin's read-only `review --background --scope working-tree` does work and was used on step 1. + Fix for the sandbox: `sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0`. +- Codex ran out of ChatGPT usage after ~560k tokens (step 1 complete, steps 2 and 3 cut off + mid-way); Claude `general-purpose` sub-agents finished steps 2 and 3 from the partial trees and + resolved the step 2/3 merge. Budget roughly one large step per Codex usage window. +- Steps 2 and 3 were run in parallel (step 3 in a git worktree); the merge cost ~19 conflict hunks + in config/pipeline/report/main/README/config.example. Steps 4 → 5 → 6 → 7 are sequential. + +## Next session: exact starting instructions + +1. `git checkout curation-v2 && cargo test` (expect green). +2. Read this file, then `docs/plans/curation-v2-briefs/00-preamble.md` and `step4.md`. +3. Launch the step 4 agent with `cat 00-preamble.md step4.md` as the prompt (Codex CLI as above, or a + Claude general-purpose agent — tell it to ignore the "host quirk" paragraph in that case). +4. Review the diff against plan §10–§11, run the checks, commit as "Curation v2 step 4: …". +5. Repeat for steps 5, 6, 7. After step 7: `git merge --no-ff curation-v2` into `main`, deploy, and + do the operator to-dos above. diff --git a/docs/plans/curation-v2-briefs/00-preamble.md b/docs/plans/curation-v2-briefs/00-preamble.md new file mode 100644 index 0000000..3a83080 --- /dev/null +++ b/docs/plans/curation-v2-briefs/00-preamble.md @@ -0,0 +1,52 @@ +# Implementation agent brief — The Daily EPUB, "Personalized Curation v2" + +You are implementing ONE step of a multi-step plan in the Rust repository at +`/home/thallada/workspace/the-daily-epub` (branch `curation-v2`, already checked out). + +The orchestrator hands you `docs/plans/curation-v2-briefs/00-preamble.md` (this file) followed by one +`stepN.md` from the same directory; `docs/plans/2026-09-02-curation-v2-progress.md` records what has +landed so far. + +The plan is `docs/plans/2026-09-02-personalized-curation-v2.md`. Read it in full before +writing code: §0 records settled decisions (do not reopen them), §1 lists the files to +read first, §21 is the implementation sequence. Also read +`docs/plans/2026-08-15-implementation-notes.md` §"Cross-cutting implementation decisions" +(note: item 5 is stale — the code uses a hand-rolled `reqwest` client in `src/curate/llm.rs`, +keep that). + +## Hard rules + +- Implement ONLY the step assigned below. Later steps land separately; do not start them. + Where this step needs a type or table that a later step fills in, create it now exactly as + the plan specifies and leave it empty/unused with a short `// filled in by step N` comment. +- Conventions: sqlx *runtime* queries (`sqlx::query(...).bind(...)`, never the `query!` macros), + `jiff` for time, RFC3339 UTC strings in SQLite, `thiserror`/`anyhow` error style, no `unwrap()` + outside tests, tracing spans per stage, rustfmt defaults, askama templates. +- Tests never touch the network. Use mock backends following `MockBackend` in `src/curate/llm.rs`. +- Follow the plan's names for modules, functions, config keys, table and column names, enum + strings and prompt constants exactly. If the plan is internally contradictory or impossible + for this step, pick the closest behaviour, keep the paper publishable, and describe the + deviation in your final report. Do not invent scope beyond the plan. +- API keys only from env vars; never in config files, logs, tests or the database. +- Prefer existing dependencies in `Cargo.toml`. Add a new crate only if there is no reasonable + way without it, and say so in the report. +- Do NOT commit and do NOT create branches. Leave all changes in the working tree; the + orchestrator reviews and commits. Do not edit anything under `docs/`. +- Keep `config.example.toml` and README in sync with any config keys you add or remove + (the test `shipped_example_config_parses` must pass). +- Before finishing, run and make pass: `cargo fmt`, `cargo clippy --all-targets` (no new + warnings), `cargo test`. Fix what you broke; do not delete or `#[ignore]` tests to get green + unless the plan removes the feature they cover (then move/replace the tests as §18 says). + +## Final report + +End with a concise report (this is what the orchestrator reads): what you implemented, every +deviation from the plan and why, anything from the step you could not finish, and the exact +`cargo test` summary line(s). Keep it under 60 lines. + +## Host quirk (important) + +On this machine the built-in `apply_patch` tool's filesystem helper fails with +`bwrap: ... Operation not permitted`. Do not keep retrying it. Edit files through shell commands +instead (the `apply_patch` CLI invoked from bash works, as do `python3 - <<'EOF'` rewrite scripts, +`sed -i`, and heredocs). Shell commands, `cargo build`, `cargo test` and `cargo clippy` all work. diff --git a/docs/plans/curation-v2-briefs/step1.md b/docs/plans/curation-v2-briefs/step1.md new file mode 100644 index 0000000..e6aea9d --- /dev/null +++ b/docs/plans/curation-v2-briefs/step1.md @@ -0,0 +1,73 @@ + +--- + +# YOUR STEP: 1 — Feedback and profile (plan §21 step 1) + +Scope (plan §6, §7, §8; tests in §20 under Vote, Rating events, Migration, and the prompt): + +1. **Migration `migrations/0002_curation_v2.sql`** (§7). Create `rating_events`, + `article_embeddings`, `interest_embeddings`, `article_assessments`, `candidate_runs` with + the exact columns/indexes in §7; add `runs.config_json`, `runs.provider_costs_json`, + `issue_articles.why`. Copy `ratings` rows into `rating_events` per §6.2 (`vote=1 → 'loved',1.0`; + `vote=-1 → 'not_for_me',-1.0`; `source='migration'`; `kind='explicit'`; `event_at=rated_at`), + then `DROP TABLE ratings; DROP TABLE feed_priors;`. + **Orchestrator decision:** do NOT drop `scores` in this step. Stage A scoring + (`db::upsert_score`, `recently_low_scored_ids`) still writes/reads it until step 4 replaces + it with `article_assessments`; step 4 will add a `0003` migration dropping it. Do not edit + `0001_init.sql`. Migration is plain SQL, no Rust bootstrap. +2. **Three-way `Vote`** (§6.1) in `src/types.rs`: `Loved | Good | NotForMe`, `as_str` → + `"loved" | "good" | "down"`, `parse` also accepts legacy `"up"` → `Loved`, `value(&FeedbackConfig)` + → configured 1.0 / 0.35 / −1.0. Add `[curation.feedback]` config + (`loved_value`, `good_value`, `not_for_me_value`, `verdicts_in_prompt = 60`). Remove `Vote::as_i64`, + `Rating`, `FeedPrior`, and everything that depended on `ratings`/`feed_priors` + (`db::upsert_rating`, `ratings_with_feed`, `upsert_feed_prior`, `feed_priors`, + `profile::rebuild_feed_priors`, the feed-prior term in `prefilter.rs` and `ScoredArticle.feed_prior` + — set the prefilter's feed-prior contribution to nothing; the field can go). HMAC links + (`src/auth.rs`) must verify for all three votes; message stays `{issue_date}/{article_id}/{vote}`. +3. **Footer + confirmation page** (§6.1): `RatingLinks` in `src/epub/chapters.rs` and + `src/epub/templates/chapter.xhtml` render three links on one line sized for e-ink: + `Was this a good pick? [ Loved it ] [ Good ] [ Not for me ] Read online ↗`. + X4 edition still renders no links. `server::handle_rating` appends one `rating_events` row + (`kind='explicit'`, `source='epub'`, label `loved|good|not_for_me`, value from config) and + returns "Recorded: Loved it — thanks." (etc.) plus the other two links so a mis-tap can be + corrected. No feed-prior rebuild, no provider call. +4. **`rating_events` access** (§6.2) in `src/db.rs`: `append_rating_event(...)`, + `db::current_ratings(lookback_days) -> Vec` implementing the latest-explicit-event + rule (`ORDER BY event_at DESC, id DESC`, `cleared` removes the article from the learned set), + joining `articles`, the best entry's feed title, and the most recent `issue_articles.summary`. + Move `RatedArticle` to `src/types.rs` with `summary: Option`, `facets: Option` + (define `Facets` per §12.1 now, all `Option`s, unused until step 5), `note: Option`, + plus `RatingEvent`. Also define `RatedArticle.value: f64` and `label`. +5. **Ratings CLI** (§6.3) in `src/main.rs`: `ratings list [--days 90] [--label loved|good|down|cleared]`, + `ratings set --article ID|--url URL --label loved|good|down [--note "..."]`, + `ratings clear --article ID|--url URL`. `set`/`clear` append rows with `source='cli'` and + `issue_date` from the latest `issue_articles` row for the article if any; `--url` canonicalizes + with `dedupe`'s canonical URL function then `db::article_id_for_url`. Print something useful. +6. **`data/profile.md`** (§8.2): create the file with the initial content shown in §8.2 verbatim; + add `profile_path` config (default `data/profile.md`); loader that parses any `## Interests` + section (one per line, leading `- ` stripped) and unions it case-insensitively with the OPML + interests, passing everything else through verbatim. Remove `STATED_PREFERENCES` and the + reader/wants/does-not-want/how-to-judge parts of `PROFILE_PREAMBLE` from code (they now live + in the file); the editor-in-chief framing paragraph stays in code. If the file is missing, + warn and continue with the OPML interests only. +7. **System prompt** (§8.4): rebuilt every run in this exact order: framing → profile.md verbatim + (minus Interests) → standing interests grouped by `profile::themes::group_into_themes` → + learned adjustments → **recent verdicts**: up to `verdicts_in_prompt` most recent explicit + ratings, newest first, `LOVED | title | feed | one-line summary` (labels `LOVED`/`GOOD`/`NOT FOR ME`), + cleared and duplicate articles removed. Byte-identical within a run. `profile_version` bumps only + on the weekly rebuild, as today. `kv` keeps `ingest_watermark`, `taste_profile`, + `taste_profile_learned`, `profile_version` (§7.8). +8. **Weekly rebuild** (§8.3): still on the DeepSeek client this step (the Claude editor client + arrives in step 2). Each rated line carries + `LOVED | title | feed | summary | facets: format/depth/evidence/technicality/topic_group | note: …` + (omit empty parts), up to 200 most recent explicit ratings, cleared excluded; the softened + "strong prior, not a rule" instruction; the required "Diversity check:" bullet. +9. Pipeline: remove the `rebuild_feed_priors` call and every `feed_priors` reference; the paper + must still build. `handle_rating` and `profile rebuild` must not touch dropped tables. + +Tests to add/adapt (§20): Vote parse/serialize incl. `up`→Loved; HMAC for all three; template +renders three links standard / none X4; latest explicit event wins; `cleared` removes from the +learned set; migration on a temp DB through 0001 then 0002 copies old ratings with the right +labels/values and `ratings`/`feed_priors` are gone; CLI `set`/`clear` append `source='cli'` rows +(test the db-level function); profile.md Interests parsing and union; system prompt section order +and verdict block content. Update `tests/m7_server.rs` and `tests/m4_epub.rs` as needed. diff --git a/docs/plans/curation-v2-briefs/step2.md b/docs/plans/curation-v2-briefs/step2.md new file mode 100644 index 0000000..6049134 --- /dev/null +++ b/docs/plans/curation-v2-briefs/step2.md @@ -0,0 +1,86 @@ + +--- + +# YOUR STEP: 2 — Claude editor and editorial (plan §21 step 2) + +Step 1 has landed (three-way votes, `rating_events`, `data/profile.md`, the rebuilt system +prompt). Read `git log -3` and the current `src/curate/llm.rs`, `select.rs`, `editorial.rs`, +`pipeline.rs`, `config.rs`, `report.rs`, `db.rs`, `src/epub/chapters.rs` + templates before coding. + +Scope (plan §4.1, §4.2, §5, §7.6, §13, §14, §15.1 chapter/in-this-issue `why`, §18 types, §19 config): + +1. **`AnthropicBackend`** in `src/curate/llm.rs` implementing `ChatBackend` exactly per §4.2: + `POST {base_url}/v1/messages`; headers `x-api-key`, `anthropic-version: 2023-06-01`, + `content-type: application/json`, `anthropic-beta: server-side-fallback-2026-07-01`; + body `{model, max_tokens: 16000, system: [{type:"text", text, cache_control:{type:"ephemeral"}}], + messages:[{role:"user", content}], output_config:{effort}, fallbacks:"default"}`. + **Never** send `temperature`, `top_p`, `top_k`, `thinking`, or an assistant prefill. Ask for JSON + in the prompt text and parse tolerantly (reuse `strip_code_fence` + the existing tolerant parsing). + Response: concatenate `content[]` blocks with `type == "text"`. Usage: `input_tokens`, + `cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`; cost = input×price_input + + cache_creation×price_cache_write + cache_read×price_cache_read + output×price_output (per M). + `stop_reason == "refusal"` (HTTP 200) surfaces as a distinct `LlmError` variant that triggers the + fallback; 429/5xx/network are `Transient` (retried by the existing `RetryPolicy`); 400 is never + retried. Request timeout 300 s. API key only from `DAILY_EPUB_ANTHROPIC__API_KEY`. + `ChatRequest` gains `effort: Option`; the Anthropic backend ignores `temperature`/`json` + and maps `system` to the cached system block. `TokenUsage` needs a cache-write counter (keep the + DeepSeek path's cost identical to today). Generalize `UsageMeter` to take a per-provider price + table instead of `&DeepseekConfig`. +2. **`Llms { bulk: Option, editor: Option }`** with `editor_or_bulk()` (the + editor when configured and its meter is not tripped, else bulk). Both clients share the exact same + system prompt string (§8.4). `LlmError::MissingApiKey` and friends must name the provider. +3. **Config** (§4.2, §19): `[anthropic]` with `enabled`, `base_url`, `model = "claude-opus-5"`, + `api_key` (env only), `effort = "high"`, `price_input_per_mtok = 5.0`, + `price_cache_write_per_mtok = 6.25`, `price_cache_read_per_mtok = 0.5`, `price_output_per_mtok = 25.0`, + `max_daily_usd = 3.0`, `max_concurrent_requests = 4`; `[deepseek].max_concurrent_requests = 4`; + `[curation].max_article_count = 28`; `[editorial]` with `summary_model = "editor"` (`editor|bulk`) + and `summary_input_tokens = 3000`. Validate `max_article_count >= target_article_count`, effort ∈ + {low, medium, high, xhigh, max}, batch/concurrency ≥ 1. Startup logs the resolved models and whether + each provider is enabled (§19). Update `config.example.toml` and the README (prerequisites, cost + line, the note that server-side fallback is enabled, dashboard spend limits as the real backstop). +4. **Budget and concurrency** (§5): one `UsageMeter` per provider, each with its own `max_daily_usd`. + The budget day is the **UTC date of the run's `started_at`**, preloaded by summing + `runs.provider_costs_json` for earlier runs that UTC day; this replaces `db::spend_for_date` + (remove it). Write `runs.provider_costs_json` (`{"deepseek": {...usage, cost_usd}, "anthropic": {...}}`) + and `runs.config_json` (the resolved `[curation]`, `[editorial]`, model names and prompt versions, + as JSON) in `finish_run`. Keep `runs.cost_usd` as the total across providers. Existing Stage A + scoring batches run through `futures::stream::iter(...).buffer_unordered(max_concurrent_requests)` + with the budget check before each request is spawned; a tripped meter skips remaining calls, + lets in-flight finish, records the number of unscored candidates in the report, and continues. +5. **The editor** (§13): `EDITOR_INSTRUCTIONS` replaces `SELECT_INSTRUCTIONS` verbatim from the plan + with `{soft_target}`/`{hard_max}` substituted. Runs on `editor_or_bulk()`; on refusal/error fall + back to the same prompt on the bulk client; if that fails too, `select_without_llm`. Per-item + rendering follows §13 with what exists today (Stage A score/category/rationale instead of + quality/fit/facets — those arrive in step 5; render `flags: always-include | excerpt only`; + `opening:` first 60 words). Do not put the numeric blend in the prompt. `assemble()` keeps section + validation, unique lead, auto-include reinsertion, duplicate-id defence, malformed-response + fallback and the `hard_max` trim (by today's ranking key, utility arrives in step 5). + **Delete the "too few: top up" branch.** `--max-articles N` is a ceiling: + `hard_max = min(config.curation.max_article_count, N)`, `soft_target = min(target_article_count, hard_max)`. + `select_without_llm` respects `soft_target` as its size. Each pick's `why` (≤14 words) lands on + `Pick.why: Option` and in `issue_articles.why` (`replace_issue_articles`). +6. **Editorial** (§14): summaries run on the editor client per `editorial.summary_model` with the + input budget from config (3,000 tokens), concurrency 4 via `buffer_unordered`, fallback per article: + bulk client, then the excerpt. `BRIEF_INSTRUCTIONS` replaces `FRONT_PAGE_INSTRUCTIONS` verbatim + from §14.2 (JSON `{"brief": "..."}`); input is the lineup with sections, each pick's title, feed, + `why`, summary and the Stage A score. `Editorial.section_intros` is removed (or always empty) and + `section.xhtml` renders only the section name; `front_page.xhtml` renders the brief under the + masthead; fallback stays `fallback_front_page_html`. The weekly profile rebuild runs on + `editor_or_bulk()` (pipeline and the `profile rebuild` CLI). +7. **Paper** (§15.1): `chapter.xhtml` gets a small italic `Why it's here: ` line under the meta + line; the In-this-issue page shows the `why` line under each summary. `Colophon` gains + `provider_costs` and `models`; the colophon template prints per-provider cost lines and the models + (editor and summaries model, bulk model). `StageCounts`/`RunReport` carry per-provider usage; + `print_report` in `main.rs` prints per-provider cost. +8. Remove `Vote`-era leftovers you notice only if they are in your files; do not touch embeddings, + signals, telemetry or triage (steps 3–5). + +Tests to add/adapt (§20 "Anthropic backend", "Editor", parts of "Pipeline"): request body has the +system block with `cache_control`, no `temperature`, `output_config.effort`, `fallbacks`, the beta +header; usage fields parsed into cost with cache read/write prices; `stop_reason: refusal` surfaces as +the fallback-triggering error; 429 retried, 400 not (use a mock backend or a local `axum`/`tokio` +listener — never the real network); a nine-pick response is published as nine; `hard_max` trims; +`--max-articles` is a ceiling; auto-includes reinserted; `why` lines land on picks and in +`issue_articles.why`; refusal/error on the Anthropic mock falls back to the DeepSeek mock with the +same prompt; the brief is parsed and rendered, section intros gone; `provider_costs_json` and +`config_json` are written; budget day preload sums by UTC date. diff --git a/docs/plans/curation-v2-briefs/step3.md b/docs/plans/curation-v2-briefs/step3.md new file mode 100644 index 0000000..afbe699 --- /dev/null +++ b/docs/plans/curation-v2-briefs/step3.md @@ -0,0 +1,103 @@ + +--- + +# YOUR STEP: 3 — Embeddings, signals, telemetry (plan §21 step 3) + +You are working in a git **worktree** on branch `curation-v2-step3`, which branches from the step 1 +commit. Another agent is implementing step 2 (the Claude editor) in parallel on the main checkout; +you will not see its changes and must not need them. Read `git log -3` and the current +`src/curate/`, `src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs` first. +Do not run `git checkout`, `git stash`, `git worktree` or `git commit`. + +Scope (plan §4.3, §7.1, §7.2, §7.4, §7.5, §9, §12.2 normalization only, §12.4 preliminary blend, +§15.2 `explain`, §15.4 log line parts, §16, §17 Voyage row, §19 `[voyage]` + `[curation.ranking]`). +The old heuristic prefilter still gates in this step; signals are computed for every article that +passes hygiene and persisted, nothing about *selection* changes yet. + +1. **`src/curate/embedding.rs`** (§4.3, §7.1, §7.2): `EmbeddingBackend` trait mirroring `ChatBackend` + (with a mock for tests), `VoyageBackend` (`POST {base_url}/embeddings`, `Authorization: Bearer`, + body `{input, model, input_type: "document"|"query", truncation: true, output_dimension, output_dtype: "float"}`, + response mapped by index and length-checked, usage `total_tokens` metered), batching by `batch_size` + with `buffer_unordered(max_concurrent_requests)`, retry 429/5xx via `RetryPolicy`, f32 + little-endian BLOB encode/decode with length and finiteness checks, `dot(a, b)` with a dimension + check, and cache orchestration: `article_embeddings` keyed by `(model, dimension, input_hash = sha256 + of the embedded text)`, overwritten on change. Embedded text = `"Title: {title}\n\n{plain body}"` + via `curate::prompt_text`, whitespace collapsed, cut at `max_input_chars` on a char boundary — + **no feed name, author or scores**. Interests go to `interest_embeddings` with `input_type = "query"` + and the bare interest string. A failed batch leaves those articles without embeddings; never fatal. + A Voyage `UsageMeter` with `price_per_mtok = 0.02` and `max_daily_usd` (0.50) as a runaway guard. + Key only from `DAILY_EPUB_VOYAGE__API_KEY`. Raw vectors never reach logs or reports. +2. **Config** (§4.3, §19): `[voyage]` (`enabled`, `base_url = "https://api.voyageai.com/v1"`, + `model = "voyage-4-lite"`, `output_dimension = 512`, `batch_size = 32`, `max_concurrent_requests = 4`, + `max_input_chars = 60000`, `max_daily_usd = 0.50`) and the complete `[curation.ranking]` block from + §19 (`triage_max`, `deep_keep`, `shortlist_keep`, `assessment_reuse_days`, `rating_lookback_days`, + `rating_half_life_days`, `neighbour_k`, `negative_coefficient`, `knn_floor/full`, `feed_floor/full`, + `semantic_min_words`, `exploration_slots`, `embedding_retention_days`, `telemetry_retention_days`, + `[curation.ranking.quotas]`, `[curation.ranking.weights.preliminary]`, `[curation.ranking.weights.utility]`, + `[curation.ranking.diversity]`) with the plan's defaults and the validation rules in §19 (weights + non-negative; `deep_keep ≥ shortlist_keep ≥ target_article_count`; `*_full > *_floor ≥ 0`; + `0 ≤ cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; batch sizes ≥ 1; dimension ∈ {256,512,1024,2048}). + Steps 4–5 will *use* the ranking keys; you only add and validate them. Keep `prefilter_keep` for now. + Startup logs whether Voyage is enabled. Update `config.example.toml` and the README. +3. **`src/curate/signals.rs`** (§9): a `Signals` type where every signal is `Option` (absent ≠ 0): + `interest` (§9.1 z-scored interest match, top-three interests recorded; raw top-1 cosine fallback + under 30 embedded articles, logged), `knn` (§9.2 preference state from `db::current_ratings(rating_lookback_days)` + joined to `article_embeddings`, decayed weights `value × 0.5^(age/half_life)`, top-k positives and + negatives, `knn = pos − negative_coefficient × neg`, top-three neighbours recorded, gate ramp + `clamp((n − knn_floor)/(knn_full − knn_floor), 0, 1)` multiplying the weight, absent at 0), + `feed` (§9.3 Beta-smoothed decayed rate credited to distinct direct `SourceKind::Feed` feeds, mean + over the article's rated feeds, gate `feed_floor/feed_full`), `social` (existing composite; absent + with no rows), `heuristic` (`longform_points(word_count)` − excerpt-only − roundup penalty, from + `prefilter.rs` **without** the social, Scour/HN, multi-source terms; expose those pieces as + functions). Log once per run: + `preference: N rated articles with embeddings → knn gate X; feed gate Y (n=…)`. + Also implement the **mid-rank percentile normalizer** of §12.2 and the **preliminary blend** of + §12.4 (present-and-active weights renormalized) so `signals_json.norm`/`weights` can be written now. +4. **Pipeline**: after social enrichment, for every article that passes hygiene, compute embeddings + (cached), signals, and the preliminary blend, then run the old prefilter as today. New stage + timings `embed` and `signals`. `--skip-embeddings` uses cached embeddings only (zero Voyage calls); + `--skip-llm` unchanged. `StageCounts` gains `eligible`, `embedded`, `rated_with_embeddings`, and + the report/`print_report` show them. The paper must still build when Voyage is down or unconfigured + (§17): `interest`/`knn` absent, never a penalty. +5. **`src/curate/telemetry.rs`** (§7.4, §7.5): the `candidate_runs` writer. One row per considered + article per run, upserted with `INSERT … ON CONFLICT(run_id, article_id) DO UPDATE` setting every + column. Hygiene-excluded articles get thin rows (`stage='excluded'`, `excluded_reason` ∈ + `blocked | published_before | recently_rejected`, `signals_json='{}'`). In this step map the old + pipeline onto the stage vocabulary: passed hygiene → `eligible`; cut by the prefilter → + `eligible` + `excluded_reason='not_admitted'`; prefilter survivors → `admitted`; Stage-A scored → + `assessed`; sent to selection → `shortlisted`; picked → `selected` (+ `editor_why` if a `why` + exists on the pick, else NULL); unpicked shortlist → `excluded_reason='not_selected'`. `signals_json` + follows §7.5 exactly (`v: 1`, `raw`, `norm`, `present`, `weights` = effective preliminary weights, + `top_interests`, `neighbours`, `exploration: false`, `auto_include`, `notes`). `utility` and + `rank_utility` stay NULL until step 5. +6. **CLI** (§15.2, §16) in `src/main.rs`: + `explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]` and + `explain --date YYYY-MM-DD --near-misses [N]` printing the persisted `candidate_runs` row for the + latest non-dry run of that date (or `--run-id`): stage and reason; every raw/normalized signal with + presence and effective weight; top interests with z; nearest rated neighbours; assessments from + `article_assessments` when present (empty until step 4); admitted_by; the editor's `why`. `--url` + canonicalizes (`dedupe`) and looks the article up; if absent from `articles`, print that it was never + ingested. `--near-misses` lists the top N by preliminary blend (utility from step 5 when present) + that were not selected. `features backfill [--days 30] [--rated-only] [--all] [--yes]` embeds rated + and published articles first, then interests, then other recent articles only under `--all`, prints + an estimate and asks for confirmation above 5M tokens unless `--yes`, and is idempotent (warm cache + ⇒ zero calls). `features prune` removes `article_embeddings` rows for articles neither rated nor + published older than `embedding_retention_days` and `candidate_runs` rows whose run started more + than `telemetry_retention_days` ago. `features backfill` takes no lock yet (step 6 adds `lock.rs`). +7. Types (§18): `Signals`, `Signal` helpers, `RatedArticle` already exists — extend if needed. Do not + create `Candidate` yet (step 4/5). + +Tests (§20 "Embeddings", "Interest z-scores", "Preference", "Normalization", plus telemetry/explain): +BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed +text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the +others; embedded text contains no feed name or author; a broad interest with uniformly high cosine does +not dominate while a specific interest with one strong match does; raw fallback under 30 articles; one +loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high +(anti-centroid); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate +0 below `knn_floor`, 1 at `knn_full`, linear between; feed credit sums to 1 across direct feeds; feed +affinity uses the mean; constant signal → 0.5 for everyone; ties get equal percentiles (400 identical +zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate +missing a signal is scored on the rest; `candidate_runs` rows written for every considered article with +the right stage/reason on a mocked run; `--skip-embeddings` makes zero Voyage calls; a Voyage failure +still publishes; `explain` renders from persisted rows and reports "never ingested"; `features prune` +respects rated/published. diff --git a/docs/plans/curation-v2-briefs/step4.md b/docs/plans/curation-v2-briefs/step4.md new file mode 100644 index 0000000..68348f1 --- /dev/null +++ b/docs/plans/curation-v2-briefs/step4.md @@ -0,0 +1,89 @@ + +--- + +# YOUR STEP: 4 — Triage replaces the gate (plan §21 step 4) + +Steps 1–3 have landed: three-way votes and `rating_events`; the Claude editor (`Llms { bulk, editor }`, +`AnthropicBackend`, per-provider `UsageMeter`s, `why` lines, the Brief); Voyage embeddings, +`signals.rs` (interest/knn/feed/social/heuristic, percentile normalizer, preliminary blend), +`telemetry.rs` (`candidate_runs` writer), `explain`, `features backfill|prune`, and the full +`[curation.ranking]` config. Read `git log -6`, then the current `src/curate/` (all files), +`src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs`, `src/types.rs`. + +Scope (plan §7.3 reuse and churn rules, §8.1 hygiene, §10 triage, §11 admission and exploration, +§12.4 preliminary blend already exists, §15.4 log line parts, §17 DeepSeek row, §19 `deep_keep`): + +1. **Migration `migrations/0003_drop_scores.sql`**: `DROP TABLE scores;` (deferred from step 1). + Remove `db::upsert_score`, `recently_low_scored_ids` and every `scores` reader. The churn rule + now reads `article_assessments`: an article whose latest `triage` `score < recent_rejection_floor` + (3.0) or `deep` `score < recent_rejection_floor` within `recent_rejection_days` (7) is + `recently_rejected`, unless auto-include (§7.3, §8.1). +2. **`src/curate/triage.rs`** (§10): `TRIAGE_INSTRUCTIONS` and `TRIAGE_PROMPT_VERSION = 1` verbatim + from the plan; per-article block exactly as §10 (id, title, feed (category), author, length · + excerpt only, opening = first 200 words via `prompt_text` + `truncate_words`, `matches interests:` + from `top_interests` with z ≥ 1.5 labelled strong (z ≥ 2.5) / weak, `closest rated:` from + neighbours with cosine ≥ 0.55; omit lines with nothing). Batches of `deepseek.triage_batch_size` + (25) through `buffer_unordered(max_concurrent_requests)` on the **bulk** client, budget check + before each batch. Tolerant parsing generalized from `score.rs::parse_score_response`: a + malformed item never sinks the batch, ids not in the batch are dropped, unknown `kind` → `other`, + `interest` clamped 0–10. Persist every result to `article_assessments (stage='triage', model, + prompt_version, profile_version, score, kind, rationale=why, assessed_at)`. + **Cache** (§7.3): skip articles with a reusable `triage` or `deep` assessment (same `model` and + `prompt_version`, `assessed_at` within `assessment_reuse_days`); `generate --rescore` ignores the + cache. A failed batch leaves `triage` absent for its articles. + **Pool cap** (§10): if eligible > `triage_max` (800), triage the union of top `0.7 × triage_max` + by preliminary blend, top 100 by `interest`, top 100 by `knn` (if active), all auto-includes, + filled to `triage_max` by blend; the rest get `stage='eligible'`, `excluded_reason='not_admitted'`. +3. **`src/curate/admit.rs`** (§8.1, §11): hygiene moves here (`blocked`, `published_before` = in + `issue_articles` for any issue date before this run's date, `recently_rejected` per item 1; + auto-includes never excluded) writing the thin `candidate_runs` rows, and the union admission + into `deep_keep` (120) slots in this order, each retriever taking its top-N by its own signal among + not-yet-admitted, not-excluded articles, recording every retriever that would have taken an + article in `admitted_by` (first = the one that admitted it): + `auto_include` (uncapped) → `triage` (quota 60, floor `interest ≥ 5`) → `interest` (20; floors + `word_count ≥ semantic_min_words`, not `looks_like_roundup`, triage `interest ≥ 3` if triaged) + → `knn` (20, gate > 0, same floors) → `exploration` (5, §11.1) → `blend` (remaining). + Inactive retrievers release their quota to `blend`. Not admitted → `stage='triaged'` (or + `'eligible'` if never triaged) + `excluded_reason='not_admitted'`. + **Exploration** (§11.1): five slots for articles ranked between `deep_keep` and `deep_keep × 2.5` + by the preliminary blend with `word_count ≥ 300`, not roundups, triage `interest ≥ 4`; ordered by + `sha256(run_date || article_id)`; flagged `exploration = true` through to the editor prompt. +4. **Prefilter reduced** (§18): `prefilter.rs` keeps only hygiene helpers (`is_blocked`, + `is_auto_include`, `looks_like_roundup`) and the text heuristic pieces `signals.rs` uses. Delete + `prefilter::run`, `score_article`'s social/Scour/HN/multi-source terms, `PrefilterContext`, the + `prefilter_keep` config key (replace uses with `curation.ranking.deep_keep`; the config validation + `deep_keep ≥ shortlist_keep ≥ target_article_count` already exists), and `Curator::prefilter`. + A stale `prefilter_keep` key in a config file must fail loudly with a message naming + `curation.ranking.deep_keep` (follow the `bookorbit_dir` precedent in `config.rs`). +5. **Pipeline**: hygiene → embeddings → signals → preliminary blend → triage → admission → the + *existing* Stage A scoring (`score.rs`, unchanged this step) over the admitted deep set → the + editor → editorial, as today. The deep set (`admitted`) is what Stage A scores and what the editor + sees; step 5 replaces Stage A with deep assessment and adds the shortlist. Telemetry stages now: + `excluded | eligible | triaged | admitted | assessed | shortlisted | selected` with reasons per §7.4; + `admitted_by` and `exploration` written; `signals_json.raw/norm/present` gain `triage` (÷10 for norm). + Stage timings `triage` and `admit`; `StageCounts` gains `triaged`, `admitted`, `admitted_by` (map), + `exploration_admitted`, `exploration_selected`. Log the admission line from §15.4: + `admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0`. + `--skip-llm` skips triage; DeepSeek down or tripped ⇒ no triage, admission by + `interest`/`knn`/`blend`, editor still runs (§17). The editor prompt (§13 rendering) shows the + triage score and `flags: exploration` where present. `explain` prints the triage assessment + (score, kind, why) and `admitted_by`. +6. `types.rs` (§18): introduce `Assessment { triage: Option, deep: Option }` and + `Triage { interest, kind, why, model, prompt_version, assessed_at }` (keep `Deep`/`LlmScore` + compatible with the existing Stage A output for now), and `Candidate { article, auto_include, + exploration, signals, assessment, utility: Option, cluster: Option<...>, admitted_by: Vec, + stage, excluded_reason }` replacing `ScoredArticle` where the pipeline flows through admission. + Keep the change mechanical where `score.rs`/`select.rs` still consume the old shape (step 5 + retires them); an adapter is acceptable. +7. `runs.config_json` gains `TRIAGE_PROMPT_VERSION`; `config.example.toml` and README updated + (`prefilter_keep` removed, `[deepseek].triage_batch_size`, `--rescore`). + +Tests (§20 "Triage and deep parsing" triage half, "Admission", churn/cache): realistic triage +fixture parsed; malformed items do not sink a batch; unknown kind → other; every `kind` token in the +prompt round-trips; cached triage reused within `assessment_reuse_days` and ignored with `--rescore`; +churn rule excludes a recent low triage score and spares auto-includes; a strong-interest, +weak-heuristic, no-social article reaches the deep set; a 60-word stub with high interest similarity +is not admitted by `interest`/`knn`; quotas honoured; inactive retrievers release quota; exploration +deterministic per date and rotating across dates; auto-includes always admitted; excluded articles get +thin rows with the right reason; pool cap path yields `not_admitted` rows; the mocked full pipeline +still publishes with DeepSeek failing; migration 0003 drops `scores`. diff --git a/docs/plans/curation-v2-briefs/step5.md b/docs/plans/curation-v2-briefs/step5.md new file mode 100644 index 0000000..9e06900 --- /dev/null +++ b/docs/plans/curation-v2-briefs/step5.md @@ -0,0 +1,69 @@ + +--- + +# YOUR STEP: 5 — Deep assessment, utility, diversity (plan §21 step 5) + +Steps 1–4 have landed; triage and union admission now gate the deep set and the old prefilter is +reduced to hygiene helpers. Read `git log -8`, then all of `src/curate/`, `src/pipeline.rs`, +`src/types.rs`, `src/config.rs`, `src/report.rs`, `src/main.rs`, `src/db.rs`. + +Scope (plan §12 in full, §13 rendering with facets/neighbours, §7.3 deep rows, §18 module retirements): + +1. **`src/curate/assess.rs`** replaces `score.rs` (§12.1): `DEEP_INSTRUCTIONS` and + `DEEP_PROMPT_VERSION = 1` verbatim from the plan (the section palette is substituted from + `curation.sections`). Per article: title, feed, author, length, excerpt-only flag, the triage + `why`, the interest and neighbour hint lines from §10, and a **representative sample** of + ~2,000 tokens: body under ~1,500 words sent whole; otherwise first 600 words, 500 around the + midpoint, last 400, with visible `[BEGINNING]`/`[MIDDLE]`/`[END]` markers, split on word + boundaries. Batches of `deepseek.deep_batch_size` (8, replaces `score_batch_size`) through + `buffer_unordered`, bulk client, budget check per batch. Auto-includes are assessed too. + Tolerant parsing: `quality`/`fit` clamped 0–10, `category` validated against the palette (invalid → + `None`, the editor decides), `paywalled_guess`, and `facets` where every unknown enum token degrades + to `None` and `specific_topics` is capped at 3. **Every enum token in the prompt must round-trip + through the parser (test).** Persist to `article_assessments (stage='deep', score=quality, fit, + kind=facets.format, facets_json, rationale, category, paywalled_guess, ...)`; reuse per §7.3 with + `--rescore` bypass. Facets are shown to the editor, the profile rebuild and `explain`; **not** a + numeric signal. +2. **`src/curate/rank.rs`** (§12.2–§12.5): normalization (LLM scores ÷ 10; everything else mid-rank + percentile over the deep set — reuse/move the step-3 normalizer), the **utility** blend over + present signals with `[curation.ranking.weights.utility]` renormalized and learned signals + multiplied by their gate ramp first, stored 0–100; the preliminary blend stays for the eligible set. + **Diversified shortlist** (§12.5): leader clustering by embedding cosine with `cluster_threshold` + (0.85) and `per_cluster_cap` (2): sort by utility desc then article id asc; assign each article to + the first existing cluster whose *leader* has cosine ≥ threshold, else make it a new leader; + articles without embeddings are singletons; admit in order while the cluster's admitted count is + below the cap until `shortlist_keep` (60); the top `utility_protected` (10) by utility and all + auto-includes are admitted regardless and still count toward their cluster; exploration picks that + reached the deep set get up to 3 reserved slots; if short, relax to cap 3, then uncapped. Persist + `cluster_id`, `cluster_rank`, `rank_utility`, `utility`, `excluded_reason ∈ cluster_suppressed | + shortlist_cap`, stage `shortlisted`. +3. **`src/curate/editor.rs`** replaces `select.rs`: the §13 rendering in full — + `quality 8.5 · fit 7.0 · triage 8.0 — `, `facets: format · depth · evidence · + technicality · topic_group`, `matches:`, `closest rated:`, `flags: exploration | always-include | + excerpt only`, `opening:` first 60 words. The editor sees the 60-item shortlist. `assemble()`'s + `hard_max` trim and `select_without_llm` now order by **utility**, falling back to the preliminary + blend. Delete `ScoredArticle::combined_score()` and `ScoredArticle` itself; `Candidate` is the only + flow type. Move the tests from `score.rs`/`select.rs` into `assess.rs`/`editor.rs` and delete the + old files (update `curate/mod.rs`, `Curator`). +4. **Editorial inputs**: the Brief and the summaries use quality/fit where the plan says so (§14.2 + input lists quality/fit). The weekly profile rebuild's rated lines now get real facets. +5. **Pipeline/report**: stage timings `assess` and `rank`; `StageCounts` gains `assessed`, + `shortlisted`, `clusters`; the §15.4 curation line + `curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected`. + `explain` prints the deep assessment (quality, fit, category, rationale, facets), utility and rank, + cluster id and what suppressed it; `--near-misses` orders by utility. `runs.config_json` gains + `DEEP_PROMPT_VERSION`. `--skip-llm`/DeepSeek down ⇒ no deep assessment, utility over present + signals (triage/interest dominate, §12.3), editor still runs on what it has (§17). +6. `config.example.toml`/README: `deep_batch_size` replaces `score_batch_size`; a stale + `score_batch_size` key fails loudly naming `deep_batch_size`. + +Tests (§20 "Triage and deep parsing" deep half, "Normalization", "Clustering", "Editor" utility +parts): realistic deep fixture; malformed items do not sink a batch; unknown facet tokens → `None`; +every enum token round-trips; representative sample has the three markers and respects word +boundaries, short bodies are sent whole; cached deep rows reused / bypassed with `--rescore`; constant +signal → 0.5; ties equal; absent values do not shift others; effective weights sum to 1; a candidate +missing a signal is scored on the rest; near-duplicates share a cluster and the third is suppressed; +protected top-N survive and count; the bridge case (A~C, B~C, A≁B, utility A>B>C) yields two clusters; +articles without embeddings are never suppressed; `hard_max` trims by utility; `select_without_llm` +orders by utility; the mocked full pipeline publishes with DeepSeek down and writes `shortlisted` +rows with cluster ids. diff --git a/docs/plans/curation-v2-briefs/step6.md b/docs/plans/curation-v2-briefs/step6.md new file mode 100644 index 0000000..fc46a54 --- /dev/null +++ b/docs/plans/curation-v2-briefs/step6.md @@ -0,0 +1,62 @@ + +--- + +# YOUR STEP: 6 — Paper telemetry, stats, lock (plan §21 step 6) + +Steps 1–5 have landed: the full v2 ranking pipeline (hygiene → embeddings → signals → triage → +admission → deep assessment → utility → cluster-capped shortlist → Claude editor → editorial), +`candidate_runs` telemetry, `explain`, `features`, `ratings`. Read `git log -10`, then +`src/pipeline.rs`, `src/report.rs`, `src/main.rs`, `src/curate/telemetry.rs`, `src/epub/chapters.rs` +and `src/epub/templates/`, `src/types.rs` (`Colophon`), `src/db.rs`, README, `config.example.toml`. + +Scope (plan §5 lock, §15.1 Behind the paper + In-this-issue, §15.3 `stats`, §15.4 report/log block, +§16 lock coverage, §19 startup logging, README/config docs): + +1. **Behind the paper** (§15.1): a new short chapter after the World Briefing and before the colophon + (`src/epub/templates/behind.xhtml`, `render_behind_the_paper`) with exactly the content shape in + the plan: the considered/eligible/triaged/read-closely/shortlisted/selected line; the + `Admitted via:` line; the `Learned signals:` line (rated-with-embeddings count and knn percentage, + feed affinity state); `Near misses` — the 10 highest-utility non-selected articles as + ` — <feed> · quality X · fit Y · <stage/reason>`; the `Models:` line with triage/assessment, + editor/summaries and embeddings models; cost and generation time. Both editions (X4 gets the same + text, no links). The data comes from the run's `RunReport`/`StageCounts` and the run's + `candidate_runs` rows (add a `BehindThePaper` struct to `types.rs` filled by the pipeline; keep + templates pure). Chapter ids stable (`behind`). +2. **In this issue** page: verify each entry shows the `why` line under the summary (step 2 was asked + to do this; complete it if missing). The colophon keeps its fields and shows per-provider cost + lines and models (verify; complete if missing). +3. **`stats`** (§15.3): `daily-epub stats [--days 14]` printing issues, articles published, explicit + ratings by label, ratings per issue, up/down ratio per admitting retriever (`admitted_by[0]` of + rated picks, from `candidate_runs` joined to `rating_events` via the latest explicit event), + exploration yield (exploration picks selected / admitted, and how many were rated positively), + mean issue size, cost per day per provider (from `runs.provider_costs_json`), mean generation time. + Plain text, one fact per line, no tables wider than 80 columns. +4. **Run report and log block** (§15.4): make sure `StageCounts` has every field listed — + `eligible`, `embedded`, `triaged`, `admitted`, `admitted_by` (map), `assessed`, `shortlisted`, + `clusters`, `exploration_admitted`, `exploration_selected`, `verdicts_in_prompt`, + `rated_with_embeddings`, per-provider usage — and that stage timings include `embed`, `signals`, + `triage`, `admit`, `assess`, `rank`, `editor`, `summaries`, `brief`. Emit the four-line info block + from §15.4 once per run (`curation:`, `admission:`, `preference:`, `providers:`). `print_report` + in `main.rs` prints the same lines. +5. **`src/lock.rs`** (§5): `flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`, taken in `main` + for `generate`, `profile rebuild`, `features backfill`, and `backfill-social`; a second invocation + exits with `generate is already running` (name the command that holds it if cheap, else the + generic message). `serve`, `explain`, `stats`, `ratings`, `db migrate`, `features prune` do not + take it. Use `libc`/`rustix` only if already in the dependency tree (check `Cargo.lock`); otherwise + `std::fs::File` + the `fs2`-free approach via `libc::flock` is acceptable to add as a tiny dep — + say which in the report. Twenty lines, no table, no TTL. +6. **Docs**: README rewritten where it describes the pipeline, feedback links, costs, prerequisites + (Anthropic and Voyage keys, dashboard spend limits as the real backstop, the server-side fallback + note), the CLI (`ratings`, `explain`, `stats`, `features`, `--rescore`, `--skip-embeddings`, + `--max-articles` as ceiling), and `data/profile.md`. `config.example.toml` carries every key from + plan §19 with comments. Startup logs resolved models and per-provider enabled state (verify). +7. `systemd/daily-epub-generate.service`: no change needed unless an env var name changed; note the + two new env vars in the README's systemd section. + +Tests (§20 "Lock", "Pipeline" behind-the-paper, stats): two processes/one wins (spawn the binary or +use two `File` handles from separate threads with `try_lock` semantics — no sleeping longer than a +second), a killed holder frees the lock, `serve` does not take it (parse-level or by checking no lock +file is created); the behind-the-paper chapter renders with the counts and near misses on the mocked +pipeline run and is parseable XHTML in both editions (extend `tests/m4_epub.rs`); `stats` runs on a +temp DB with a couple of runs, issues and rating events and prints every line; the info block appears +in the log (capture with `tracing` test subscriber or assert on the formatted strings). diff --git a/docs/plans/curation-v2-briefs/step7.md b/docs/plans/curation-v2-briefs/step7.md new file mode 100644 index 0000000..e44be2d --- /dev/null +++ b/docs/plans/curation-v2-briefs/step7.md @@ -0,0 +1,29 @@ + +--- + +# YOUR STEP: 7 — Cleanup (plan §21 step 7) + +Steps 1–6 have landed. This step removes what the plan retired and updates the implementation +notes. Read `git log -12` and skim every file under `src/` for leftovers. + +1. **Dead code**: anything only the old prefilter/scores/select/feed-prior paths used — unused + functions, types (`LlmScore` if nothing reads it, `ScoredArticle`, `FeedPrior`, `Rating`), + `kv` keys no longer read, config aliases and `#[serde(alias)]`s introduced as transition aids, + `// filled in by step N` comments, stale doc comments that still describe 👍/👎, "From the Editor", + section intros, prefilter gating, or `async-openai`. `cargo clippy --all-targets -- -W dead_code` + must be clean; do not add `#[allow(dead_code)]`. +2. **Prune paths** (§7.1, §7.4): confirm `features prune` covers `article_embeddings` + (`embedding_retention_days`) and `candidate_runs` (`telemetry_retention_days`); add + `article_assessments` older than `telemetry_retention_days` to it. Make `generate` call the prune + once per run after publishing (best effort, logged). +3. **Docs**: update `docs/plans/2026-08-15-implementation-notes.md`: fix item 5 (hand-rolled + `reqwest` client, not `async-openai`), add the new providers (Anthropic Messages API facts from + plan §4.2, Voyage facts from §4.3, each with "verified 2026-09-02"), the new tables, the lock, the + budget-day rule, and a short "curation v2" section pointing at + `docs/plans/2026-09-02-personalized-curation-v2.md`. This is the one step allowed to edit `docs/`. + Do not edit the plan itself. +4. **Consistency pass**: `config.example.toml` matches `Config::default()` key for key (write a test + that loads the example and compares every `[curation.*]`, `[anthropic]`, `[voyage]`, `[editorial]` + value to the defaults, since the defaults are the plan's numbers); README CLI section matches + `daily-epub --help` output for every subcommand (spot check by running the binary). +5. Run `cargo fmt`, `cargo clippy --all-targets`, `cargo test`; report the summary lines.