diff --git a/docs/plans/2026-09-02-curation-v2-progress.md b/docs/plans/2026-09-02-curation-v2-progress.md index 5b589ad..7eacb61 100644 --- a/docs/plans/2026-09-02-curation-v2-progress.md +++ b/docs/plans/2026-09-02-curation-v2-progress.md @@ -1,6 +1,6 @@ # Personalized Curation v2 — implementation progress and handoff -**Updated:** 2026-09-02 (end of session 1) +**Updated:** 2026-09-02 (end of session 2) **Plan:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§21 is the step sequence) **Branch:** `curation-v2` (branched from `main` at `a599745`; not merged, not pushed) **Briefs:** `docs/plans/curation-v2-briefs/` — `00-preamble.md` + one `stepN.md` per step. Each @@ -13,43 +13,81 @@ brief was handed to an implementation agent as `cat 00-preamble.md stepN.md`. | 1. Feedback and profile | **done**, reviewed (codex review: no actionable issues) | `3a9f4b9` | | 2. Claude editor and editorial | **done** | `57efbb4` | | 3. Embeddings, signals, telemetry | **done** (branch `curation-v2-step3`, merged) | `ea3b141` + merge commit | -| 4. Triage replaces the gate | not started — brief ready: `step4.md` | | -| 5. Deep assessment, utility, diversity | not started — brief ready: `step5.md` | | -| 6. Paper telemetry, stats, lock | not started — brief ready: `step6.md` | | -| 7. Cleanup + implementation notes | not started — brief ready: `step7.md` | | +| 4. Triage replaces the gate | **done** (Codex plugin, high effort), reviewed | `10f4afd` | +| 5. Deep assessment, utility, diversity | **done** (Codex started, cut off by quota; Claude agent finished), reviewed | `05a74a0` | +| 6. Paper telemetry, stats, lock | **done** (Claude agent), reviewed | `d261cd4` | +| 7. Cleanup + implementation notes | **done** (Claude agent), reviewed | `d403c51` | -`cargo fmt --check`, `cargo clippy --all-targets` (only the three pre-existing `src/world.rs` -`needless_borrow` warnings) and `cargo test` (294 lib tests + 6 integration suites) are green at HEAD. +All seven plan steps are implemented. `cargo fmt --check`, `cargo clippy --all-targets` +(including `-W dead_code`) and `cargo test` (329 lib tests + 6 bin tests + the 7 integration suites) are green at HEAD. Nothing has been run against the production database or the real providers. No local `config.toml`, database, or API keys exist on the dev box, so verification so far is tests only. -## Decisions and deviations made while implementing (read before step 4) +## Decisions and deviations made while implementing + +Steps 1–3 (session 1): -- **`scores` table not yet dropped.** Migration `0002_curation_v2.sql` created every new table, copied - `ratings` → `rating_events`, and dropped `ratings` and `feed_priors`, but kept `scores` because the - old Stage A scoring (`src/curate/score.rs`, `db::upsert_score`, `recently_low_scored_ids`) still - uses it. Step 4 adds `migrations/0003_drop_scores.sql` and moves the churn rule to - `article_assessments` (already in `step4.md`). -- **Old prefilter still gates.** Hygiene → embeddings → signals → preliminary blend now run for every - article, and `candidate_runs` rows are written with the stage vocabulary mapped onto the old flow - (`admitted_by` is `["prefilter"]`/`["auto"]` for now), but `prefilter::run` + `prefilter_keep` - still decide the deep set until step 4. - **Legacy `up` links.** `Vote::parse("up")` → `Loved`, and `auth::verify_token` also accepts tokens - signed over the literal `up` segment so already-published issues keep working. + signed over the literal `up` segment so already-published issues keep working. Keep both. - **Same-date regeneration** no longer excludes its own picks (`published_before` uses issue dates strictly before the run date), per plan §8.1. - **`VoyageConfig.api_key`** exists as a field (figment maps `DAILY_EPUB_VOYAGE__API_KEY` into it; `deny_unknown_fields` would otherwise reject the env var). Never document it in TOML. -- **`anthropic.max_concurrent_requests`** is validated but not consumed yet; summary concurrency is +- **`anthropic.max_concurrent_requests`** is validated but not consumed; summary concurrency is the constant `SUMMARY_CONCURRENCY = 4` in `editorial.rs`. -- `UsageMeter::new(&DeepseekConfig, ..)` survives as a compat constructor over `with_prices`. -- `tests/fixtures/deepseek_front_page.json` was replaced by `tests/fixtures/claude_brief.json`. - `RATINGS_LOOKBACK_DAYS` in `profile/mod.rs` is effectively unbounded (36,500) for the prompt verdict block and the weekly rebuild; the knn/feed preference state uses `curation.ranking.rating_lookback_days` (180) as the plan says. - Footer CSS: `.rating` has no `white-space: nowrap` (it would clip on narrow e-ink screens). +Step 4: + +- Hygiene order is blocked → published_before → recently_rejected, all skipped for auto-includes. +- `admitted_by` records every retriever whose own top-N would have taken the article (first entry + is the admitting one); exploration is flagged only when `exploration` admitted it. +- Under the pool cap (`triage_max`) the surplus gets `stage='eligible'`, `excluded_reason='not_admitted'`. +- Migration `0003_drop_scores.sql` drops `scores`; the churn rule reads `article_assessments`. + +Step 5: + +- `Candidate` is the only flow type; `ScoredArticle`, `combined_score`, `score.rs`, `select.rs` are gone. +- The editor omits the `facets:` line when every facet is unknown (plan renders it unconditionally). +- Articles cut at the shortlist keep the stage they reached (`admitted` if never deep-assessed, + else `assessed`) with `excluded_reason` `cluster_suppressed` (hit the base cap) or `shortlist_cap`. +- `Curator::assess` is a no-op under `--skip-llm`; cached deep rows are still reused when DeepSeek + is merely down. +- `--near-misses` and the Behind-the-paper list order by utility, falling back to the preliminary + blend per row. +- Fixtures: `tests/fixtures/deepseek_deep_batch{,_messy}.json` replace the old score fixtures; + `deepseek_triage_batch.json` added in step 4. + +Step 6: + +- `BehindThePaper` hangs off `Issue.behind` (filled after `build_issue`), not a `build_issue` argument. + Generation time and cost shown in the chapter are measured at issue assembly (they exclude the + EPUB build and publish, a few seconds); the cost includes Voyage. +- Voyage spend is recorded under `provider_costs["voyage"]`; `finish()` counts it once and keeps its + tokens out of the LLM aggregate. Voyage spend is not preloaded into its meter for the UTC day. +- `src/lock.rs` uses `libc::flock` (`libc` was already in `Cargo.lock`; now a direct dependency). + The holder's command name is written into the lock file for the "X is already running" message. +- The mid-run `admission:` line is `debug`; the four-line §15.4 block is logged once after + `finish()` and printed by `print_report`. Timings `summaries` and `brief` replace `editorial`. + +Step 7: + +- `features prune` sweeps `article_embeddings`, `candidate_runs` and now `article_assessments`; + `generate` runs it once after a published (non-dry) issue, best effort. +- The Brief chapter's TOC/title string is now "The Brief" (was still "From the Editor"). +- Kept on purpose: `UsageMeter::new(&DeepseekConfig, ..)` (a convenience over `with_prices`, 12 + call sites); `StageCounts.candidates` (feeds `runs.candidates` and the colophon); the + `prefilter_keep`/`score_batch_size` removed-key startup errors (operator guards); the SQL comment + "filled in by step 4/5" inside the applied migration `0002` (sqlx checksums); six v1-era + unreferenced `pub` items (`enrich_one`, `issues_before`, `today_in_tz`, `to_json_pretty`, + `with_retry_policy`, `db::issues_before`) as pre-v2 API surface. +- `config::tests::shipped_example_config_matches_the_defaults_key_for_key` compares the example + against `Config::default()` in both directions (numbers within 1e-6). + ## Operator to-dos before the first real run 1. Set `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` in the systemd env file. @@ -57,34 +95,37 @@ Nothing has been run against the production database or the real providers. No l guards, not accounting). 3. Copy `data/profile.md` to wherever `profile_path` points on the server (default is relative to `WorkingDirectory=/var/lib/daily-epub`, like `data/scour-interests.opml`). -4. Run `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`; back up the DB first). +4. Back up the DB, then `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`, 0003 drops `scores`). 5. `daily-epub features backfill --rated-only` then `--days 30` to warm the embedding cache. -6. A `generate --dry-run` and read the paper; `explain --near-misses` once step 4 lands. +6. `daily-epub generate --dry-run`, read the paper (including the new Behind-the-paper chapter), + then `explain --date … --near-misses` and `stats`. +7. Watch the first few real runs' `providers:` log line against the ~$0.80/day estimate (plan §3). ## How the work was run (so the next session can repeat it) - Orchestrator: Claude Code (this repo), one implementation agent per step, review + commit by the orchestrator after independent `cargo fmt/clippy/test`. -- Codex: `codex exec -C --sandbox workspace-write --add-dir ~/.cargo -c - sandbox_workspace_write.network_access=true -c model_reasoning_effort=high -o - < `, - detached with `setsid nohup`, exit code written to a file and watched with a monitor. The - `openai-codex` Claude Code plugin's `task` runs fail on this host (bubblewrap cannot create user - namespaces: `kernel.apparmor_restrict_unprivileged_userns = 1`); the CLI works if the brief tells - the agent to edit files via shell commands instead of the `apply_patch` tool (see the preamble). - The plugin's read-only `review --background --scope working-tree` does work and was used on step 1. - Fix for the sandbox: `sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0`. -- Codex ran out of ChatGPT usage after ~560k tokens (step 1 complete, steps 2 and 3 cut off - mid-way); Claude `general-purpose` sub-agents finished steps 2 and 3 from the partial trees and - resolved the step 2/3 merge. Budget roughly one large step per Codex usage window. -- Steps 2 and 3 were run in parallel (step 3 in a git worktree); the merge cost ~19 conflict hunks - in config/pipeline/report/main/README/config.example. Steps 4 → 5 → 6 → 7 are sequential. +- **Codex via the `openai-codex` Claude Code plugin (v1.0.6) works on this host as of session 2** + (the operator fixed bubblewrap). Launch: + `node ~/.claude/plugins/cache/openai-codex/codex/1.0.6/scripts/codex-companion.mjs task --background --write --effort high "$(cat brief.md)"`, + poll `… status --json` (`.job.status`), read `… result `. Inside its sandbox + `bind()` on 127.0.0.1 is forbidden, so exactly ten pre-existing loopback tests fail there + (`curate::llm::tests::anthropic_*` ×4, `extract::tests::relative_urls_resolve_against_the_url_we_landed_on`, + `server::tests::*` ×5, plus the two `m7_server` integration tests); the brief must tell the agent + to ignore those and the orchestrator runs the full suite outside. The old "host quirk" paragraph + in `00-preamble.md` is obsolete. +- Codex usage budget is roughly one large step per ChatGPT usage window: in session 2 it finished + step 4 (~26 min) and was cut off 21 minutes into step 5 ("try again at 7:44 PM"). The job status + becomes `failed` with the partial work left in the tree; a Claude `general-purpose` agent with a + "finish, don't restart" brief (known failures, scope audit checklist) completed it. Steps 6 and 7 + ran on Claude agents directly. +- Steps 4 → 5 → 6 → 7 were sequential, each reviewed against the plan sections named in its brief + before committing. -## Next session: exact starting instructions +## Next session: what remains 1. `git checkout curation-v2 && cargo test` (expect green). -2. Read this file, then `docs/plans/curation-v2-briefs/00-preamble.md` and `step4.md`. -3. Launch the step 4 agent with `cat 00-preamble.md step4.md` as the prompt (Codex CLI as above, or a - Claude general-purpose agent — tell it to ignore the "host quirk" paragraph in that case). -4. Review the diff against plan §10–§11, run the checks, commit as "Curation v2 step 4: …". -5. Repeat for steps 5, 6, 7. After step 7: `git merge --no-ff curation-v2` into `main`, deploy, and - do the operator to-dos above. +2. Optional: a Codex `review --background --scope branch --base main` pass over the whole branch. +3. `git merge --no-ff curation-v2` into `main`, build, deploy, and do the operator to-dos above. +4. After a week of real runs: read `stats`, tune `[curation.ranking]` from what `explain` shows, + and revisit the deferred items in plan §23. diff --git a/docs/plans/curation-v2-briefs/00-preamble.md b/docs/plans/curation-v2-briefs/00-preamble.md index 3a83080..2348c41 100644 --- a/docs/plans/curation-v2-briefs/00-preamble.md +++ b/docs/plans/curation-v2-briefs/00-preamble.md @@ -44,9 +44,15 @@ End with a concise report (this is what the orchestrator reads): what you implem deviation from the plan and why, anything from the step you could not finish, and the exact `cargo test` summary line(s). Keep it under 60 lines. -## Host quirk (important) +## Host notes -On this machine the built-in `apply_patch` tool's filesystem helper fails with -`bwrap: ... Operation not permitted`. Do not keep retrying it. Edit files through shell commands -instead (the `apply_patch` CLI invoked from bash works, as do `python3 - <<'EOF'` rewrite scripts, -`sed -i`, and heredocs). Shell commands, `cargo build`, `cargo test` and `cargo clippy` all work. +- The `openai-codex` Claude Code plugin sandbox works on this host (bubblewrap fixed 2026-09-02): + `apply_patch`, the shell, `cargo build/test/clippy` and the warm `~/.cargo/registry` all work. +- Inside that sandbox `bind()` on 127.0.0.1 is forbidden, so ten pre-existing tests fail there for + environmental reasons: the four `curate::llm::tests::anthropic_*` tests, + `extract::tests::relative_urls_resolve_against_the_url_we_landed_on`, and the five + `server::tests::*` (the two `tests/m7_server.rs` tests likewise). They are out of scope; do not + touch them. Every other test must pass; the orchestrator runs the full suite outside the sandbox. +- Cargo's registry is already warm; do not add dependencies that would need a network fetch. +- Work autonomously to completion. Do not stop to ask questions; make the closest-to-plan choice + and record it in the final report.