diff --git a/README.md b/README.md index 315c7ae..46498ff 100644 --- a/README.md +++ b/README.md @@ -99,8 +99,7 @@ enabled on every request to an `anthropic`-kind provider. | **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF — see below. It has **no global npm bin** — it is invoked as `node /cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. | | A reverse proxy for `daily.hallada.net` → `127.0.0.1:3499` | rating links must be reachable from e-readers on the internet | TLS via your existing setup. | -`data/profile.md` is the hand-maintained reader profile; its optional interests are merged -with `data/scour-interests.opml`. Both paths are configurable. +`data/profile.md` is the hand-maintained reader profile; its path is configurable. --- @@ -136,6 +135,8 @@ daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes] daily-epub features prune # stale embeddings, old telemetry and assessments daily-epub backfill-social [--days 7] # re-poll social scores for recent articles daily-epub feeds discover [--days 14] [--limit 50] # seed feed candidates from recent aggregator articles +daily-epub interests import [--opml PATH] [--profile PATH] # one-time standing-interest import +daily-epub interests backfill # match compatible cached article embeddings to interests daily-epub db migrate # run migrations (also automatic on every start) daily-epub config check # validate the config, print the resolved roles, keys and paths daily-epub users add USER [--admin] [--password-stdin] @@ -213,7 +214,10 @@ sees complete issues and article chapters and can download available formats from a single download menu. An `admin` can additionally follow direct dashboard links from issue entries and article chapters, rate articles, and use every `/dashboard/*` page, including -settings and jobs. Personalization is shared across accounts for now. +settings and jobs. `/dashboard/interests` lists the standing interests with +their LLM-assigned category and a rating-derived weight, adds or deletes them, +and links every interest name in the UI to the articles it matched. +Personalization is shared across accounts for now. Visitors can request an account with their preferred username at `/request-access`; admins review open requests on `/dashboard/users`. Approving @@ -249,6 +253,7 @@ Set `server.jobs_enabled = false` to make starts unavailable. | `backfill-social` | Refresh recent social scores. | | `features-prune` | Remove stale embeddings and curation telemetry. | | `import-ratings` | Fetch, embed, and rate URLs queued from the Ratings page. | +| `interests-categorize` | File uncategorized interests under categories with the bulk model. | The Ratings page accepts up to 500 historical article URLs at a time with one verdict and optional note. Imports run in the background and show per-URL @@ -366,7 +371,6 @@ prints what resolved. | `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. | | `out_dir` | `/var/lib/daily-epub/out` | Where `generate` writes artifacts before publishing. | | `profile_path` | `data/profile.md` | Hand-maintained reader profile, loaded every run. | -| `interests_opml` | `data/scour-interests.opml` | Scour interests merged with the profile interests. | | `miniflux.base_url` | `http://127.0.0.1:8082` | Miniflux root (no `/v1`). | | `miniflux.api_key` | — | **`DAILY_EPUB_MINIFLUX__API_KEY`**. Required. | | `miniflux.page_limit` | `250` | Entries per page; Miniflux caps this at 250. | @@ -409,6 +413,12 @@ prints what resolved. | `curation.recent_rejection_days` | `7` | Churn window for recent low triage/deep assessments. | | `curation.recent_rejection_floor` | `3.0` | Scores below this floor are excluded during the churn window (except auto-includes). | | `curation.ranking.*` | see below | Every weight, quota, gate and threshold of the personalized ranker. | +| `curation.ranking.affinity_floor` / `affinity_full` | `15` / `40` | Interest-attributable ratings where affinity starts and reaches full weight. | +| `curation.ranking.weights.preliminary.affinity` | `0.10` | Rating-derived interest affinity in the preliminary blend. | +| `curation.ranking.weights.preliminary.interest` | `0.30` | Interest similarity in the preliminary blend. | +| `curation.ranking.weights.preliminary.social` | `0.05` | Social signal in the preliminary blend. | +| `curation.ranking.weights.utility.affinity` | `0.05` | Rating-derived interest affinity in the utility score. | +| `curation.ranking.weights.utility.knn` | `0.10` | Rated-neighbour preference in the utility score. | | `editorial.summary_model` | `editor` | Which `[llm]` role writes the per-article summaries: `editor` (with per-article bulk fallback) or `bulk`. | | `editorial.summary_input_tokens` | `3000` | Article text offered to the summary prompt. | | `publish.epub_dir` | `/srv/bookorbit/libraries/daily-epub` | Both EPUB editions land here by atomic copy, and this is the directory the OPDS feed lists. The editions are distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. Point a BookOrbit watched folder at it if you want its UI too. **Renamed from `bookorbit_dir`**; the old key is a hard config error. | @@ -448,7 +458,9 @@ prints what resolved. gated: `knn` (rated-neighbour preference) ramps from `knn_floor` (8) to `knn_full` (25) rated articles with embeddings, `feed` (feed affinity) from `feed_floor` (15) to `feed_full` (40) attributable ratings; below the floor the -signal is absent. Ratings decay with `rating_half_life_days` (60) over +signal is absent. `affinity` (rating-derived interest affinity) likewise ramps +from `affinity_floor` (15) to `affinity_full` (40) interest-attributable ratings. +Ratings decay with `rating_half_life_days` (60) over `rating_lookback_days` (180); `neighbour_k` (5) neighbours per side and `negative_coefficient` (0.75) shape the signal. `slop_author_penalty` (0.75) is the fraction of the blend and utility removed from every candidate whose @@ -457,13 +469,13 @@ limit. `triage_max` (800), `deep_keep` (120), `shortlist_keep` (60), `assessment_reuse_days` (3), `semantic_min_words` (300), `exploration_slots` (5), `[curation.ranking.quotas]` (`triage` 60 · `interest` 20 · `knn` 20), `[curation.ranking.weights.utility]` -(`quality` 0.40 · `fit` 0.20 · `knn` 0.15 · `interest` 0.10 · `feed` 0.05 · +(`quality` 0.40 · `fit` 0.20 · `knn` 0.10 · `affinity` 0.05 · `interest` 0.10 · `feed` 0.05 · `triage` 0.05 · `social` 0.03 · `heuristic` 0.02, over the signals present for each article of the deep set) and `[curation.ranking.diversity]` (`cluster_threshold` 0.85, `per_cluster_cap` 2, `utility_protected` 10) drive the LLM triage, deep assessment, utility ranking and diversification stages. -`[curation.ranking.weights.preliminary]` (`interest` 0.35 · `knn` 0.25 · -`heuristic` 0.20 · `feed` 0.10 · `social` 0.10) blends the cheap signals; weights +`[curation.ranking.weights.preliminary]` (`interest` 0.30 · `knn` 0.25 · +`affinity` 0.10 · `heuristic` 0.20 · `feed` 0.10 · `social` 0.05) blends the cheap signals; weights are renormalized over the signals present for each article, so they need not sum to 1. `embedding_retention_days` (120) and `telemetry_retention_days` (180) are what `features prune` enforces. Validation: weights non-negative; `deep_keep ≥ @@ -923,6 +935,7 @@ images/ article images comments.rs discussion chapters encode re-encode publish.rs BookOrbit + XTC embed into the page server.rs ratings, OPDS html.rs markup helpers db.rs SQLite +interests.rs standing interests and stored article matches lock.rs one writer at a time (flock on .lock) ``` @@ -992,11 +1005,12 @@ From spec §7, plus what implementation turned up: ceiling; a protocol that is neither means another impl. Voyage AI embeddings sit behind the analogous `EmbeddingBackend` trait in `curate/embedding.rs`. - **Triage and union admission replace the heuristic gate.** Every eligible - article gets interest, rated-neighbour, feed-affinity, social and heuristic - signals, then DeepSeek reads its opening (up to `triage_max`). The deep set is + article gets interest, rated-neighbour, feed-affinity, interest-affinity, + social and heuristic signals, then DeepSeek reads its opening (up to + `triage_max`). The deep set is the union of triage, interest, neighbour, exploration, blend and auto-include retrievers. `explain` shows the assessment and `admitted_by`. Learned signals - stay absent until their gates open (8 and 15 ratings respectively). + stay absent until their gates open (8, 15, and 15 ratings respectively). - **Deep assessment and diversity are live.** DeepSeek reads a representative beginning/middle/end sample, separates editorial quality from reader fit, and records descriptive facets. Utility is normalized over the deep set; embedding diff --git a/config.example.toml b/config.example.toml index 99153b5..b9b6a0a 100644 --- a/config.example.toml +++ b/config.example.toml @@ -24,9 +24,8 @@ database_path = "/var/lib/daily-epub/daily-epub.db" # Default output directory for generated artifacts (overridden by `--out`). out_dir = "/var/lib/daily-epub/out" -# Hand-maintained reader profile and Scour interests merged into the system prompt. +# Hand-maintained reader profile loaded into the system prompt. profile_path = "data/profile.md" -interests_opml = "data/scour-interests.opml" [miniflux] base_url = "http://127.0.0.1:8082" @@ -138,7 +137,7 @@ slop_value = -1.0 # AI slop: a full negative; the author pena verdicts_in_prompt = 60 # Every weight, quota, gate and threshold of the personalized ranker. The -# learned signals (`knn`, `feed`) contribute nothing until their gates open: +# learned signals (`knn`, `feed`, `affinity`) contribute nothing until their gates open: # the weight ramps linearly from `*_floor` to `*_full` rated articles. [curation.ranking] triage_max = 800 # eligible articles the triage LLM reads @@ -153,6 +152,8 @@ knn_floor = 8 knn_full = 25 feed_floor = 15 feed_full = 40 +affinity_floor = 15 +affinity_full = 40 slop_author_penalty = 0.75 # blend and utility × 0.25 for authors with an AI slop verdict semantic_min_words = 300 exploration_slots = 5 @@ -166,16 +167,18 @@ knn = 20 # Weights need not sum to 1; they are renormalized over the present signals. [curation.ranking.weights.preliminary] -interest = 0.35 +interest = 0.30 knn = 0.25 +affinity = 0.10 heuristic = 0.20 feed = 0.10 -social = 0.10 +social = 0.05 [curation.ranking.weights.utility] quality = 0.40 fit = 0.20 -knn = 0.15 +knn = 0.10 +affinity = 0.05 interest = 0.10 feed = 0.05 triage = 0.05 diff --git a/docs/plans/2026-09-12-first-class-interests.md b/docs/plans/2026-09-12-first-class-interests.md new file mode 100644 index 0000000..78414f4 --- /dev/null +++ b/docs/plans/2026-09-12-first-class-interests.md @@ -0,0 +1,247 @@ +# First-class interests: a table, a page, categories, links, and a rating-driven weight + +**Date:** 2026-09-12 +**Repository:** `thallada/the-daily-epub` +**Status:** implementation plan, ready to execute +**Builds on:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§8 profile and interests, §9 cheap signals, §12 blend and utility), `docs/plans/2026-09-03-web-dashboard.md` (page conventions), `docs/plans/2026-09-07-feed-discovery.md` (the most recent page + job + CLI addition; copy its shapes) + +Written for a fresh implementation agent. Facts about this repo were checked against `main` (`e82f7b0`) on 2026-09-12. Nothing here depends on an external service beyond the LLM providers and Voyage that the pipeline already uses. + +--- + +## 1. Goal + +Today the reader's ~230 standing interests live in a Scour OPML export (`data/scour-interests.opml`) plus an optional `## Interests` section of `profile.md`. They are parsed on every run, embedded, matched against each day's articles (`signals::interest_matches`), grouped for the system prompt by a hand-written keyword table (`profile/themes.rs`), and — since the last commit — shown to the reader as the **Matches:** line under every article header. They cannot be added from the dashboard, ratings never touch them, and nothing links from an interest to the articles that matched it. + +This plan makes interests a first-class record: + +1. **A database table is the only source of interests.** The OPML and the profile's `## Interests` section become one-time import inputs. +2. **An Interests dashboard page** lists them with category, weight, and match counts; adds new ones; filters by category; changes a category by hand; deletes. +3. **Categories** are kept (the grouped view the operator likes) and assigned by an LLM in one daily batch, only when uncategorized interests exist, with a manual **Categorize now** button. +4. **Every interest name in the web UI is a link** to the Articles page filtered (and sorted) to the articles that matched it, with a cut-off so unrelated articles never appear. The Articles page gains an interest filter and an interests column. +5. **Ratings nudge interests.** Each explicit rating credits the interests the article matched, scaled by how strongly it matched; the resulting per-interest weight is shown on the Interests page (default sort: highest first) and feeds curation as one bounded signal, so diversity is preserved. +6. **Backfill** is two CLI commands the operator runs once. + +## 2. Verified facts + +### Interests today + +- `profile::parse_interests(opml)` reads `` names; `parse_profile_str` strips `## Interests` from `profile.md` and returns its lines; `union_interests` dedupes case-insensitively (`src/curate/profile/mod.rs:36–150`). `load_standing_interests(opml, profile)` is called from `pipeline::prepare_features` (`src/pipeline.rs:946`), `embedding::plan_backfill` (`src/curate/embedding.rs:901`), and `profile::load_or_build`/`rebuild` (through `prompt_inputs`). `config.interests_opml` is referenced in 20 places in `src/` (mostly tests that write a temp OPML), `config.example.toml:29`, README (lines 103 and 369), and `docs/runbooks/curation-v2-migration.md`. +- The system prompt's "Standing interests" section groups names with `profile::group_into_themes` (`src/curate/profile/themes.rs`): twelve keyword themes plus "Other standing interests", deterministic, sorted. The profile dashboard page renders the same grouping (`src/web/dashboard/profile.rs:249`, template `dashboard/profile.html:42`). +- `interest_embeddings` (migration 0002) is keyed by the interest **name**; `EmbeddingService::interests(&[String])` returns cached vectors and fetches misses (`src/curate/embedding.rs:743`). A new name is embedded automatically the next time it is passed. +- `signals::interest_matches(article_embeddings, interest_embeddings)` (`src/curate/signals.rs:492`) computes cosine per interest × article, z-scores each interest across the day's embedded articles (std floored at 1e-3; raw top-1 cosine fallback under 30 articles), keeps the top `RECORDED_TOP = 3` per article as `TopInterest { name, z, cos }`, and scores `0.7·z₁ + 0.3·mean(top-3 z)`. `signals.top_interests` is serialized into `candidate_runs.signals_json` (`telemetry::serialize_signals`), copied into `Pick.top_interests` as bare names by the editor (`src/curate/editor.rs:598`), and rendered as the Matches line by `chapters::understanding` → `_understanding.html` (web) and `chapter.xhtml` / `in_this_issue.xhtml` (EPUB). Prompts list interests with `z ≥ 1.5` as "matches" (`triage.rs:135`, `assess.rs:192`, `editor.rs:159`). +- Interest names also appear on: the dashboard article detail (`dashboard/article.html:61`, "Top interests" table), `_signals_table.html:5` (runs and article history), and the Feeds page "Why" badges (`dashboard/feeds.html:14`, from `discovery::why`). +- Daily volume: ~365 entries → a few hundred articles per run; 512-dim `voyage-4-lite` vectors; `embedding_retention_days = 120`, so the embedding cache holds on the order of 30–50 k vectors (≈2 KB each). Scanning the cache per web request is not an option; a per-run cosine pass is milliseconds (§9.1 of the curation plan). + +### Ratings and the learned signals + +- `rating_events` is append-only; `Db::current_ratings(lookback_days) -> Vec` returns the latest explicit verdict per article with `value` (`loved` 1.0, `good` 0.35, `not_for_me` −1.0, `slop` per `feedback.slop_value`) and `event_at` (`src/db.rs:713`). Ratings are written by `web::rate::post` (dashboard and issue pages), `rate::record_explicit` (CLI), and `imports::run`. +- `signals::PreferenceState::load` (`src/curate/signals.rs:238`) builds the run's learned state from current ratings: decayed weights `value × 0.5^(age/half_life)`, kNN over rated embeddings, and **feed/author affinity** as Beta-smoothed rates `(up+1)/(up+down+2)` with a gate `gate(n, feed_floor=15, feed_full=40)`. Every cheap signal is `Option`, percentile-normalized over the eligible set (`normalize`, `PERCENTILE_SIGNALS`), blended with renormalized weights (`preliminary_blend`; `rank::calculate_utility_for` for the deep set). Absent is never zero. The slop-author factor scales both blends. Diversity is enforced downstream by `rank::shortlist` (cluster threshold 0.85, `per_cluster_cap = 2`, `utility_protected = 10`) and by the editor prompt. +- Signal names are enumerated in: `signals::PERCENTILE_SIGNALS`, `Signals::raw`, `preliminary_blend`'s candidate list, `rank::calculate_utility_for`'s weighted list, `telemetry::serialize_signals` (raw list) and `RENDERED_SIGNALS`, `dashboard::SIGNAL_NAMES`, `config::{PreliminaryWeights, UtilityWeights}`, and `settings::SETTINGS_HELP`. A new signal touches all of them. +- The Ratings dashboard page computes each verdict's feed credit on the fly with a pure function (`ratings::contribution`) rather than storing it. Follow that precedent. + +### Dashboard, jobs, CLI, config + +- One submodule per page group under `src/web/dashboard/`, each with `routes()`, merged in `dashboard::router()`; admin gating is applied by the caller. Nav tabs are hard-coded in `src/web/templates/layout.html` (keyed on `page.active_nav`); overview tiles in `dashboard/overview.html`. POST → flash → redirect via `jobs::set_flash`. `Page::is_admin()` is available in every template. Pager partial `dashboard/_pager.html`; client-side row filter `data-table-filter`. +- The Articles list (`src/web/dashboard/articles.rs`) builds `ARTICLE_INNER` (articles ⨝ best entry ⨝ latest `candidate_runs` row ⨝ assessments, with correlated subqueries for rating and publication) and wraps it in `SELECT * FROM (…) x WHERE 1=1 {clauses} ORDER BY {sort}`. Filters are allow-listed; sorts come from `ARTICLE_SORTS`; `Pager::new(pagination, path, &filters.params())` round-trips them. +- Jobs are a fixed catalogue (`jobs::Job`, `CATALOGUE`, `parse`, `name`, `description`, `takes_lock`, `dangerous`) run as `daily-epub-job@.service` and dispatched in `main::run_job`. The profile page starts `profile-rebuild` with a plain form posting to `/dashboard/jobs/profile-rebuild`. The only daily entry point is the `daily-epub-generate.timer` (05:30 America/New_York). +- `pipeline::build_llms` (`src/pipeline.rs:1133`) builds the taste-profile prompt, the clients, and runs the weekly learned-adjustments rebuild, then rebuilds the clients with the new prompt. An LLM step that must precede the prompt goes here. +- Config sections are `#[serde(deny_unknown_fields, default)]`; an unknown top-level key makes `Config` fail to load. The settings page has a hard-coded `GROUP_ORDER`, `PATH_KEYS`, and `SETTINGS_HELP`, with a test over the section list. Migrations: `sqlx::migrate!("./migrations")`; latest is `0012_article_publication.sql`, so the new file is `0013_interests.sql`. `db.rs` has a migration test asserting a table list (`src/db.rs:1720`). + +## 3. Options considered + +### 3.1 Where "articles matching interest X" comes from + +| Option | How | Verdict | +|---|---|---| +| A. Compute on request | Load the interest vector and every cached article embedding, dot, sort. | **Rejected.** 30–50 k × 2 KB per request; the dashboard is 1–7 ms today and should stay there. | +| B. Query `candidate_runs.signals_json` with `json_each` | The top-3 names are already persisted per run. | **Rejected.** A JSON scan over every telemetry row (hundreds of thousands, pruned at 180 days) per request, no index, and rows vanish with telemetry retention. | +| C. **A junction table written by the signals stage** | `article_interests(article_id, interest_id, cos, z)`: the same top-3 the run already computes, one row each, indexed by interest. The filter is an indexed join. | **Chosen.** ~1 k rows/day, zero extra computation, and the backfill is one pass over cached embeddings. | + +### 3.2 How an interest weight is computed and stored + +| Option | How | Verdict | +|---|---|---| +| A. Stored counters updated on every rating event | Add `up/down` columns to `interests`; the rating handlers, CLI, and importer bump them; write a backfill migration script. | **Rejected.** Four write paths to keep in sync, decay cannot be stored (it is a function of *now*), and it duplicates the ratings history that already exists. | +| B. **Derived on the fly from current ratings × matches** | One pure function over `current_ratings` and their `article_interests` rows, exactly like feed affinity. The run computes it in `PreferenceState::load`; the Interests page computes it on render. | **Chosen.** No new write path, always current, decay and lookback for free, backfill is "make sure rated articles have match rows". ≤ a few hundred ratings × 3 rows: microseconds. | +| C. Ask the weekly learned-adjustments rebuild to write per-interest weights | An LLM judges the rating history per interest. | **Rejected.** Non-deterministic, weekly, and the numbers would not be explainable. The existing prose rebuild already sees the ratings. | + +### 3.3 How the weight enters curation + +| Option | How | Verdict | +|---|---|---| +| A. **A new bounded cheap signal, `affinity`** | Per article: match-strength-weighted mean of its matched interests' weights; percentile-normalized; gated on rating count; small configured weight in the preliminary blend and the utility. | **Chosen.** Fits the existing design (absent ≠ zero, renormalized weights, `explain` shows it), and its influence is capped at its weight share, so one runaway interest cannot dominate. Diversity machinery downstream is untouched. | +| B. Multiply each interest's z by its weight before the top-3 is taken | Changes which interests appear as matches. | **Rejected.** Entangles "what does this article match" with "what does he like", and the Matches line would drift with ratings. | +| C. Annotate the prompt's Standing interests with ↑/↓ | Cheap and the LLM would use it. | **Deferred** (§9). Worth adding once the weights have a few weeks of ratings behind them; it is a five-line change on top of this plan. | +| D. A new deep-set admission retriever by affinity | Like the `interest` and `knn` quotas. | **Rejected.** More slots for the same signal; the blend fill already admits high-affinity articles. | + +### 3.4 Categories + +| Option | Verdict | +|---|---| +| A. **`interests.category TEXT NULL`; the category set is the distinct values** | **Chosen.** No FK, no second page, renaming is an `UPDATE`. | +| B. A separate `interest_categories` table with FK | Rejected: a table with one meaningful column. | +| C. Keep the keyword table in `themes.rs` | Rejected as the source of truth (the user wants DB-tracked interests and LLM categorization), but **kept for the one-time import** so the current grouping survives unchanged. | + +### 3.5 When the categorizer runs + +| Option | Verdict | +|---|---| +| A. **Inside `generate`, before the prompt is built, only when uncategorized interests exist; plus a catalogue job for the button** | **Chosen.** The morning timer is the only daily trigger that exists; a run without new interests spends nothing. | +| B. Its own systemd timer | Rejected: another unit to install for a call that takes seconds. | +| C. Synchronously in the Add handler | Rejected: an LLM call in a request path, and the user asked for a daily batch. | + +## 4. Design decisions (settled) + +| Topic | Decision | +|---|---| +| Match rule (one definition everywhere) | An interest **matches** an article when it is among the article's top three interests by z **and** `z ≥ MATCH_MIN_Z = 1.0`. `interest_matches` applies this when it truncates, so `signals.top_interests`, the Matches line, the stored rows, the Articles filter, and the interest weights all agree. The score formula is unchanged (computed before the cut). The prompts keep their stricter `z ≥ 1.5` for "matches interests". | +| Stored rows | `article_interests(article_id, interest_id, cos, z, run_id)`, upserted per eligible article per run (the same article can be eligible on consecutive days; the latest run wins; cosine is stable, z is that day's). Rows are never pruned (≈40 bytes each, ~1 k/day). | +| Link target and filter key | `/dashboard/articles?interest=`: names are unique (case-insensitive), human-readable, and `Pick.top_interests` already carries names, so no id has to travel through `issue_json`. The handler resolves the name to an id; an unknown name yields an empty list, never an error. When `interest` is set and `sort` is absent, the sort defaults to `match` (cosine descending). | +| Who sees links | The dashboard is admin-only, so the Matches line links only for admin viewers (`page.is_admin()`); readers and anonymous visitors see plain text as today. The EPUB never links. | +| Interest weight | Beta-smoothed rate over decayed, strength-scaled credits (§5.2): `(up + 1) / (up + down + 2)`, in (0, 1), 0.5 = no information. Shown with two decimals plus `up`, `down`, and the number of rated matches. | +| Match strength | `s = clamp(z / 3, 0, 1)`: a z of 3 credits the full rating, a bare match (z = 1) a third. | +| Curation signal | `affinity` (§5.3): signed, centred on zero, gated on the number of ratings that credited at least one interest (`affinity_floor = 15`, `affinity_full = 40`, same shape as the feed gate). Default weights: preliminary 0.10 (taken from `interest` 0.35→0.30 and `social` 0.10→0.05), utility 0.05 (taken from `knn` 0.15→0.10). | +| Diversity | Guaranteed by construction: the signal's share of the blend is its configured weight (≤ 10 % / 5 %), percentile normalization caps a favourite interest's articles at percentile 1.0 of that one signal, Beta smoothing means a single *loved* moves a weight from 0.50 to at most 0.67, ratings decay with the 60-day half-life, and `rank::shortlist`'s per-cluster cap and the editor's diversity instructions are unchanged. No new cap is needed. | +| Source of truth | The `interests` table. `interests_opml` is **removed** from config; `profile.md`'s `## Interests` section is still stripped by the parser but no longer read (the importer consumes it once; the profile page says so). | +| Prompt grouping | "Standing interests" groups by `interests.category` (sorted by category name, members sorted case-insensitively); `NULL` renders under the existing label "Other standing interests". `themes.rs` is used only by the importer. | +| Categorizer | One JSON call on the bulk provider over every uncategorized interest, given the existing category list; may create a category only when none fits. Unassigned names stay `NULL` and are retried the next day. Also the catalogue job `interests-categorize` behind the **Categorize now** button. | +| Interests page actions | Add (name + optional category), change category (per-row select + Save), Delete (confirm; cascades match rows, deletes the cached embedding). No rename (delete + add). | +| Backfill | `daily-epub interests import` (OPML + profile section → rows, categorized by the keyword table) and `daily-epub interests backfill` (match rows for every cached embedding). Weights need no backfill: they are derived. | +| Config | No new section. `[curation.ranking]` gains `affinity_floor`, `affinity_full`; the two weight tables gain `affinity`. | + +## 5. The numbers + +### 5.1 Match rows + +In `prepare_features` (`src/pipeline.rs`), after `signals::compute_all` and before the candidates are handed on: for every candidate with non-empty `signals.top_interests`, upsert `(article_id, interest_id, cos, z, run_id)`. One transaction, `INSERT … ON CONFLICT(article_id, interest_id) DO UPDATE SET cos, z, run_id`. Names map to ids through the `interests` rows loaded at the top of the stage (the same rows whose names go to `service.interests`). Best effort: a failure is a report warning, never a failed run. + +### 5.2 Interest weight (pure, `interests::rates`) + +Inputs: `current_ratings(rating_lookback_days)` and the `article_interests` rows of those articles. + +```text +for each current rating r on article a (value v_r, decay d_r = 0.5^(age_days / half_life_days)) + for each match row (a, i, z): + s = clamp(z / 3, 0, 1) + credit = v_r × d_r × s + up_i += max(credit, 0) + down_i += max(−credit, 0) + n_i += 1 +weight_i = (up_i + 1) / (up_i + down_i + 2) # 0.5 when n_i = 0 +``` + +`cleared` verdicts are already excluded by `current_ratings`; `slop` carries `feedback.slop_value` like everywhere else. A rating is *attributable to interests* when it credits at least one interest; that count drives the gate. + +### 5.3 The `affinity` signal (pure, in `PreferenceState`) + +```text +matched = the article's top_interests with n_i > 0 +affinity = Σ s_i × (weight_i − 0.5) / Σ s_i absent when matched is empty or the gate is 0 +gate = gate(attributable_interest_ratings, affinity_floor, affinity_full) +``` + +The raw value lives in [−0.5, 0.5]; it is percentile-normalized with the other cheap signals, weighted by `weights.preliminary.affinity × gate` in the blend and `weights.utility.affinity × gate` in the utility. `explain` and `_signals_table.html` show it like any other signal. The once-per-run preference log line gains `affinity gate … (n=…)`. + +## 6. Data model — `migrations/0013_interests.sql` + +```sql +CREATE TABLE interests ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL COLLATE NOCASE UNIQUE, + category TEXT, -- NULL until categorized + created_at TEXT NOT NULL, + categorized_at TEXT +); +CREATE INDEX idx_interests_category ON interests(category); + +CREATE TABLE article_interests ( + article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE, + interest_id INTEGER NOT NULL REFERENCES interests(id) ON DELETE CASCADE, + cos REAL NOT NULL, + z REAL NOT NULL, + run_id INTEGER, -- NULL for backfilled rows + PRIMARY KEY (article_id, interest_id) +); +CREATE INDEX idx_article_interests_interest ON article_interests(interest_id, cos DESC); +``` + +`interest_embeddings` stays keyed by name (no migration): a deleted interest also deletes its embedding row; a renamed one is a new interest. + +## 7. Implementation steps + +Sizes are rough line counts including tests. Steps 1–3 must be in order; 4–8 can proceed in parallel after 3; 9 last. + +1. **Migration + `src/interests.rs`** (~350). The table above; the db-list test in `db.rs` gains both tables. Module (following `discovery.rs`: data access, pure functions, and the LLM step in one file): + - `Interest { id, name, category, created_at, categorized_at }`; `list(db) -> Vec` (ordered by name), `add(db, name, category, now) -> Result` (trimmed, 1–80 chars, unique case-insensitive), `set_category(db, id, Option, now)`, `delete(db, id)` (also `DELETE FROM interest_embeddings WHERE interest = name`), `names(db) -> Vec`, `grouped(db) -> Vec<(String, Vec)>` (for the prompt), `uncategorized(db)`. + - `replace_matches(db, run_id, &[(ArticleId, &[TopInterest])], names→ids)`; `matches_for_articles(db, ids) -> Vec`; `match_counts(db) -> HashMap`. + - `pub fn rates(ratings: &[(ArticleId, value, decay)], rows: &[(ArticleId, interest_id, z)]) -> (HashMap, attributable)` per §5.2, and `Rate::weight()`. + - Tests: uniqueness is case-insensitive; `rates` matches hand-checked numbers (a loved z=3 → up 1.0; a good z=1.5 → up 0.175; a not-for-me z=0.9 → nothing, below the cut); delete cascades; `grouped` puts `NULL` last under "Other standing interests". + +2. **Signals** (~250). In `signals.rs`: `MATCH_MIN_Z = 1.0` applied inside `interest_matches` after truncation; `Signals.affinity: Option`; `"affinity"` in `PERCENTILE_SIGNALS`, `Signals::raw`, `preliminary_blend` (gate `affinity_gate`); `PreferenceState` gains `interest_rates: HashMap` (keyed by name — `TopInterest` carries names), `affinity_gate`, `attributable_interest_ratings`, `fn affinity(&self, top: &[TopInterest]) -> Option`, loaded in `load` from `interests::matches_for_articles(rated ids)` and exposed on `PreferenceSummary` and in `log`. `rank::calculate_utility_for` adds `("affinity", configured.affinity, candidate.signals.affinity_gate)`. `telemetry`: `serialize_signals` raw list and `RENDERED_SIGNALS`; `render_explain` prints it with the others. `dashboard::SIGNAL_NAMES` and `_signals_table.html` pick it up automatically once the name is in the list. Config: `RankingConfig { affinity_floor: 15, affinity_full: 40 }`, `PreliminaryWeights { affinity: 0.10, interest: 0.30, social: 0.05 }`, `UtilityWeights { affinity: 0.05, knn: 0.10 }`, `SETTINGS_HELP` lines, `config.example.toml`, README table. + Tests: the top-3 cut drops a z=0.4 third interest; affinity is absent under the gate and with no rated interests; a candidate matching a 0.8-weight interest outranks an otherwise identical one matching a 0.3-weight interest in the blend; blend weights still renormalize to 1. + +3. **Replace the OPML/profile plumbing** (~300, mostly deletions and test edits). `profile::load_standing_interests`, `parse_interests`, `union_interests`, and `prompt_inputs`' OPML argument go; `build`, `load_or_build`, `rebuild`, and `weekly_rebuild_if_due` take `grouped: Vec<(String, Vec)>` from `interests::grouped(db)` instead of `interests: &[String]` + `group_into_themes`. `pipeline::prepare_features` and `embedding::plan_backfill` take names from `interests::names(db)`. `themes.rs` stays but is only referenced by the importer (step 8); `pub use themes::group_into_themes` is removed from the profile module. Remove `Config.interests_opml` (struct, `Default`, `config check` line, the `deny_unknown_fields` implication is a rollout note in §10), `settings::PATH_KEYS`/`SETTINGS_HELP`, `config.example.toml`, README (lines 103, 369), and the runbook mention. Every test that writes a temp OPML instead inserts rows with `interests::add`. The profile page loses the OPML card and the "Extracted `## Interests` lines" preview; in their place one line: "N standing interests in M categories — manage them on the Interests page", and the editor note says the `## Interests` section is ignored. + +4. **Match rows in the run** (~80). `prepare_features` writes them per §5.1 (timing folded into the existing `signals` timing; count `interest_matches` on the report counts is optional — skip unless free). + +5. **Articles page** (~200). `ArticlesQuery.interest: Option` → `ArticleFilters.interest: Option<(i64, String)>` resolved by name (case-insensitive) in `from_query`'s caller (it needs the db; resolve in `list` before building filters, or make `from_query` async — pick the former). When set: `ARTICLE_INNER` gains `JOIN article_interests ai ON ai.article_id = a.id AND ai.interest_id = ?` (inside the inner query so `idx_article_interests_interest` drives it; the join is a `{interest_join}` placeholder that is empty otherwise) and exposes `ai.cos AS match_cos`; `ARTICLE_SORTS` gains `("match", "x.match_cos DESC, x.id DESC")`, which is the default when `interest` is set and `sort` is absent, and is ignored (falls back to `first_seen`) when it is not. Filter UI: a ` + @@ -18,11 +19,12 @@ {% include "dashboard/_pager.html" %} {% if articles.len() > 1 %}{% endif %}
- +{% for article in articles %} + @@ -32,6 +34,6 @@ -{% endfor %}{% if articles.is_empty() %}{% endif %}
first seentitlefeedwordslast stagereasonutilitytriagequalityfitratingpublished
first seentitlefeedinterestswordslast stagereasonutilitytriagequalityfitratingpublished
{{ article.first_seen }} {{ article.title }} {% if let Some(feed_id) = article.feed_id %}{{ article.feed }}{% else %}{{ article.feed }}{% endif %}{% for interest in article.interests %}{{ interest.name }}{% if !loop.last %} {% endif %}{% endfor %} {{ article.words }} {% if let Some(stage) = article.stage %}{{ stage }}{% if let Some(run_id) = article.run_id %} {% if let Some(date) = article.run_date %}{{ date }}{% else %}run {{ run_id }}{% endif %}{% endif %}{% else %}never considered{% endif %} {% if let Some(reason) = article.reason %}{{ reason }}{% endif %}{{ article.fit }} {% if let Some(rating) = article.rating %}{{ rating }}{% endif %} {% if let Some(date) = article.published %}{{ date }}{% endif %}
No articles match this filter.
+{% endfor %}{% if articles.is_empty() %}No articles match this filter.{% endif %} {% include "dashboard/_pager.html" %} {% endblock %} diff --git a/src/web/templates/dashboard/feeds.html b/src/web/templates/dashboard/feeds.html index 905ac01..4255ab1 100644 --- a/src/web/templates/dashboard/feeds.html +++ b/src/web/templates/dashboard/feeds.html @@ -11,7 +11,7 @@ {% for row in rows %} {{ row.score }} {{ row.label }}{{ row.host }} -{% for interest in row.interests %}{{ interest }} {% endfor %}{% for article in row.articles %}
{{ article.title }}
{% endfor %} +{% for interest in row.interests %}{{ interest.name }} {% endfor %}{% for article in row.articles %}
{{ article.title }}
{% endfor %} {{ row.article_count }} {{ row.seen }} {% if status == "candidate" %}
{% else %}{{ row.decided_at }}{% if let Some(href) = row.miniflux_href %} · in Miniflux{% endif %}{% endif %} diff --git a/src/web/templates/dashboard/interests.html b/src/web/templates/dashboard/interests.html new file mode 100644 index 0000000..806cd86 --- /dev/null +++ b/src/web/templates/dashboard/interests.html @@ -0,0 +1,33 @@ +{% extends "layout.html" %}{% block content %}
+
+

Interests

+

{{ total }} standing interests. Weights use the last {{ lookback_days }} days of ratings with a {{ half_life_days }}-day half-life · affinity gate {{ affinity_gate }} · {{ attributable }} of {{ affinity_full }} attributable ratings.

+
{{ uncategorized }} uncategorized
{% if !jobs_enabled %} Jobs are disabled on this server (server.jobs_enabled = false); run daily-epub job run interests-categorize instead.{% endif %}
+ +

Add an interest

+ + +
+
+ +
+ + +
Reset
+
+{% if rows.len() > 1 %}{% endif %} +
+ +{% for row in rows %} + + + + + + + + + +{% endfor %}{% if rows.is_empty() %}{% endif %} +
interestcategoryweightupdownrated matchesmatched articlesaddedactions
{{ row.name }}{% if let Some(category) = row.category %}{{ category }}{% else %}uncategorized{% endif %}{{ row.weight }}{{ row.up }}{{ row.down }}{{ row.rated_matches }}{{ row.matched_articles }}{{ row.added }}
No interests match this filter.
+
{% endblock %} diff --git a/src/web/templates/dashboard/overview.html b/src/web/templates/dashboard/overview.html index e1e3e1b..0488cbc 100644 --- a/src/web/templates/dashboard/overview.html +++ b/src/web/templates/dashboard/overview.html @@ -11,6 +11,7 @@
Unrated picks{{ unrated.len() }}from the last three issues
Active jobs{{ active_jobs.len() }}all jobs
Feed candidates{{ feed_candidates }}review feeds
+
Interests{{ interests_total }}{{ uncategorized_interests }} uncategorized
Access requests{{ access_requests }}review requests
diff --git a/src/web/templates/dashboard/profile.html b/src/web/templates/dashboard/profile.html index a2d7535..d57660c 100644 --- a/src/web/templates/dashboard/profile.html +++ b/src/web/templates/dashboard/profile.html @@ -1,13 +1,13 @@ {% extends "layout.html" %}{% block content %}

Profile

-

The standing taste file the curator reads before every run: what you like, what the OPML declares, and what the editor model has learned from your verdicts.

+

The standing taste file the curator reads before every run, alongside stored interests and what the editor model has learned from your verdicts.

RatingsFeedback settings

profile.md

-

{{ path }}{% if !exists %} — missing; saving creates it{% else %} · {{ bytes }} bytes{% endif %} · limit {{ max_bytes }} bytes. Any ## Interests section is parsed one interest per line; everything else goes into the system prompt verbatim.

+

{{ path }}{% if !exists %} — missing; saving creates it{% else %} · {{ bytes }} bytes{% endif %} · limit {{ max_bytes }} bytes. Any ## Interests section is ignored; everything else goes into the system prompt verbatim.

The next run rebuilds the system prompt from the saved file.
@@ -19,8 +19,6 @@

A live read of the text on the left, exactly as curate::profile splits it.

Passthrough sections

{% if preview_body.trim().is_empty() %}

Nothing passes through — the file is empty or only has an Interests section.

{% else %}
{{ preview_body }}
{% endif %} -

Extracted ## Interests lines

-{% if preview_interests.is_empty() %}

None. The prompt uses the OPML interests alone.

{% else %}
    {% for interest in preview_interests %}
  • {{ interest }}
    • {% endfor %}
{% endif %}
@@ -40,9 +38,8 @@ {% endif %}

Standing interests

-

{{ opml_path }} · {{ opml_count }} interests, grouped the way the system prompt lists them. The union of these and the ## Interests lines above is what the prompt uses; edit the OPML file to change them.

-{% if !opml_error.is_empty() %}

{{ opml_error }}

{% endif %} -{% if !themes.is_empty() %}
{% for theme in themes %}
{{ theme.name }} ({{ theme.count }})
{{ theme.members }}
{% endfor %}
{% endif %} +

{{ interest_count }} standing interests in {{ category_count }} categories — manage them on the Interests page.

+{% if !categories.is_empty() %}
{% for category in categories %}
{{ category.name }} ({{ category.count }})
{{ category.members }}
{% endfor %}
{% endif %}

Learned adjustments

Rebuilt weekly from ratings by the editor model (prompt version {{ prompt_version }}, built {{ prompt_built_at }}, {{ learned_age }}). {% if rebuild_due %}A rebuild is due — the next run performs it, or start it now.{% else %}The next scheduled rebuild is at least {{ rebuild_interval_days }} days after the last one; the next run performs it when due.{% endif %}

diff --git a/src/web/templates/layout.html b/src/web/templates/layout.html index 5d629ca..83abf07 100644 --- a/src/web/templates/layout.html +++ b/src/web/templates/layout.html @@ -47,6 +47,7 @@ Runs Articles Ratings + Interests Feeds Profile Stats diff --git a/tests/config_check.rs b/tests/config_check.rs index 57a4b3e..172ae0f 100644 --- a/tests/config_check.rs +++ b/tests/config_check.rs @@ -35,7 +35,6 @@ fn config_check_prints_the_facts_and_exits_zero_without_keys() { "config: ", "database_path: /var/lib/daily-epub/daily-epub.db", "profile_path: data/profile.md", - "interests_opml: data/scour-interests.opml", "llm.bulk: deepseek · openai · deepseek-v4-flash", "key MISSING (set DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY)", "llm.editor: anthropic · anthropic · claude-opus-5 · effort high · max_daily_usd $3.00", @@ -66,4 +65,8 @@ fn config_check_exits_non_zero_on_an_invalid_config() { let (code, _, stderr) = run("[deepseek]\nmodel = \"x\"\n"); assert_ne!(code, 0); assert!(stderr.contains("[providers.deepseek]"), "{stderr}"); + + let (code, _, stderr) = run("interests_opml = \"data/scour-interests.opml\"\n"); + assert_ne!(code, 0); + assert!(stderr.contains("interests_opml"), "{stderr}"); } diff --git a/tests/m3_curation.rs b/tests/m3_curation.rs index 96212a3..f18c909 100644 --- a/tests/m3_curation.rs +++ b/tests/m3_curation.rs @@ -21,6 +21,7 @@ use daily_epub::curate::assess::parse_deep_response; use daily_epub::curate::editor::parse_selection_response; use daily_epub::curate::editorial::BriefResponse; use daily_epub::curate::profile; +use daily_epub::interests; fn repo(rel: &str) -> PathBuf { Path::new(env!("CARGO_MANIFEST_DIR")).join(rel) @@ -183,13 +184,14 @@ fn stage_c_fixture_parses_into_the_brief() { assert!(value.get("section_intros").is_none()); } -/// The taste profile is seeded from this file; a broken export would silently -/// gut the system prompt (§3.6a). +/// The importer relies on this file; a broken export would silently lose +/// standing interests (§3.6a). #[test] fn scour_opml_still_yields_the_interest_list() { - let interests = profile::parse_interests(&repo("data/scour-interests.opml")) - .expect("the shipped OPML must parse"); - let unique: BTreeSet = interests.iter().map(|n| n.to_lowercase()).collect(); + let raw = std::fs::read_to_string(repo("data/scour-interests.opml")) + .expect("the shipped OPML must be readable"); + let names = interests::parse_opml(&raw); + let unique: BTreeSet = names.iter().map(|n| n.to_lowercase()).collect(); assert!( unique.len() > 180, @@ -206,7 +208,7 @@ fn scour_opml_still_yields_the_interest_list() { assert!(unique.contains(expected), "{expected} disappeared"); } assert!( - !interests.iter().any(|n| n.contains("token=")), + !names.iter().any(|n| n.contains("token=")), "interest names must not leak the Scour token" ); @@ -217,9 +219,10 @@ fn scour_opml_still_yields_the_interest_list() { "/data/profile.md" ))) .expect("profile file"); + let grouped = vec![("Imported interests".to_string(), names)]; let document = profile::build( &profile_file.body, - &interests, + &grouped, profile::NO_LEARNED_ADJUSTMENTS, &[], 60,