From c971fd931713db188587fb67e4b44b7994571a20 Mon Sep 17 00:00:00 2001 From: Tyler Hallada Date: Mon, 7 Sep 2026 04:17:29 +0000 Subject: [PATCH] Register the [discovery] section in settings and document it Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4 --- README.md | 12 ++ config.example.toml | 17 +++ docs/plans/2026-09-07-feed-discovery.md | 157 ++++++++++++++++++++++++ src/web/dashboard/settings.rs | 5 + 4 files changed, 191 insertions(+) create mode 100644 docs/plans/2026-09-07-feed-discovery.md diff --git a/README.md b/README.md index 1ed4f14..babb28f 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,7 @@ daily-epub stats [--days 14] # the evaluation framework, one fact per line daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes] daily-epub features prune # stale embeddings, old telemetry and assessments daily-epub backfill-social [--days 7] # re-poll social scores for recent articles +daily-epub feeds discover [--days 14] [--limit 50] # seed feed candidates from recent aggregator articles daily-epub db migrate # run migrations (also automatic on every start) daily-epub config check # validate the config, print the resolved roles, keys and paths daily-epub users add USER [--admin] [--password-stdin] @@ -245,6 +246,14 @@ verdict and optional note. Imports run in the background and show per-URL status in the page plus live logs on the job page. If dashboard jobs are disabled, queueing still works; run `daily-epub job run import-ratings` by hand. +The Feeds page lists feeds discovered behind articles that reached the paper +only through an aggregator (Hacker News, Lobsters, Reddit, Scour), ranked by +the same per-article signals the paper selects on, so the strongest leads sit +at the top. Each row offers **Add**, which subscribes the feed in a chosen +Miniflux category, and **Dismiss**. Discovery runs as a best-effort stage +during `generate`; `daily-epub feeds discover --days 14 --limit 50` seeds the +page from articles already in the database. + The Settings page derives its fields from `Config`, rewrites `config.toml` in place with `toml_edit`, preserves comments/order and file permissions, validates before an atomic rename, and records attributed history. It re-reads hand edits @@ -407,6 +416,9 @@ prints what resolved. | `mail.smtp_pass` | — | **`DAILY_EPUB_MAIL__SMTP_PASS`**, environment only. | | `mail.from` | `""` | Sender mailbox, either a bare address or `Name
`. | | `mail.notify_to` | unset | Recipient for access-request notifications. | +| `discovery.enabled` | `true` | Run the feed discovery stage during `generate`. | +| `discovery.max_lookups_per_run` | `30` | Hosts one run may look up in Miniflux. Each host is re-checked at most every 90 days. | +| `discovery.skip_hosts` | aggregators, code hosts, social networks | Hosts never looked up. A host matches itself or any subdomain of it. | | `xtc.enabled` | `true` | Set `false` to skip the converter entirely. | | `xtc.command` | `node` | Converter executable. | | `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends ` -o -f ` (plus `-c `). | diff --git a/config.example.toml b/config.example.toml index 74b1ccf..4ea0aa2 100644 --- a/config.example.toml +++ b/config.example.toml @@ -237,3 +237,20 @@ smtp_user = "" # smtp_pass: environment only (DAILY_EPUB_MAIL__SMTP_PASS) from = "" # e.g. The Daily EPUB # notify_to = "operator@example.com" # recipient for access-request notifications + +[discovery] +enabled = true # run the feed discovery stage during generate +max_lookups_per_run = 30 # hosts looked up in Miniflux per run; each host is re-checked at most every 90 days +skip_hosts = [ # never looked up: the host itself or any subdomain of it + "news.ycombinator.com", + "lobste.rs", + "reddit.com", + "github.com", + "gist.github.com", + "x.com", + "twitter.com", + "youtube.com", + "en.wikipedia.org", + "arxiv.org", + "docs.google.com", +] diff --git a/docs/plans/2026-09-07-feed-discovery.md b/docs/plans/2026-09-07-feed-discovery.md new file mode 100644 index 0000000..e7baee9 --- /dev/null +++ b/docs/plans/2026-09-07-feed-discovery.md @@ -0,0 +1,157 @@ +# Feed discovery: propose new Miniflux subscriptions from aggregator hits + +**Date:** 2026-09-07 +**Repository:** `thallada/the-daily-epub` +**Status:** small implementation plan, ready to execute +**Builds on:** `docs/plans/2026-09-03-web-dashboard.md` (the admin dashboard this adds a page to), `docs/plans/2026-09-02-personalized-curation-v2.md` (the per-article signals this reuses) + +Written for a fresh implementation agent. Facts about this repo were checked against `main` (`e69940f`) on 2026-09-07. Facts about the Miniflux API were checked against `https://miniflux.app/docs/api.html` the same day; anything marked *assumption* must be re-checked against the live instance before relying on it. + +--- + +## 1. Goal + +Most of the paper's articles arrive through aggregators (Hacker News, Lobsters, Reddit, Scour), not through the author's own feed. Every such article is a lead on a feed the operator does not subscribe to yet. This feature: + +1. During `generate`, after articles are persisted, asks Miniflux to discover the feed(s) behind each aggregator-only article and records the ones the operator is not already subscribed to as **feed candidates**. +2. Adds an admin dashboard page, **Feeds** (`/dashboard/feeds`), listing candidates ranked by how likely the operator is to enjoy the feed, computed **algorithmically from telemetry the pipeline already persists** (utility / preliminary blend, stage reached, explicit ratings). No LLM calls, no new embeddings. +3. Each row has **Add** (with a Miniflux category picker; calls `POST /v1/feeds`) and **Dismiss**. + +Over time the subscribed set grows and the pool of direct-feed articles each issue draws from grows with it. + +Out of scope (deliberately, see §7): auto-subscribing, feed previews/health checks, Scour integration, OPML, per-feed LLM judgement, un-dismissing from the UI. + +## 2. Verified facts + +### About this repo + +- **Pipeline order** (`src/pipeline.rs:354` `run_stages`): ingest → dedupe → extraction → `persist_articles` (stage 4, ~line 443, mints real article ids) → social enrichment (stage 5, best effort, ~line 448) → hygiene/embeddings/signals (stage 6) → … Failure policy is documented at the top of the file: *best effort* stages log, push a warning onto `report` (status `degraded`) and continue. The new stage is best effort. +- **How an article arrived** is stored per article in `articles.sources_json` as `Vec` (`src/types.rs:64`), each with a `SourceKind` (`Scour | HnFrontpage | Lobsters | Reddit | Feed`, `src/types.rs:49`). Classification is `dedupe::classify_source_with_feed` (`src/dedupe.rs:177`). `Article::came_via(kind)` exists (`src/types.rs:147`). "Aggregator-only" therefore means `!article.sources.iter().any(|s| s.kind == SourceKind::Feed)`. +- **Ingest already loads the whole subscription list once per run**: `MinifluxClient::ingest_window` returns `(Vec, HashMap)` where `FeedMeta` has `feed_url` and `site_url` (`src/miniflux.rs:105`). That map is the "already subscribed" set; no extra call needed in the pipeline. +- **Miniflux client** (`src/miniflux.rs`) is currently GET-only: one private `get_json` helper with retry (`RetryPolicy`, retry on network/5xx/429), `X-Auth-Token` auth, `MinifluxError::{Http, Status, Decode, MissingApiKey}`. Module doc says "Reads only … read state is never mutated" — still true for the pipeline; the dashboard **Add** action is the only writer and only creates feeds. +- **Per-article ranking evidence is persisted** in `candidate_runs` (`migrations/0002_curation_v2.sql:59`): `(run_id, article_id) PK, stage, excluded_reason, signals_json, utility REAL, rank_utility, …`. `signals_json` decodes to `curate::telemetry::SignalsJson` (`src/curate/telemetry.rs:114`) whose `blend()` returns the preliminary blend on 0–100 and whose `top_interests: Vec` names the matched interests. `utility` (0–100) is set only for articles that reached the deep set (`admitted`/`assessed`/`shortlisted`/`selected`). Stage names: `dashboard::STAGES` (`src/web/dashboard/mod.rs`). `runs.status` distinguishes finished runs; dry runs are not flagged in `runs` (a dry run still writes telemetry, which is fine to use). +- **Ratings**: `Db::current_ratings(lookback_days) -> Vec` (`src/db.rs:708`) returns one current verdict per article with `value` (`loved` 1.0, `good` 0.35, `not_for_me` −1.0) — the same numbers `signals::RatedExample` uses. +- **Dashboard conventions**: one submodule per page group under `src/web/dashboard/`, each exposing `routes() -> Router` merged in `dashboard::router()` (`src/web/dashboard/mod.rs:41`). Admin gating is applied by the caller. POST-then-redirect with a session flash is the established pattern (`users::approve`, `src/web/dashboard/users.rs:122`, uses `set_flash(&session, "error"|"ok", text)` then `Redirect::to`). Nav tabs are hardcoded in `src/web/templates/layout.html:47–54` keyed on `page.active_nav`. Overview tiles live in `src/web/templates/dashboard/overview.html`. Templates are askama; shared pager is `dashboard/_pager.html`. +- **`AppState`** (`src/server.rs:79`) holds `db`, `config: Arc>>` (read via `state.config()`), `web: Arc`, `mailer`. Handlers construct outbound clients from config as needed (`bookorbit.rs` does this for OPDS); reuse whatever shared `reqwest::Client` `WebState` already carries, otherwise `http::build_client(http::DEFAULT_TIMEOUT)`. +- **Config**: sections are `#[serde(deny_unknown_fields, default)]` structs on `Config` with `Default` impls; env override is `DAILY_EPUB_
__`. The settings dashboard has a **hardcoded section list**, help text and a test asserting the list (`src/web/dashboard/settings.rs`, see the BookOrbit plan §2 for the exact spots) — a new section must be added there or the settings page/test breaks. `config.example.toml` and the README config table document every key. +- **Migrations**: `sqlx::migrate!("./migrations")`; latest is `0009_request_username.sql`, so the new file is `0010_feed_discovery.sql`. +- **Jobs / CLI**: catalogue jobs are in `src/jobs.rs` and run through `main.rs` (`Job::ImportRatings => imports::run(...)`). A one-off backfill only needs a CLI subcommand, not a job (§4 step 6). + +### About Miniflux (docs, 2026-09-07) + +- **Discover**: `POST /v1/discover` with body `{"url": ""}` → `[{"url": "", "title": "…", "type": "rss|atom|json"}]`. Optional body fields (`username`, `password`, `user_agent`, `fetch_via_proxy`) are not needed. **Observed live on 2026-09-07** (operator ran it against several hosts): + - When the page has no ``, Miniflux falls back to probing well-known paths on the site root (`/atom.xml`, `/feed.atom`, `/feed.xml`, `/feed/`, `/index.rss`, `/index.xml`, `/rss.xml`, `/rss/`, `/rss/feed.xml`) and **returns every path that answered 200 without checking that the body is a feed**. A site that serves its SPA shell for any path (zombo.com, immich, BookOrbit) yields up to nine bogus results, each with `title == url`. **Discover results are leads, not facts; every one must be validated before it is stored (§3, §4 step 3).** + - When the target site fails, the response is a 4xx/5xx with `{"error_message": "fetcher: bad gateway (502 status code)"}`, `"fetcher: unexpected status code (412 status code)"`, or `"resource not found"` (nothing found / 404). Treat all of these as "host checked, zero candidates" and move on; never retry within the run. + - Real `` hits work from an **article** URL. Observed for `https://blog.philz.dev/blog/language-server-db/`: + `[{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.xml","type":"atom"},{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.json","type":"json"}]`. + The same content is offered twice (Atom and JSON Feed) under one title, so link-tag hits need deduping by title too (§3). Miniflux substitutes the URL for the title when the tag has none, so `title == url` marks a result as a well-known-path guess in practice. +- **Create feed duplicate**: observed `HTTP 400` with `{"error_message":"This feed already exists."}`. The flash shows `error_message` verbatim; on that exact message the row is also flipped to `added` (without a `miniflux_feed_id`), since the operator subscribed by other means. +- **Create feed**: `POST /v1/feeds` with `{"feed_url": "…", "category_id": }` → `{"feed_id": }`. Errors come back as 4xx/5xx with a JSON `error_message`; a duplicate subscription is a 4xx (*assumption:* 400 with "This feed already exists"). Surface `error_message` in the flash rather than special-casing. +- **Categories**: `GET /v1/categories` → `[{"id", "user_id", "title", "hide_globally"}]`. +- **Feeds**: `GET /v1/feeds` elements carry `feed_url` and `site_url` (already modelled as `MinifluxFeed`). +- API keys are full-access; the existing key can create feeds. Nothing to provision. + +## 3. Design decisions (settled) + +| Topic | Decision | Why | +|---|---|---| +| Where discovery runs | A best-effort **pipeline stage right after social enrichment** (needs real article ids, before hygiene). Plus a CLI backfill (`daily-epub feeds discover --days N`) that runs the same function over recent `articles`, for seeding. | The user asked for "a step in article processing". The backfill is the same function behind a flag; without it the page is empty until tomorrow. | +| Which articles | Aggregator-only articles: no `SourceKind::Feed` in `sources`. Skip when the canonical URL's host is in `discovery.skip_hosts` or matches the host of any subscribed feed's `site_url`/`feed_url`. | Articles that already came through a direct feed prove nothing. Host match against subscriptions catches "same site, different feed URL" cheaply. | +| Discovery mechanism | **Miniflux `POST /v1/discover`** with the article URL, followed by **our own validation fetch** of each returned URL. No new crate. | Miniflux already does `` + well-known paths + YouTube etc. and runs on the same host, but its well-known-path guesses are unverified (§2). Validating is ~40 lines; replacing discover with a home-built finder is far more. | +| Validation | `GET` each discovered URL with the shared client (10 s timeout, read at most 256 KB). Accept only if the body, after an optional BOM, whitespace and `` declaration, starts with `` from that body (first `…` in the buffer, unescaped, trimmed) as the candidate title when Miniflux gave `title == url`. Any other body, a non-2xx status or an error rejects the URL. | Content-type alone is unreliable (SPAs return `text/html` for everything, some real feeds return `text/xml` or `application/octet-stream`). Sniffing the root element is cheap and decisive. | +| Duplicate results | After validation: group link-tag hits (`title != url`) by `title`, keeping one per group and preferring `type` `rss`/`atom` over `json`; then at most **one** `title == url` guess (the first in Miniflux's order); at most 3 per host overall. | blog.philz.dev offers the same feed as Atom and JSON Feed under one title; WordPress answers `/feed/` and `/rss/`, Hugo `/index.xml` and `/feed.xml`, all with the same content. One is enough. Real tag/category feeds arrive with distinct link-tag titles and are kept. | +| Per-host memo | Table `feed_discovery_hosts(host PK, checked_at, candidates)`. A host is looked up **once**; re-checked only after `RECHECK_DAYS = 90` (const). New articles on an already-checked host are just **linked** to that host's existing candidates (no network). | Keeps per-run lookups bounded and lets a candidate accumulate evidence for free. | +| Per-run cap | `discovery.max_lookups_per_run` (default 30), concurrency 4, 10 s timeout per lookup. Order: articles with the highest social score first. | A run already takes ~12 min; discovery must not add more than ~1 min. | +| Multiple feeds per host | Drop results whose URL or title contains `comment` before validating. Each kept result is one candidate keyed by canonicalized `feed_url` (unique). | Comment feeds are noise; tag/category feeds are a real choice the operator can make on the page. | +| Ranking | Pure function over the candidate's linked articles (§4 step 4): per-article evidence `s_i ∈ [−1, 1]` = explicit rating value if rated; else `0.85` if `stage = selected`; else `utility/100` if present; else `blend()/100` if present; else no evidence. Feed score = shrunk mean `(Σ s_i + prior·k) / (n + k)` with `prior = 0.3`, `k = 2`, shown as 0–100. Sort: score desc, then `n` desc, then `last_seen` desc. | Reuses exactly the signals the paper selects on (interest, knn, feed, social, heuristic, triage, quality, fit) with zero new cost. Shrinkage stops a single 95-utility article outranking three 80s. Constants live in one place; not configurable (YAGNI). | +| "Why" column | The three most frequent `top_interests[].name` across the linked articles, plus the linked article titles (top 3 by `s_i`) linking to `/dashboard/articles/{id}`. | Enough to decide at a glance; all read from `signals_json`, no extra work. | +| Add | Row form: `