the-daily-epub/docs/plans/2026-09-07-feed-discovery.md

Feed discovery: propose new Miniflux subscriptions from aggregator hits

Date: 2026-09-07 Repository: thallada/the-daily-epub Status: small implementation plan, ready to execute Builds on: docs/plans/2026-09-03-web-dashboard.md (the admin dashboard this adds a page to), docs/plans/2026-09-02-personalized-curation-v2.md (the per-article signals this reuses)

Written for a fresh implementation agent. Facts about this repo were checked against main (e69940f) on 2026-09-07. Facts about the Miniflux API were checked against https://miniflux.app/docs/api.html the same day; anything marked assumption must be re-checked against the live instance before relying on it.

---

1. Goal

Most of the paper's articles arrive through aggregators (Hacker News, Lobsters, Reddit, Scour), not through the author's own feed. Every such article is a lead on a feed the operator does not subscribe to yet. This feature:

1. During generate, after articles are persisted, asks Miniflux to discover the feed(s) behind each aggregator-only article and records the ones the operator is not already subscribed to as feed candidates.
2. Adds an admin dashboard page, Feeds (/dashboard/feeds), listing candidates ranked by how likely the operator is to enjoy the feed, computed algorithmically from telemetry the pipeline already persists (utility / preliminary blend, stage reached, explicit ratings). No LLM calls, no new embeddings.
3. Each row has Add (with a Miniflux category picker; calls POST /v1/feeds) and Dismiss.

Over time the subscribed set grows and the pool of direct-feed articles each issue draws from grows with it.

Out of scope (deliberately, see §7): auto-subscribing, feed previews/health checks, Scour integration, OPML, per-feed LLM judgement, un-dismissing from the UI.

2. Verified facts

About this repo

• Pipeline order (src/pipeline.rs:354 run_stages): ingest → dedupe → extraction → persist_articles (stage 4, ~line 443, mints real article ids) → social enrichment (stage 5, best effort, ~line 448) → hygiene/embeddings/signals (stage 6) → … Failure policy is documented at the top of the file: best effort stages log, push a warning onto report (status degraded) and continue. The new stage is best effort.
• How an article arrived is stored per article in articles.sources_json as Vec<SourceRef> (src/types.rs:64), each with a SourceKind (Scour | HnFrontpage | Lobsters | Reddit | Feed, src/types.rs:49). Classification is dedupe::classify_source_with_feed (src/dedupe.rs:177). Article::came_via(kind) exists (src/types.rs:147). "Aggregator-only" therefore means !article.sources.iter().any(|s| s.kind == SourceKind::Feed).
• Ingest already loads the whole subscription list once per run: MinifluxClient::ingest_window returns (Vec<Entry>, HashMap<FeedId, FeedMeta>) where FeedMeta has feed_url and site_url (src/miniflux.rs:105). That map is the "already subscribed" set; no extra call needed in the pipeline.
• Miniflux client (src/miniflux.rs) is currently GET-only: one private get_json helper with retry (RetryPolicy, retry on network/5xx/429), X-Auth-Token auth, MinifluxError::{Http, Status, Decode, MissingApiKey}. Module doc says "Reads only … read state is never mutated" — still true for the pipeline; the dashboard Add action is the only writer and only creates feeds.
• Per-article ranking evidence is persisted in candidate_runs (migrations/0002_curation_v2.sql:59): (run_id, article_id) PK, stage, excluded_reason, signals_json, utility REAL, rank_utility, …. signals_json decodes to curate::telemetry::SignalsJson (src/curate/telemetry.rs:114) whose blend() returns the preliminary blend on 0–100 and whose top_interests: Vec<TopInterest{name, z, cos}> names the matched interests. utility (0–100) is set only for articles that reached the deep set (admitted/assessed/shortlisted/selected). Stage names: dashboard::STAGES (src/web/dashboard/mod.rs). runs.status distinguishes finished runs; dry runs are not flagged in runs (a dry run still writes telemetry, which is fine to use).
• Ratings: Db::current_ratings(lookback_days) -> Vec<RatedArticle> (src/db.rs:708) returns one current verdict per article with value (loved 1.0, good 0.35, not_for_me −1.0) — the same numbers signals::RatedExample uses.
• Dashboard conventions: one submodule per page group under src/web/dashboard/, each exposing routes() -> Router<AppState> merged in dashboard::router() (src/web/dashboard/mod.rs:41). Admin gating is applied by the caller. POST-then-redirect with a session flash is the established pattern (users::approve, src/web/dashboard/users.rs:122, uses set_flash(&session, "error"|"ok", text) then Redirect::to). Nav tabs are hardcoded in src/web/templates/layout.html:47–54 keyed on page.active_nav. Overview tiles live in src/web/templates/dashboard/overview.html. Templates are askama; shared pager is dashboard/_pager.html.
• AppState (src/server.rs:79) holds db, config: Arc<RwLock<Arc<Config>>> (read via state.config()), web: Arc<WebState>, mailer. Handlers construct outbound clients from config as needed (bookorbit.rs does this for OPDS); reuse whatever shared reqwest::Client WebState already carries, otherwise http::build_client(http::DEFAULT_TIMEOUT).
• Config: sections are #[serde(deny_unknown_fields, default)] structs on Config with Default impls; env override is DAILY_EPUB_<SECTION>__<KEY>. The settings dashboard has a hardcoded section list, help text and a test asserting the list (src/web/dashboard/settings.rs, see the BookOrbit plan §2 for the exact spots) — a new section must be added there or the settings page/test breaks. config.example.toml and the README config table document every key.
• Migrations: sqlx::migrate!("./migrations"); latest is 0009_request_username.sql, so the new file is 0010_feed_discovery.sql.
• Jobs / CLI: catalogue jobs are in src/jobs.rs and run through main.rs (Job::ImportRatings => imports::run(...)). A one-off backfill only needs a CLI subcommand, not a job (§4 step 6).

About Miniflux (docs, 2026-09-07)

• Discover: POST /v1/discover with body {"url": "<page or site url>"} → [{"url": "<feed url>", "title": "…", "type": "rss|atom|json"}]. Optional body fields (username, password, user_agent, fetch_via_proxy) are not needed. Observed live on 2026-09-07 (operator ran it against several hosts):
  • When the page has no <link rel="alternate">, Miniflux falls back to probing well-known paths on the site root (/atom.xml, /feed.atom, /feed.xml, /feed/, /index.rss, /index.xml, /rss.xml, /rss/, /rss/feed.xml) and returns every path that answered 200 without checking that the body is a feed. A site that serves its SPA shell for any path (zombo.com, immich, BookOrbit) yields up to nine bogus results, each with title == url. Discover results are leads, not facts; every one must be validated before it is stored (§3, §4 step 3).
  • When the target site fails, the response is a 4xx/5xx with {"error_message": "fetcher: bad gateway (502 status code)"}, "fetcher: unexpected status code (412 status code)", or "resource not found" (nothing found / 404). Treat all of these as "host checked, zero candidates" and move on; never retry within the run.
  • Real <link rel="alternate"> hits work from an article URL. Observed for https://blog.philz.dev/blog/language-server-db/: [{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.xml","type":"atom"},{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.json","type":"json"}]. The same content is offered twice (Atom and JSON Feed) under one title, so link-tag hits need deduping by title too (§3). Miniflux substitutes the URL for the title when the tag has none, so title == url marks a result as a well-known-path guess in practice.
• Create feed duplicate: observed HTTP 400 with {"error_message":"This feed already exists."}. The flash shows error_message verbatim; on that exact message the row is also flipped to added (without a miniflux_feed_id), since the operator subscribed by other means.
• Create feed: POST /v1/feeds with {"feed_url": "…", "category_id": <int>} → {"feed_id": <int>}. Errors come back as 4xx/5xx with a JSON error_message; a duplicate subscription is a 4xx (assumption: 400 with "This feed already exists"). Surface error_message in the flash rather than special-casing.
• Categories: GET /v1/categories → [{"id", "user_id", "title", "hide_globally"}].
• Feeds: GET /v1/feeds elements carry feed_url and site_url (already modelled as MinifluxFeed).
• API keys are full-access; the existing key can create feeds. Nothing to provision.

3. Design decisions (settled)

Topic	Decision	Why
Where discovery runs	A best-effort pipeline stage right after social enrichment (needs real article ids, before hygiene). Plus a CLI backfill (daily-epub feeds discover --days N) that runs the same function over recent articles, for seeding.	The user asked for "a step in article processing". The backfill is the same function behind a flag; without it the page is empty until tomorrow.
Which articles	Aggregator-only articles: no SourceKind::Feed in sources. Skip when the canonical URL's host is in discovery.skip_hosts or matches the host of any subscribed feed's site_url/feed_url.	Articles that already came through a direct feed prove nothing. Host match against subscriptions catches "same site, different feed URL" cheaply.
Discovery mechanism	Miniflux POST /v1/discover with the article URL, followed by our own validation fetch of each returned URL. No new crate.	Miniflux already does <link rel=alternate> + well-known paths + YouTube etc. and runs on the same host, but its well-known-path guesses are unverified (§2). Validating is ~40 lines; replacing discover with a home-built finder is far more.
Validation	GET each discovered URL with the shared client (10 s timeout, read at most 256 KB). Accept only if the body, after an optional BOM, whitespace and <?xml …?> declaration, starts with <rss, <feed, or <rdf:RDF, or is JSON containing "https://jsonfeed.org/version/. Take the feed's own <title> from that body (first <title>…</title> in the buffer, unescaped, trimmed) as the candidate title when Miniflux gave title == url. Any other body, a non-2xx status or an error rejects the URL.	Content-type alone is unreliable (SPAs return text/html for everything, some real feeds return text/xml or application/octet-stream). Sniffing the root element is cheap and decisive.
Duplicate results	After validation: group link-tag hits (title != url) by title, keeping one per group and preferring type rss/atom over json; then at most one title == url guess (the first in Miniflux's order); at most 3 per host overall.	blog.philz.dev offers the same feed as Atom and JSON Feed under one title; WordPress answers /feed/ and /rss/, Hugo /index.xml and /feed.xml, all with the same content. One is enough. Real tag/category feeds arrive with distinct link-tag titles and are kept.
Per-host memo	Table feed_discovery_hosts(host PK, checked_at, candidates). A host is looked up once; re-checked only after RECHECK_DAYS = 90 (const). New articles on an already-checked host are just linked to that host's existing candidates (no network).	Keeps per-run lookups bounded and lets a candidate accumulate evidence for free.
Per-run cap	discovery.max_lookups_per_run (default 30), concurrency 4, 10 s timeout per lookup. Order: articles with the highest social score first.	A run already takes ~12 min; discovery must not add more than ~1 min.
Multiple feeds per host	Drop results whose URL or title contains comment before validating. Each kept result is one candidate keyed by canonicalized feed_url (unique).	Comment feeds are noise; tag/category feeds are a real choice the operator can make on the page.
Ranking	Pure function over the candidate's linked articles (§4 step 4): per-article evidence s_i ∈ [−1, 1] = explicit rating value if rated; else 0.85 if stage = selected; else utility/100 if present; else blend()/100 if present; else no evidence. Feed score = shrunk mean (Σ s_i + prior·k) / (n + k) with prior = 0.3, k = 2, shown as 0–100. Sort: score desc, then n desc, then last_seen desc.	Reuses exactly the signals the paper selects on (interest, knn, feed, social, heuristic, triage, quality, fit) with zero new cost. Shrinkage stops a single 95-utility article outranking three 80s. Constants live in one place; not configurable (YAGNI).
"Why" column	The three most frequent top_interests[].name across the linked articles, plus the linked article titles (top 3 by s_i) linking to /dashboard/articles/{id}.	Enough to decide at a glance; all read from signals_json, no extra work.
Add	Row form: <select name="category_id"> + Add → POST /dashboard/feeds/{id}/add → Miniflux POST /v1/feeds → set status = 'added', miniflux_feed_id, decided_at; flash "Added title to category". The select defaults to the last category used (kv key feed_discovery_last_category).	The operator will click Add many times in a row; remembering the category saves a click each time.
Categories	GET /v1/categories on page render, one call. If it fails: flash the error and render the table with Add disabled.	Categories change rarely but caching them is more code than a call.
Dismiss	POST /dashboard/feeds/{id}/dismiss → status = 'dismissed', decided_at. No restore button.	Dismissals are cheap to undo with SQL if ever needed.
Reconciliation	At each run's discovery stage, any candidate whose feed_url (or host) now appears in the subscribed feed map becomes added (with miniflux_feed_id).	Feeds added directly in Miniflux disappear from the list without manual bookkeeping.
Page filter	`?status=candidate	added
Config	[discovery] enabled = true, max_lookups_per_run = 30, skip_hosts = [...].	Exactly the knobs that need turning: kill switch, run-time bound, noise list.

4. Implementation steps

1. Migration migrations/0010_feed_discovery.sql:

   CREATE TABLE feed_candidates (
       id               INTEGER PRIMARY KEY AUTOINCREMENT,
       feed_url         TEXT NOT NULL UNIQUE,
       host             TEXT NOT NULL,
       title            TEXT,
       status           TEXT NOT NULL CHECK (status IN ('candidate', 'added', 'dismissed')),
       first_seen       TEXT NOT NULL,
       last_seen        TEXT NOT NULL,
       miniflux_feed_id INTEGER,
       decided_at       TEXT
   );
   CREATE INDEX idx_feed_candidates_status_host ON feed_candidates(status, host);
   
   CREATE TABLE feed_candidate_articles (
       candidate_id INTEGER NOT NULL REFERENCES feed_candidates(id) ON DELETE CASCADE,
       article_id   INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
       PRIMARY KEY (candidate_id, article_id)
   );
   
   CREATE TABLE feed_discovery_hosts (
       host       TEXT PRIMARY KEY,
       checked_at TEXT NOT NULL,
       candidates INTEGER NOT NULL
   );
   

   Db methods (in a new src/discovery.rs, not db.rs, following imports.rs): hosts_checked(hosts) -> HashMap<host, (checked_at, candidates)>, mark_host_checked, upsert_candidate(feed_url, host, title, now) -> id (bumps last_seen, never demotes added/dismissed), link_article(candidate_id, article_id), candidates_for_host(host), list(status, page), set_status(id, status, miniflux_feed_id, now), reconcile_added(subscribed: &[(feed_url, site_host)], now), count(status).

2. Miniflux client (src/miniflux.rs): add a private post_json<B: Serialize, T: DeserializeOwned>(path, body) mirroring get_json (same retry predicate; do not retry create_feed on 5xx — a retried create can double-subscribe; pass RetryPolicy::none() or a flag). Wire types Discovered { url, title, type }, Category { id, title }, CreatedFeed { feed_id }. Methods: discover(url) -> Vec<Discovered>, categories() -> Vec<Category>, create_feed(feed_url, category_id) -> i64. Make MinifluxError::Status keep the error_message string when the body is JSON so the flash reads "This feed already exists" rather than raw JSON. Update the module doc's "reads only" sentence. Tests: deserialize fixtures for all three.

3. Discovery stage (src/discovery.rs + wiring in src/pipeline.rs after stage 5, timing key "discovery", report count feed_candidates_new, warning on error):

   select aggregator-only articles → drop skip_hosts / subscribed hosts
     → reconcile_added(subscribed)
     → for each distinct host: if checked within RECHECK_DAYS → link article to its candidates
                              else queue (cap max_lookups_per_run, highest social score first)
     → discover() ×4 concurrent (an error_message response = host checked, 0 candidates)
     → drop "comment" results and already-subscribed feed_urls
     → validate each remaining URL (GET, ≤256 KB, feed-root sniff, pull <title>)
     → one link-tag hit per title (xml over json) + at most one guessed URL, max 3
     → upsert_candidate + link_article + mark_host_checked
   

   The function signature is pub async fn run(db, miniflux: &MinifluxClient, http: &reqwest::Client, cfg: &DiscoveryConfig, articles: &[Article], subscribed: &HashMap<FeedId, FeedMeta>, now) -> Result<Summary> so the pipeline and the CLI share it. Host = url::Url::host_str() lowercased with a leading www. stripped; compare subscriptions the same way. Budget: one discover call plus up to nine validation GETs per host, so with the 30-host cap the stage is bounded at ~300 small requests, 4 in flight; log the count. Pure helpers (aggregator_only, host_of, keep_result, sniff_feed(&[u8]) -> Option<FeedKind>, feed_title(&[u8]), dedupe_guesses) get unit tests, including the observed zombo.com response (nine title == url guesses that all fail the sniff → zero candidates), the observed blog.philz.dev response (Atom + JSON under one title → exactly the Atom candidate), and an SPA shell body (<!doctype html>…) rejected by sniff_feed.

4. Ranking (src/discovery.rs, pure): pub fn score(evidence: &[ArticleEvidence]) -> f64 and pub fn why(evidence) -> Vec<String> per §3. Evidence loading is one query per page render: for the page's candidates, SELECT article_id, stage, utility, signals_json FROM candidate_runs WHERE article_id IN (…) ORDER BY run_id DESC keeping the latest row per article, joined with current_ratings. The page holds ≤ 50 candidates × a handful of articles each, so ranking happens in Rust after loading all candidate-status rows (sorting must be global, not per page). If the candidate count ever exceeds ~2 000 this needs a stored score; not now. Unit tests: rated article dominates; shrinkage (one 0.95 article scores below three 0.8s); no evidence → prior; why picks the modal interests.

5. Dashboard page src/web/dashboard/feeds.rs + src/web/templates/dashboard/feeds.html:

   • Routes: GET /dashboard/feeds, POST /dashboard/feeds/{id}/add, POST /dashboard/feeds/{id}/dismiss. Merge into dashboard::router(). Nav tab Feeds in layout.html (active_nav = "feeds"). Overview tile "Feed candidates → review" linking to the page.
   • Table columns: Score (0–100, tabular numerals), Feed (title → feed_url, host beneath), Why (interest names + up to 3 article titles linking to the article page), Articles (n), Seen (first/last via format_time), Actions (category select + Add; Dismiss). ?status= filter links at the top. Added rows show the category and a link to the Miniflux feed page ({base_url}/feeds/{miniflux_feed_id}); dismissed rows show only the date.
   • Add handler: read category_id from the form and pass it through (Miniflux validates it), call create_feed. On Ok set status + kv last category + flash ok. On a 400 whose error_message is This feed already exists. set status added and flash that message as info. On any other Err flash error_message (or the transport error) and leave the row as candidate. Requires state.config().miniflux.api_key; if the client fails to build, flash "Miniflux API key is not configured".
   • Server test in the existing dashboard test harness: page renders with an empty table; add/dismiss flip status (point miniflux.base_url at a tokio listener that answers the create call, as the BookOrbit tests do for OPDS; note the loopback-bind caveat in memory/codex-sandbox-quirk.md).

6. CLI backfill (src/main.rs): daily-epub feeds discover [--days 14] [--limit 200] — loads articles with first_seen in the window, fetches the feed map (MinifluxClient::feed_map), calls discovery::run with max_lookups_per_run = limit, prints the summary. Takes the pipeline run lock (it writes the same tables the run does). No job-catalogue entry.

7. Config + docs: DiscoveryConfig { enabled: bool, max_lookups_per_run: usize, skip_hosts: Vec<String> } with defaults true, 30, ["news.ycombinator.com", "lobste.rs", "reddit.com", "github.com", "gist.github.com", "x.com", "twitter.com", "youtube.com", "en.wikipedia.org", "arxiv.org", "docs.google.com"]. Register the section in the settings dashboard (list, help text, test). Document in config.example.toml and the README table. Add a short "Feeds" paragraph to the dashboard section of the README.

Order matters only for 1 → 2 → 3 → 4 → 5; 6 and 7 can be done alongside. Expect roughly 900 lines including tests.

5. Verification

• cargo test green; new pure-function tests in discovery.rs, wire-type tests in miniflux.rs, one handler test for the page.
• Live, on the server after deploy: run daily-epub feeds discover --days 14 --limit 50, open /dashboard/feeds, confirm the top rows look like feeds the operator would actually want, click Add on one, confirm it appears in Miniflux under the chosen category and the row flips to added. Run generate --dry-run and check the report has a discovery timing under ~60 s and no warning.
• Check the journalctl line for how many hosts were skipped as already subscribed; if it is near zero the www./host normalisation is wrong.
• Check the line for discover results rejected by validation. A healthy day should reject a good share of them (SPA catch-alls); zero rejections means the sniff is too lenient, and near-total rejection means it is too strict (look for feeds served with a leading comment or DOCTYPE).

6. Rollout notes

• Migrations run automatically on the first daily-epub command after deploy.
• No new secrets: the existing DAILY_EPUB_MINIFLUX__API_KEY can create feeds.
• Seed the page once with the CLI backfill so it is useful on day one.
• Every Miniflux behaviour the design depends on was observed live on 2026-09-07 (§2); the recorded responses are the test fixtures. Nothing remains to confirm before implementation.

7. Non-goals (do not add)

• Auto-subscribing without a click, or any LLM judgement of feeds.
• Fetching or previewing feed contents, feed health checks, unread counts.
• Un-dismiss / undo buttons, per-row notes, bulk actions, sorting controls (the list is already sorted by the only order that matters).
• A stored/cached score column, a scheduled job, OPML export, Scour "recommended feeds" integration (ideas.md item, separate).
• Configurable ranking constants.