Compare commits

..
7 Commits
Author SHA1 Message Date
thallada 72b32dc69b Add justfile
Makes running commands easier.
2026-09-07 06:08:33 +00:00
thallada 56d2d74a97 Fix styling of Feeds dashboard overflowing container 2026-09-07 06:08:08 +00:00
thalladaandClaude Fable 5.1 81cbbb407f Add the Feeds dashboard page for reviewing feed candidates
Ranked by the shrunk mean of the linked articles' telemetry and ratings,
with Add (into a chosen Miniflux category) and Dismiss per row, a status
filter, a nav tab and an overview tile.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4
2026-09-07 04:45:37 +00:00
thallada c0f086e0da Merge branch 'fd-core' into feed-discovery 2026-09-07 04:31:24 +00:00
thalladaandClaude Fable 5.1 2bbb4809b7 Discover feeds behind aggregator-only articles
A best-effort pipeline stage after social enrichment asks Miniflux to
discover the feeds behind each aggregator-only article, validates every
result by sniffing the body, and records the survivors as feed candidates
with a per-host memo. Ranking is a shrunk mean over the linked articles'
existing telemetry and ratings. `daily-epub feeds discover` runs the same
pass over recent articles for seeding.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4
2026-09-07 04:31:13 +00:00
thalladaandClaude Fable 5.1 c971fd9317 Register the [discovery] section in settings and document it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4
2026-09-07 04:17:29 +00:00
thalladaandClaude Fable 5.1 4a2fc6d1c4 Add the [discovery] config section for feed discovery
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4
2026-09-07 04:09:37 +00:00
19 changed files with 3063 additions and 24 deletions

No files matched your search

+19
View File
@@ -133,6 +133,7 @@ daily-epub stats [--days 14] # the evaluation framework, one fact per line
daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes] daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes]
daily-epub features prune # stale embeddings, old telemetry and assessments daily-epub features prune # stale embeddings, old telemetry and assessments
daily-epub backfill-social [--days 7] # re-poll social scores for recent articles daily-epub backfill-social [--days 7] # re-poll social scores for recent articles
daily-epub feeds discover [--days 14] [--limit 50] # seed feed candidates from recent aggregator articles
daily-epub db migrate # run migrations (also automatic on every start) daily-epub db migrate # run migrations (also automatic on every start)
daily-epub config check # validate the config, print the resolved roles, keys and paths daily-epub config check # validate the config, print the resolved roles, keys and paths
daily-epub users add USER [--admin] [--password-stdin] daily-epub users add USER [--admin] [--password-stdin]
@@ -145,6 +146,13 @@ daily-epub users logout USER # revoke all of USER's sessions
daily-epub job run NAME # systemd job-unit entry point; normally not run by hand daily-epub job run NAME # systemd job-unit entry point; normally not run by hand
``` ```
On the server, run these through the `justfile` (`just --list`): every server
recipe wraps the binary in `systemd-run` as the `daily-epub` user with
`/etc/daily-epub/env` loaded, exactly as the units do, so API keys resolve.
`just de <subcommand...>` runs anything; `just feeds-discover 30`,
`just config-check`, `just deploy` and `just logs-generate` cover the common cases.
Plain `sudo -u daily-epub daily-epub ...` does **not** read the env file.
`--dry-run` does everything except deliver: it still ingests, persists entries and `--dry-run` does everything except deliver: it still ingests, persists entries and
articles, curates and **builds both EPUBs into `--out`**, but it does not copy to articles, curates and **builds both EPUBs into `--out`**, but it does not copy to
BookOrbit, does not run the retention sweep, does not write the `issues` row and BookOrbit, does not run the retention sweep, does not write the `issues` row and
@@ -245,6 +253,14 @@ verdict and optional note. Imports run in the background and show per-URL
status in the page plus live logs on the job page. If dashboard jobs are status in the page plus live logs on the job page. If dashboard jobs are
disabled, queueing still works; run `daily-epub job run import-ratings` by hand. disabled, queueing still works; run `daily-epub job run import-ratings` by hand.
The Feeds page lists feeds discovered behind articles that reached the paper
only through an aggregator (Hacker News, Lobsters, Reddit, Scour), ranked by
the same per-article signals the paper selects on, so the strongest leads sit
at the top. Each row offers **Add**, which subscribes the feed in a chosen
Miniflux category, and **Dismiss**. Discovery runs as a best-effort stage
during `generate`; `daily-epub feeds discover --days 14 --limit 50` seeds the
page from articles already in the database.
The Settings page derives its fields from `Config`, rewrites `config.toml` in The Settings page derives its fields from `Config`, rewrites `config.toml` in
place with `toml_edit`, preserves comments/order and file permissions, validates place with `toml_edit`, preserves comments/order and file permissions, validates
before an atomic rename, and records attributed history. It re-reads hand edits before an atomic rename, and records attributed history. It re-reads hand edits
@@ -407,6 +423,9 @@ prints what resolved.
| `mail.smtp_pass` | — | **`DAILY_EPUB_MAIL__SMTP_PASS`**, environment only. | | `mail.smtp_pass` | — | **`DAILY_EPUB_MAIL__SMTP_PASS`**, environment only. |
| `mail.from` | `""` | Sender mailbox, either a bare address or `Name <address>`. | | `mail.from` | `""` | Sender mailbox, either a bare address or `Name <address>`. |
| `mail.notify_to` | unset | Recipient for access-request notifications. | | `mail.notify_to` | unset | Recipient for access-request notifications. |
| `discovery.enabled` | `true` | Run the feed discovery stage during `generate`. |
| `discovery.max_lookups_per_run` | `30` | Hosts one run may look up in Miniflux. Each host is re-checked at most every 90 days. |
| `discovery.skip_hosts` | aggregators, code hosts, social networks | Hosts never looked up. A host matches itself or any subdomain of it. |
| `xtc.enabled` | `true` | Set `false` to skip the converter entirely. | | `xtc.enabled` | `true` | Set `false` to skip the converter entirely. |
| `xtc.command` | `node` | Converter executable. | | `xtc.command` | `node` | Converter executable. |
| `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends `<input.epub> -o <output> -f <format>` (plus `-c <settings>`). | | `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends `<input.epub> -o <output> -f <format>` (plus `-c <settings>`). |
+17
View File
@@ -237,3 +237,20 @@ smtp_user = ""
# smtp_pass: environment only (DAILY_EPUB_MAIL__SMTP_PASS) # smtp_pass: environment only (DAILY_EPUB_MAIL__SMTP_PASS)
from = "" # e.g. The Daily EPUB <daily@hallada.net> from = "" # e.g. The Daily EPUB <daily@hallada.net>
# notify_to = "operator@example.com" # recipient for access-request notifications # notify_to = "operator@example.com" # recipient for access-request notifications
[discovery]
enabled = true # run the feed discovery stage during generate
max_lookups_per_run = 30 # hosts looked up in Miniflux per run; each host is re-checked at most every 90 days
skip_hosts = [ # never looked up: the host itself or any subdomain of it
"news.ycombinator.com",
"lobste.rs",
"reddit.com",
"github.com",
"gist.github.com",
"x.com",
"twitter.com",
"youtube.com",
"en.wikipedia.org",
"arxiv.org",
"docs.google.com",
]
+157
View File
@@ -0,0 +1,157 @@
# Feed discovery: propose new Miniflux subscriptions from aggregator hits
**Date:** 2026-09-07
**Repository:** `thallada/the-daily-epub`
**Status:** small implementation plan, ready to execute
**Builds on:** `docs/plans/2026-09-03-web-dashboard.md` (the admin dashboard this adds a page to), `docs/plans/2026-09-02-personalized-curation-v2.md` (the per-article signals this reuses)
Written for a fresh implementation agent. Facts about this repo were checked against `main` (`e69940f`) on 2026-09-07. Facts about the Miniflux API were checked against `https://miniflux.app/docs/api.html` the same day; anything marked *assumption* must be re-checked against the live instance before relying on it.
---
## 1. Goal
Most of the paper's articles arrive through aggregators (Hacker News, Lobsters, Reddit, Scour), not through the author's own feed. Every such article is a lead on a feed the operator does not subscribe to yet. This feature:
1. During `generate`, after articles are persisted, asks Miniflux to discover the feed(s) behind each aggregator-only article and records the ones the operator is not already subscribed to as **feed candidates**.
2. Adds an admin dashboard page, **Feeds** (`/dashboard/feeds`), listing candidates ranked by how likely the operator is to enjoy the feed, computed **algorithmically from telemetry the pipeline already persists** (utility / preliminary blend, stage reached, explicit ratings). No LLM calls, no new embeddings.
3. Each row has **Add** (with a Miniflux category picker; calls `POST /v1/feeds`) and **Dismiss**.
Over time the subscribed set grows and the pool of direct-feed articles each issue draws from grows with it.
Out of scope (deliberately, see §7): auto-subscribing, feed previews/health checks, Scour integration, OPML, per-feed LLM judgement, un-dismissing from the UI.
## 2. Verified facts
### About this repo
- **Pipeline order** (`src/pipeline.rs:354` `run_stages`): ingest → dedupe → extraction → `persist_articles` (stage 4, ~line 443, mints real article ids) → social enrichment (stage 5, best effort, ~line 448) → hygiene/embeddings/signals (stage 6) → … Failure policy is documented at the top of the file: *best effort* stages log, push a warning onto `report` (status `degraded`) and continue. The new stage is best effort.
- **How an article arrived** is stored per article in `articles.sources_json` as `Vec<SourceRef>` (`src/types.rs:64`), each with a `SourceKind` (`Scour | HnFrontpage | Lobsters | Reddit | Feed`, `src/types.rs:49`). Classification is `dedupe::classify_source_with_feed` (`src/dedupe.rs:177`). `Article::came_via(kind)` exists (`src/types.rs:147`). "Aggregator-only" therefore means `!article.sources.iter().any(|s| s.kind == SourceKind::Feed)`.
- **Ingest already loads the whole subscription list once per run**: `MinifluxClient::ingest_window` returns `(Vec<Entry>, HashMap<FeedId, FeedMeta>)` where `FeedMeta` has `feed_url` and `site_url` (`src/miniflux.rs:105`). That map is the "already subscribed" set; no extra call needed in the pipeline.
- **Miniflux client** (`src/miniflux.rs`) is currently GET-only: one private `get_json` helper with retry (`RetryPolicy`, retry on network/5xx/429), `X-Auth-Token` auth, `MinifluxError::{Http, Status, Decode, MissingApiKey}`. Module doc says "Reads only … read state is never mutated" — still true for the pipeline; the dashboard **Add** action is the only writer and only creates feeds.
- **Per-article ranking evidence is persisted** in `candidate_runs` (`migrations/0002_curation_v2.sql:59`): `(run_id, article_id) PK, stage, excluded_reason, signals_json, utility REAL, rank_utility, …`. `signals_json` decodes to `curate::telemetry::SignalsJson` (`src/curate/telemetry.rs:114`) whose `blend()` returns the preliminary blend on 0–100 and whose `top_interests: Vec<TopInterest{name, z, cos}>` names the matched interests. `utility` (0–100) is set only for articles that reached the deep set (`admitted`/`assessed`/`shortlisted`/`selected`). Stage names: `dashboard::STAGES` (`src/web/dashboard/mod.rs`). `runs.status` distinguishes finished runs; dry runs are not flagged in `runs` (a dry run still writes telemetry, which is fine to use).
- **Ratings**: `Db::current_ratings(lookback_days) -> Vec<RatedArticle>` (`src/db.rs:708`) returns one current verdict per article with `value` (`loved` 1.0, `good` 0.35, `not_for_me` −1.0) — the same numbers `signals::RatedExample` uses.
- **Dashboard conventions**: one submodule per page group under `src/web/dashboard/`, each exposing `routes() -> Router<AppState>` merged in `dashboard::router()` (`src/web/dashboard/mod.rs:41`). Admin gating is applied by the caller. POST-then-redirect with a session flash is the established pattern (`users::approve`, `src/web/dashboard/users.rs:122`, uses `set_flash(&session, "error"|"ok", text)` then `Redirect::to`). Nav tabs are hardcoded in `src/web/templates/layout.html:47–54` keyed on `page.active_nav`. Overview tiles live in `src/web/templates/dashboard/overview.html`. Templates are askama; shared pager is `dashboard/_pager.html`.
- **`AppState`** (`src/server.rs:79`) holds `db`, `config: Arc<RwLock<Arc<Config>>>` (read via `state.config()`), `web: Arc<WebState>`, `mailer`. Handlers construct outbound clients from config as needed (`bookorbit.rs` does this for OPDS); reuse whatever shared `reqwest::Client` `WebState` already carries, otherwise `http::build_client(http::DEFAULT_TIMEOUT)`.
- **Config**: sections are `#[serde(deny_unknown_fields, default)]` structs on `Config` with `Default` impls; env override is `DAILY_EPUB_<SECTION>__<KEY>`. The settings dashboard has a **hardcoded section list**, help text and a test asserting the list (`src/web/dashboard/settings.rs`, see the BookOrbit plan §2 for the exact spots) — a new section must be added there or the settings page/test breaks. `config.example.toml` and the README config table document every key.
- **Migrations**: `sqlx::migrate!("./migrations")`; latest is `0009_request_username.sql`, so the new file is `0010_feed_discovery.sql`.
- **Jobs / CLI**: catalogue jobs are in `src/jobs.rs` and run through `main.rs` (`Job::ImportRatings => imports::run(...)`). A one-off backfill only needs a CLI subcommand, not a job (§4 step 6).
### About Miniflux (docs, 2026-09-07)
- **Discover**: `POST /v1/discover` with body `{"url": "<page or site url>"}` → `[{"url": "<feed url>", "title": "…", "type": "rss|atom|json"}]`. Optional body fields (`username`, `password`, `user_agent`, `fetch_via_proxy`) are not needed. **Observed live on 2026-09-07** (operator ran it against several hosts):
- When the page has no `<link rel="alternate">`, Miniflux falls back to probing well-known paths on the site root (`/atom.xml`, `/feed.atom`, `/feed.xml`, `/feed/`, `/index.rss`, `/index.xml`, `/rss.xml`, `/rss/`, `/rss/feed.xml`) and **returns every path that answered 200 without checking that the body is a feed**. A site that serves its SPA shell for any path (zombo.com, immich, BookOrbit) yields up to nine bogus results, each with `title == url`. **Discover results are leads, not facts; every one must be validated before it is stored (§3, §4 step 3).**
- When the target site fails, the response is a 4xx/5xx with `{"error_message": "fetcher: bad gateway (502 status code)"}`, `"fetcher: unexpected status code (412 status code)"`, or `"resource not found"` (nothing found / 404). Treat all of these as "host checked, zero candidates" and move on; never retry within the run.
- Real `<link rel="alternate">` hits work from an **article** URL. Observed for `https://blog.philz.dev/blog/language-server-db/`:
`[{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.xml","type":"atom"},{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.json","type":"json"}]`.
The same content is offered twice (Atom and JSON Feed) under one title, so link-tag hits need deduping by title too (§3). Miniflux substitutes the URL for the title when the tag has none, so `title == url` marks a result as a well-known-path guess in practice.
- **Create feed duplicate**: observed `HTTP 400` with `{"error_message":"This feed already exists."}`. The flash shows `error_message` verbatim; on that exact message the row is also flipped to `added` (without a `miniflux_feed_id`), since the operator subscribed by other means.
- **Create feed**: `POST /v1/feeds` with `{"feed_url": "…", "category_id": <int>}` → `{"feed_id": <int>}`. Errors come back as 4xx/5xx with a JSON `error_message`; a duplicate subscription is a 4xx (*assumption:* 400 with "This feed already exists"). Surface `error_message` in the flash rather than special-casing.
- **Categories**: `GET /v1/categories` → `[{"id", "user_id", "title", "hide_globally"}]`.
- **Feeds**: `GET /v1/feeds` elements carry `feed_url` and `site_url` (already modelled as `MinifluxFeed`).
- API keys are full-access; the existing key can create feeds. Nothing to provision.
## 3. Design decisions (settled)
| Topic | Decision | Why |
|---|---|---|
| Where discovery runs | A best-effort **pipeline stage right after social enrichment** (needs real article ids, before hygiene). Plus a CLI backfill (`daily-epub feeds discover --days N`) that runs the same function over recent `articles`, for seeding. | The user asked for "a step in article processing". The backfill is the same function behind a flag; without it the page is empty until tomorrow. |
| Which articles | Aggregator-only articles: no `SourceKind::Feed` in `sources`. Skip when the canonical URL's host is in `discovery.skip_hosts` or matches the host of any subscribed feed's `site_url`/`feed_url`. | Articles that already came through a direct feed prove nothing. Host match against subscriptions catches "same site, different feed URL" cheaply. |
| Discovery mechanism | **Miniflux `POST /v1/discover`** with the article URL, followed by **our own validation fetch** of each returned URL. No new crate. | Miniflux already does `<link rel=alternate>` + well-known paths + YouTube etc. and runs on the same host, but its well-known-path guesses are unverified (§2). Validating is ~40 lines; replacing discover with a home-built finder is far more. |
| Validation | `GET` each discovered URL with the shared client (10 s timeout, read at most 256 KB). Accept only if the body, after an optional BOM, whitespace and `<?xml …?>` declaration, starts with `<rss`, `<feed`, or `<rdf:RDF`, or is JSON containing `"https://jsonfeed.org/version/`. Take the feed's own `<title>` from that body (first `<title>…</title>` in the buffer, unescaped, trimmed) as the candidate title when Miniflux gave `title == url`. Any other body, a non-2xx status or an error rejects the URL. | Content-type alone is unreliable (SPAs return `text/html` for everything, some real feeds return `text/xml` or `application/octet-stream`). Sniffing the root element is cheap and decisive. |
| Duplicate results | After validation: group link-tag hits (`title != url`) by `title`, keeping one per group and preferring `type` `rss`/`atom` over `json`; then at most **one** `title == url` guess (the first in Miniflux's order); at most 3 per host overall. | blog.philz.dev offers the same feed as Atom and JSON Feed under one title; WordPress answers `/feed/` and `/rss/`, Hugo `/index.xml` and `/feed.xml`, all with the same content. One is enough. Real tag/category feeds arrive with distinct link-tag titles and are kept. |
| Per-host memo | Table `feed_discovery_hosts(host PK, checked_at, candidates)`. A host is looked up **once**; re-checked only after `RECHECK_DAYS = 90` (const). New articles on an already-checked host are just **linked** to that host's existing candidates (no network). | Keeps per-run lookups bounded and lets a candidate accumulate evidence for free. |
| Per-run cap | `discovery.max_lookups_per_run` (default 30), concurrency 4, 10 s timeout per lookup. Order: articles with the highest social score first. | A run already takes ~12 min; discovery must not add more than ~1 min. |
| Multiple feeds per host | Drop results whose URL or title contains `comment` before validating. Each kept result is one candidate keyed by canonicalized `feed_url` (unique). | Comment feeds are noise; tag/category feeds are a real choice the operator can make on the page. |
| Ranking | Pure function over the candidate's linked articles (§4 step 4): per-article evidence `s_i ∈ [−1, 1]` = explicit rating value if rated; else `0.85` if `stage = selected`; else `utility/100` if present; else `blend()/100` if present; else no evidence. Feed score = shrunk mean `(Σ s_i + prior·k) / (n + k)` with `prior = 0.3`, `k = 2`, shown as 0–100. Sort: score desc, then `n` desc, then `last_seen` desc. | Reuses exactly the signals the paper selects on (interest, knn, feed, social, heuristic, triage, quality, fit) with zero new cost. Shrinkage stops a single 95-utility article outranking three 80s. Constants live in one place; not configurable (YAGNI). |
| "Why" column | The three most frequent `top_interests[].name` across the linked articles, plus the linked article titles (top 3 by `s_i`) linking to `/dashboard/articles/{id}`. | Enough to decide at a glance; all read from `signals_json`, no extra work. |
| Add | Row form: `<select name="category_id">` + **Add** → `POST /dashboard/feeds/{id}/add` → Miniflux `POST /v1/feeds` → set `status = 'added'`, `miniflux_feed_id`, `decided_at`; flash "Added *title* to *category*". The select defaults to the last category used (`kv` key `feed_discovery_last_category`). | The operator will click Add many times in a row; remembering the category saves a click each time. |
| Categories | `GET /v1/categories` **on page render**, one call. If it fails: flash the error and render the table with Add disabled. | Categories change rarely but caching them is more code than a call. |
| Dismiss | `POST /dashboard/feeds/{id}/dismiss` → `status = 'dismissed'`, `decided_at`. No restore button. | Dismissals are cheap to undo with SQL if ever needed. |
| Reconciliation | At each run's discovery stage, any `candidate` whose `feed_url` (or host) now appears in the subscribed feed map becomes `added` (with `miniflux_feed_id`). | Feeds added directly in Miniflux disappear from the list without manual bookkeeping. |
| Page filter | `?status=candidate|added|dismissed` (default `candidate`), allow-listed; paginated with `_pager.html`, 50 rows. | One query param buys an audit view of what was added/dismissed. |
| Config | `[discovery] enabled = true, max_lookups_per_run = 30, skip_hosts = [...]`. | Exactly the knobs that need turning: kill switch, run-time bound, noise list. |
## 4. Implementation steps
1. **Migration** `migrations/0010_feed_discovery.sql`:
```sql
CREATE TABLE feed_candidates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
feed_url TEXT NOT NULL UNIQUE,
host TEXT NOT NULL,
title TEXT,
status TEXT NOT NULL CHECK (status IN ('candidate', 'added', 'dismissed')),
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
miniflux_feed_id INTEGER,
decided_at TEXT
);
CREATE INDEX idx_feed_candidates_status_host ON feed_candidates(status, host);
CREATE TABLE feed_candidate_articles (
candidate_id INTEGER NOT NULL REFERENCES feed_candidates(id) ON DELETE CASCADE,
article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
PRIMARY KEY (candidate_id, article_id)
);
CREATE TABLE feed_discovery_hosts (
host TEXT PRIMARY KEY,
checked_at TEXT NOT NULL,
candidates INTEGER NOT NULL
);
```
`Db` methods (in a new `src/discovery.rs`, not `db.rs`, following `imports.rs`): `hosts_checked(hosts) -> HashMap<host, (checked_at, candidates)>`, `mark_host_checked`, `upsert_candidate(feed_url, host, title, now) -> id` (bumps `last_seen`, never demotes `added`/`dismissed`), `link_article(candidate_id, article_id)`, `candidates_for_host(host)`, `list(status, page)`, `set_status(id, status, miniflux_feed_id, now)`, `reconcile_added(subscribed: &[(feed_url, site_host)], now)`, `count(status)`.
2. **Miniflux client** (`src/miniflux.rs`): add a private `post_json<B: Serialize, T: DeserializeOwned>(path, body)` mirroring `get_json` (same retry predicate; **do not retry** `create_feed` on 5xx — a retried create can double-subscribe; pass `RetryPolicy::none()` or a flag). Wire types `Discovered { url, title, type }`, `Category { id, title }`, `CreatedFeed { feed_id }`. Methods: `discover(url) -> Vec<Discovered>`, `categories() -> Vec<Category>`, `create_feed(feed_url, category_id) -> i64`. Make `MinifluxError::Status` keep the `error_message` string when the body is JSON so the flash reads "This feed already exists" rather than raw JSON. Update the module doc's "reads only" sentence. Tests: deserialize fixtures for all three.
3. **Discovery stage** (`src/discovery.rs` + wiring in `src/pipeline.rs` after stage 5, timing key `"discovery"`, report count `feed_candidates_new`, warning on error):
```text
select aggregator-only articles → drop skip_hosts / subscribed hosts
→ reconcile_added(subscribed)
→ for each distinct host: if checked within RECHECK_DAYS → link article to its candidates
else queue (cap max_lookups_per_run, highest social score first)
→ discover() ×4 concurrent (an error_message response = host checked, 0 candidates)
→ drop "comment" results and already-subscribed feed_urls
→ validate each remaining URL (GET, ≤256 KB, feed-root sniff, pull <title>)
→ one link-tag hit per title (xml over json) + at most one guessed URL, max 3
→ upsert_candidate + link_article + mark_host_checked
```
The function signature is `pub async fn run(db, miniflux: &MinifluxClient, http: &reqwest::Client, cfg: &DiscoveryConfig, articles: &[Article], subscribed: &HashMap<FeedId, FeedMeta>, now) -> Result<Summary>` so the pipeline and the CLI share it. Host = `url::Url::host_str()` lowercased with a leading `www.` stripped; compare subscriptions the same way. Budget: one discover call plus up to nine validation GETs per host, so with the 30-host cap the stage is bounded at ~300 small requests, 4 in flight; log the count. Pure helpers (`aggregator_only`, `host_of`, `keep_result`, `sniff_feed(&[u8]) -> Option<FeedKind>`, `feed_title(&[u8])`, `dedupe_guesses`) get unit tests, including the observed zombo.com response (nine `title == url` guesses that all fail the sniff → zero candidates), the observed blog.philz.dev response (Atom + JSON under one title → exactly the Atom candidate), and an SPA shell body (`<!doctype html>…`) rejected by `sniff_feed`.
4. **Ranking** (`src/discovery.rs`, pure): `pub fn score(evidence: &[ArticleEvidence]) -> f64` and `pub fn why(evidence) -> Vec<String>` per §3. Evidence loading is one query per page render: for the page's candidates, `SELECT article_id, stage, utility, signals_json FROM candidate_runs WHERE article_id IN (…) ORDER BY run_id DESC` keeping the latest row per article, joined with `current_ratings`. The page holds ≤ 50 candidates × a handful of articles each, so ranking happens in Rust after loading **all** `candidate`-status rows (sorting must be global, not per page). If the candidate count ever exceeds ~2 000 this needs a stored score; not now.
Unit tests: rated article dominates; shrinkage (one 0.95 article scores below three 0.8s); no evidence → prior; `why` picks the modal interests.
5. **Dashboard page** `src/web/dashboard/feeds.rs` + `src/web/templates/dashboard/feeds.html`:
- Routes: `GET /dashboard/feeds`, `POST /dashboard/feeds/{id}/add`, `POST /dashboard/feeds/{id}/dismiss`. Merge into `dashboard::router()`. Nav tab **Feeds** in `layout.html` (`active_nav = "feeds"`). Overview tile "Feed candidates → review" linking to the page.
- Table columns: Score (0–100, tabular numerals), Feed (title → `feed_url`, host beneath), Why (interest names + up to 3 article titles linking to the article page), Articles (n), Seen (first/last via `format_time`), Actions (category select + Add; Dismiss). `?status=` filter links at the top. Added rows show the category and a link to the Miniflux feed page (`{base_url}/feeds/{miniflux_feed_id}`); dismissed rows show only the date.
- Add handler: read `category_id` from the form and pass it through (Miniflux validates it), call `create_feed`. On `Ok` set status + `kv` last category + flash ok. On a 400 whose `error_message` is `This feed already exists.` set status `added` and flash that message as info. On any other `Err` flash `error_message` (or the transport error) and leave the row as `candidate`. Requires `state.config().miniflux.api_key`; if the client fails to build, flash "Miniflux API key is not configured".
- Server test in the existing dashboard test harness: page renders with an empty table; add/dismiss flip `status` (point `miniflux.base_url` at a `tokio` listener that answers the create call, as the BookOrbit tests do for OPDS; note the loopback-bind caveat in `memory/codex-sandbox-quirk.md`).
6. **CLI backfill** (`src/main.rs`): `daily-epub feeds discover [--days 14] [--limit 200]` — loads `articles` with `first_seen` in the window, fetches the feed map (`MinifluxClient::feed_map`), calls `discovery::run` with `max_lookups_per_run = limit`, prints the summary. Takes the pipeline run lock (it writes the same tables the run does). No job-catalogue entry.
7. **Config + docs**: `DiscoveryConfig { enabled: bool, max_lookups_per_run: usize, skip_hosts: Vec<String> }` with defaults `true, 30, ["news.ycombinator.com", "lobste.rs", "reddit.com", "github.com", "gist.github.com", "x.com", "twitter.com", "youtube.com", "en.wikipedia.org", "arxiv.org", "docs.google.com"]`. Register the section in the settings dashboard (list, help text, test). Document in `config.example.toml` and the README table. Add a short "Feeds" paragraph to the dashboard section of the README.
Order matters only for 1 → 2 → 3 → 4 → 5; 6 and 7 can be done alongside. Expect roughly 900 lines including tests.
## 5. Verification
- `cargo test` green; new pure-function tests in `discovery.rs`, wire-type tests in `miniflux.rs`, one handler test for the page.
- Live, on the server after deploy: run `daily-epub feeds discover --days 14 --limit 50`, open `/dashboard/feeds`, confirm the top rows look like feeds the operator would actually want, click **Add** on one, confirm it appears in Miniflux under the chosen category and the row flips to *added*. Run `generate --dry-run` and check the report has a `discovery` timing under ~60 s and no warning.
- Check the `journalctl` line for how many hosts were skipped as already subscribed; if it is near zero the `www.`/host normalisation is wrong.
- Check the line for discover results rejected by validation. A healthy day should reject a good share of them (SPA catch-alls); zero rejections means the sniff is too lenient, and near-total rejection means it is too strict (look for feeds served with a leading comment or DOCTYPE).
## 6. Rollout notes
- Migrations run automatically on the first `daily-epub` command after deploy.
- No new secrets: the existing `DAILY_EPUB_MINIFLUX__API_KEY` can create feeds.
- Seed the page once with the CLI backfill so it is useful on day one.
- Every Miniflux behaviour the design depends on was observed live on 2026-09-07 (§2); the recorded responses are the test fixtures. Nothing remains to confirm before implementation.
## 7. Non-goals (do not add)
- Auto-subscribing without a click, or any LLM judgement of feeds.
- Fetching or previewing feed contents, feed health checks, unread counts.
- Un-dismiss / undo buttons, per-row notes, bulk actions, sorting controls (the list is already sorted by the only order that matters).
- A stored/cached score column, a scheduled job, OPML export, Scour "recommended feeds" integration (ideas.md item, separate).
- Configurable ranking constants.
+230
View File
@@ -0,0 +1,230 @@
# Operator and developer shortcuts for The Daily EPUB.
#
# just list recipes
# just de <args> run any daily-epub subcommand on the server as the service user
#
# Server recipes run the binary the way the systemd units do: same user, working
# directory and EnvironmentFile (`/etc/daily-epub/env`, where the API keys live),
# via a transient `systemd-run` unit. Plain `sudo -u daily-epub` never loads that
# file, so `miniflux api key is not configured` from a hand-run command means the
# environment, not the config.
set positional-arguments
bin := "/usr/local/bin/daily-epub"
config := "/etc/daily-epub/config.toml"
env_file := "/etc/daily-epub/env"
work_dir := "/var/lib/daily-epub"
user := "daily-epub"
# The systemd-run prefix every server recipe uses.
run := "sudo systemd-run --quiet --wait --pty --collect --uid=" + user + " --gid=" + user \
+ " -p WorkingDirectory=" + work_dir + " -p EnvironmentFile=" + env_file \
+ " " + bin + " --config " + config
default:
@just --list --unsorted
# ---------------------------------------------------------------------------
# Server: any subcommand
# ---------------------------------------------------------------------------
# Run any daily-epub subcommand as the service user with the env file loaded.
de *args:
{{run}} "$@"
# Validate the config as `generate` would; one fact per line.
config-check:
{{run}} config check
# Run pending migrations (also automatic on every start).
db-migrate:
{{run}} db migrate
# ---------------------------------------------------------------------------
# Server: issues
# ---------------------------------------------------------------------------
# Build and publish an issue by hand (same as the timer, but in the foreground).
generate *args:
{{run}} generate "$@"
# Build everything, publish nothing, record no issue.
dry-run *args:
{{run}} generate --dry-run "$@"
# Why an article was (or was not) in the paper, from persisted telemetry.
explain *args:
{{run}} explain "$@"
# The weekly numbers: issues, ratings, retriever yield, cost, timing.
stats *args:
{{run}} stats "$@"
# ---------------------------------------------------------------------------
# Server: feeds, ratings, profile, features
# ---------------------------------------------------------------------------
# Seed /dashboard/feeds from recent aggregator-only articles.
feeds-discover days="14" limit="50":
{{run}} feeds discover --days {{days}} --limit {{limit}}
# List current explicit verdicts.
ratings-list *args:
{{run}} ratings list "$@"
# Set a verdict: `just ratings-set <article-id> <loved|good|not_for_me> [--note ...]`.
ratings-set *args:
{{run}} ratings set "$@"
# Clear a verdict: `just ratings-clear <article-id>`.
ratings-clear *args:
{{run}} ratings clear "$@"
# Regenerate the taste profile from ratings history.
profile-rebuild:
{{run}} profile rebuild
# Embed rated and published articles, then interests, into the cache.
features-backfill *args:
{{run}} features backfill "$@"
# Drop stale embeddings, old telemetry and old assessments per the retention config.
features-prune:
{{run}} features prune
# Re-poll social scores for recent entries.
backfill-social days="7":
{{run}} backfill-social --days {{days}}
# ---------------------------------------------------------------------------
# Server: users and jobs
# ---------------------------------------------------------------------------
# List dashboard users and open-session counts.
users-list:
{{run}} users list
# Add a user: `just users-add NAME [--admin]` (prompts for the password).
users-add *args:
{{run}} users add "$@"
# Change a password and revoke sessions: `just users-passwd NAME`.
users-passwd *args:
{{run}} users passwd "$@"
# Any other `users` subcommand: `just users role NAME admin`, `just users disable NAME`, ...
users *args:
{{run}} users "$@"
# Run one catalogue job in-process (generate, dry-run, profile-rebuild, features-backfill, backfill-social, features-prune, import-ratings).
job name:
{{run}} job run {{name}}
# Start a catalogue job as its systemd unit, the way the dashboard does.
job-unit name:
sudo systemctl start daily-epub-job@{{name}}
# ---------------------------------------------------------------------------
# Server: services and logs
# ---------------------------------------------------------------------------
# Status of the web server, the generate timer and its last run.
status:
systemctl status --no-pager daily-epub.service daily-epub-generate.timer daily-epub-generate.service || true
systemctl list-timers --no-pager daily-epub-generate.timer
# Restart the web server (after installing a new binary or unit).
restart:
sudo systemctl restart daily-epub.service
# Follow the web server log.
logs:
journalctl -u daily-epub.service -f
# Last N lines of the most recent generate run.
logs-generate lines="120":
journalctl -u daily-epub-generate.service -n {{lines}} --no-pager
# Follow one job unit's log: `just logs-job features-prune`.
logs-job name:
journalctl -u daily-epub-job@{{name}} -f
# ---------------------------------------------------------------------------
# Deploy (run from a checkout on the server)
# ---------------------------------------------------------------------------
# Release build, install the binary and restart the web server.
deploy: build-release
sudo install -m0755 target/release/daily-epub {{bin}}
sudo systemctl restart daily-epub.service
systemctl is-active daily-epub.service
# Install the systemd units and the polkit rule, then reload systemd.
install-units:
sudo install -m0644 systemd/daily-epub.service systemd/daily-epub-generate.service \
systemd/daily-epub-generate.timer systemd/daily-epub-job@.service /etc/systemd/system/
sudo install -m0644 systemd/50-daily-epub.rules /etc/polkit-1/rules.d/
sudo systemctl daemon-reload
# Edit the environment file (API keys and other secrets).
edit-env:
sudo -e {{env_file}}
# Edit the config file as the service user would see it.
edit-config:
sudo -e {{config}}
# ---------------------------------------------------------------------------
# Development (local checkout)
# ---------------------------------------------------------------------------
build:
cargo build
build-release:
cargo build --release
# Everything CI expects: format check, clippy, tests, and the committed CSS is current.
check: fmt-check clippy test css-check
test *args:
cargo test "$@"
clippy:
cargo clippy --all-targets
fmt:
cargo fmt
fmt-check:
cargo fmt --check
# Rebuild src/web/static/app.css from the Tailwind source (commit the result).
css:
npm run css
# Fail if the committed app.css is out of date.
css-check:
npm run css:check
# Rebuild app.css on every template change.
css-watch:
npm run css:watch
# Seed a throwaway dev database under ./dev (admin/adminpassword123, reader/readerpassword123).
dev-seed:
cargo run --example seed_dev_db -- ./dev
# Serve the dev database on http://127.0.0.1:3599.
dev-serve:
DAILY_EPUB_SERVER__HMAC_SECRET=$(head -c 48 /dev/urandom | base64) \
cargo run -- --config ./dev/config.toml serve
# Run any subcommand against the local dev config: `just dev config check`.
dev *args:
cargo run -- --config ./dev/config.toml "$@"
# Replay extraction + images over published EPUBs: `just image-audit ~/books/*.epub`.
image-audit *args:
cargo run --release --example image_audit -- --cache /tmp/pagecache "$@"
+27
View File
@@ -0,0 +1,27 @@
-- Feed discovery: proposed Miniflux subscriptions found behind aggregator-only
-- articles (feed discovery plan §4 step 1).
CREATE TABLE feed_candidates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
feed_url TEXT NOT NULL UNIQUE,
host TEXT NOT NULL,
title TEXT,
status TEXT NOT NULL CHECK (status IN ('candidate', 'added', 'dismissed')),
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
miniflux_feed_id INTEGER,
decided_at TEXT
);
CREATE INDEX idx_feed_candidates_status_host ON feed_candidates(status, host);
CREATE TABLE feed_candidate_articles (
candidate_id INTEGER NOT NULL REFERENCES feed_candidates(id) ON DELETE CASCADE,
article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
PRIMARY KEY (candidate_id, article_id)
);
CREATE TABLE feed_discovery_hosts (
host TEXT PRIMARY KEY,
checked_at TEXT NOT NULL,
candidates INTEGER NOT NULL
);
+41
View File
@@ -84,6 +84,8 @@ pub struct Config {
pub server: ServerConfig, pub server: ServerConfig,
pub bookorbit: BookorbitConfig, pub bookorbit: BookorbitConfig,
pub mail: MailConfig, pub mail: MailConfig,
/// `[discovery]` — propose new feed subscriptions from aggregator hits.
pub discovery: DiscoveryConfig,
} }
impl Default for Config { impl Default for Config {
@@ -110,6 +112,7 @@ impl Default for Config {
server: ServerConfig::default(), server: ServerConfig::default(),
bookorbit: BookorbitConfig::default(), bookorbit: BookorbitConfig::default(),
mail: MailConfig::default(), mail: MailConfig::default(),
discovery: DiscoveryConfig::default(),
} }
} }
} }
@@ -823,6 +826,44 @@ impl Default for MailConfig {
} }
} }
/// `[discovery]` — feed discovery from aggregator-only articles (feed
/// discovery plan §3).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields, default)]
pub struct DiscoveryConfig {
/// Whether the discovery stage runs during `generate`.
pub enabled: bool,
/// How many not-yet-checked hosts one run may look up in Miniflux.
pub max_lookups_per_run: usize,
/// Hosts never looked up (aggregators, code hosts, social networks).
pub skip_hosts: Vec<String>,
}
impl Default for DiscoveryConfig {
fn default() -> Self {
Self {
enabled: true,
max_lookups_per_run: 30,
skip_hosts: [
"news.ycombinator.com",
"lobste.rs",
"reddit.com",
"github.com",
"gist.github.com",
"x.com",
"twitter.com",
"youtube.com",
"en.wikipedia.org",
"arxiv.org",
"docs.google.com",
]
.into_iter()
.map(String::from)
.collect(),
}
}
}
impl MailConfig { impl MailConfig {
/// Whether mail is enabled with all fields required for SMTP delivery. /// Whether mail is enabled with all fields required for SMTP delivery.
pub fn is_active(&self) -> bool { pub fn is_active(&self) -> bool {
+1577
View File
File diff suppressed because it is too large. Load diff
+1
View File
@@ -19,6 +19,7 @@ pub mod config;
pub mod curate; pub mod curate;
pub mod db; pub mod db;
pub mod dedupe; pub mod dedupe;
pub mod discovery;
pub mod epub; pub mod epub;
pub mod extract; pub mod extract;
pub mod html; pub mod html;
+60 -1
View File
@@ -17,7 +17,7 @@ use daily_epub::db::Db;
use daily_epub::pipeline::{self, GenerateOptions, GenerateOutcome}; use daily_epub::pipeline::{self, GenerateOptions, GenerateOutcome};
use daily_epub::report::{RunReport, VOYAGE_PROVIDER}; use daily_epub::report::{RunReport, VOYAGE_PROVIDER};
use daily_epub::types::{ArticleId, Vote}; use daily_epub::types::{ArticleId, Vote};
use daily_epub::{curate, http, imports, jobs, lock, rate, server, social}; use daily_epub::{curate, discovery, http, imports, jobs, lock, rate, server, social};
/// A personalized daily newspaper, delivered as an EPUB. /// A personalized daily newspaper, delivered as an EPUB.
#[derive(Debug, Parser)] #[derive(Debug, Parser)]
@@ -61,6 +61,9 @@ enum Command {
/// Manage dashboard users without taking the pipeline run lock. /// Manage dashboard users without taking the pipeline run lock.
#[command(subcommand)] #[command(subcommand)]
Users(UsersCommand), Users(UsersCommand),
/// Feed subscription candidates (feed discovery plan §4 step 6).
#[command(subcommand)]
Feeds(FeedsCommand),
/// Operator jobs (what `daily-epub-job@<name>.service` runs). /// Operator jobs (what `daily-epub-job@<name>.service` runs).
#[command(subcommand)] #[command(subcommand)]
Job(JobCommand), Job(JobCommand),
@@ -76,6 +79,23 @@ enum JobCommand {
}, },
} }
#[derive(Debug, Subcommand)]
enum FeedsCommand {
/// Look for feeds behind recent aggregator-only articles and record the
/// ones not already subscribed as candidates for `/dashboard/feeds`.
Discover(FeedsDiscoverArgs),
}
#[derive(Debug, clap::Args)]
struct FeedsDiscoverArgs {
/// How far back to look for articles.
#[arg(long, default_value_t = 14)]
days: i64,
/// How many not-yet-checked hosts to look up.
#[arg(long, default_value_t = 200)]
limit: usize,
}
#[derive(Debug, Subcommand)] #[derive(Debug, Subcommand)]
enum UsersCommand { enum UsersCommand {
/// Add a user. /// Add a user.
@@ -381,6 +401,10 @@ async fn main() -> Result<()> {
let db = Db::open_and_migrate(&config.database_path).await?; let db = Db::open_and_migrate(&config.database_path).await?;
cmd_users(&db, command).await?; cmd_users(&db, command).await?;
} }
Command::Feeds(FeedsCommand::Discover(args)) => {
let db = Db::open_and_migrate(&config.database_path).await?;
println!("{}", cmd_feeds_discover(&config, &db, args).await?);
}
Command::Job(JobCommand::Run { name }) => { Command::Job(JobCommand::Run { name }) => {
let Some(job) = jobs::Job::parse(&name) else { let Some(job) = jobs::Job::parse(&name) else {
eprintln!("unknown job {name:?}; the catalogue is:"); eprintln!("unknown job {name:?}; the catalogue is:");
@@ -411,6 +435,8 @@ async fn main() -> Result<()> {
fn lock_holder(command: &Command) -> Option<&'static str> { fn lock_holder(command: &Command) -> Option<&'static str> {
match command { match command {
Command::Generate(_) => Some("generate"), Command::Generate(_) => Some("generate"),
// Writes the same tables the discovery stage of a run writes.
Command::Feeds(FeedsCommand::Discover(_)) => Some("feeds discover"),
Command::Profile(ProfileCommand::Rebuild) => Some("profile rebuild"), Command::Profile(ProfileCommand::Rebuild) => Some("profile rebuild"),
Command::Features(FeaturesCommand::Backfill(_)) => Some("features backfill"), Command::Features(FeaturesCommand::Backfill(_)) => Some("features backfill"),
Command::BackfillSocial(_) => Some("backfill-social"), Command::BackfillSocial(_) => Some("backfill-social"),
@@ -430,6 +456,39 @@ fn lock_holder(command: &Command) -> Option<&'static str> {
} }
} }
/// `daily-epub feeds discover` — the same stage `generate` runs, over the last
/// `--days` of articles, so the dashboard page is useful before tomorrow's run.
async fn cmd_feeds_discover(config: &Config, db: &Db, args: FeedsDiscoverArgs) -> Result<String> {
let since = jiff::Timestamp::now()
.checked_sub(jiff::Span::new().hours(args.days.max(0).saturating_mul(24)))
.unwrap_or(jiff::Timestamp::UNIX_EPOCH);
let articles = discovery::articles_since(db, since).await?;
let http = http::build_client(http::DEFAULT_TIMEOUT).context("building http client")?;
let client = daily_epub::miniflux::MinifluxClient::new(&config.miniflux, http.clone())
.context("constructing the miniflux client")?;
let feeds = client
.feed_map()
.await
.context("loading the miniflux feed list")?;
let mut cfg = config.discovery.clone();
cfg.max_lookups_per_run = args.limit;
let summary = discovery::run(
db,
&client,
&http,
&cfg,
&articles,
&feeds,
jiff::Timestamp::now(),
)
.await?;
Ok(format!(
"{} articles since {}: {summary}",
articles.len(),
since
))
}
async fn cmd_users(db: &Db, command: UsersCommand) -> Result<()> { async fn cmd_users(db: &Db, command: UsersCommand) -> Result<()> {
use daily_epub::web::users; use daily_epub::web::users;
match command { match command {
+230 -21
View File
@@ -1,13 +1,16 @@
//! Miniflux API client (spec §3.1). //! Miniflux API client (spec §3.1).
//! //!
//! Reads only: entries are fetched with `published_after` inside the lookback //! Almost reads only: entries are fetched with `published_after` inside the
//! window **regardless of read/unread status**, and read state is never mutated //! lookback window **regardless of read/unread status**, and read state is never
//! so normal reader usage is undisturbed. //! mutated so normal reader usage is undisturbed. The single writing call is
//! [`MinifluxClient::create_feed`], which the dashboard's feed-discovery page
//! uses to subscribe to a candidate the operator picked (feed discovery plan
//! §4 step 2); the pipeline never calls it.
use std::collections::HashMap; use std::collections::HashMap;
use jiff::Timestamp; use jiff::Timestamp;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize, de::DeserializeOwned};
use crate::config::MinifluxConfig; use crate::config::MinifluxConfig;
use crate::http::{RetryPolicy, is_retryable}; use crate::http::{RetryPolicy, is_retryable};
@@ -24,11 +27,13 @@ pub enum MinifluxError {
MissingApiKey, MissingApiKey,
#[error("miniflux request failed: {0}")] #[error("miniflux request failed: {0}")]
Http(#[from] reqwest::Error), Http(#[from] reqwest::Error),
#[error("miniflux returned {status} for {path}: {body}")] #[error("miniflux returned {status} for {path}: {message}")]
Status { Status {
status: u16, status: u16,
path: String, path: String,
body: String, /// `error_message` when the body was Miniflux's JSON error envelope,
/// else the truncated body — what a dashboard flash should show.
message: String,
}, },
#[error("could not parse miniflux response for {path}: {source}")] #[error("could not parse miniflux response for {path}: {source}")]
Decode { Decode {
@@ -40,6 +45,46 @@ pub enum MinifluxError {
type Result<T> = std::result::Result<T, MinifluxError>; type Result<T> = std::result::Result<T, MinifluxError>;
impl MinifluxError {
/// Build a [`MinifluxError::Status`], pulling `error_message` out of
/// Miniflux's JSON error envelope when the body has one (§4 step 2).
fn status(status: u16, path: &str, body: &str) -> Self {
#[derive(Deserialize)]
struct ErrorBody {
error_message: String,
}
let body: String = body.chars().take(300).collect();
let message = serde_json::from_str::<ErrorBody>(&body)
.map(|e| e.error_message)
.unwrap_or_else(|_| body.clone());
MinifluxError::Status {
status,
path: path.to_string(),
message,
}
}
/// True for the `400 This feed already exists.` a duplicate `POST /v1/feeds`
/// returns — the dashboard flips the row to `added` on it instead of
/// treating it as a failure.
pub fn is_duplicate_feed(&self) -> bool {
matches!(
self,
MinifluxError::Status { status: 400, message, .. }
if message.trim().eq_ignore_ascii_case("This feed already exists.")
)
}
}
/// Network failures and 5xx/429 responses are worth another attempt (§3.1).
fn retryable(error: &MinifluxError) -> bool {
match error {
MinifluxError::Http(e) => is_retryable(e),
MinifluxError::Status { status, .. } => *status >= 500 || *status == 429,
_ => false,
}
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Wire types (only the fields §3.1 lists as used) // Wire types (only the fields §3.1 lists as used)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -106,6 +151,27 @@ pub struct EntriesResponse {
pub entries: Vec<MinifluxEntry>, pub entries: Vec<MinifluxEntry>,
} }
/// `POST /v1/discover` element — a *lead*, not a verified feed: Miniflux
/// returns well-known paths that merely answered 200 (plan §2), so every one is
/// validated before it becomes a candidate.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Discovered {
#[serde(default)]
pub url: String,
/// Miniflux substitutes the URL when the `<link>` tag has no title, which is
/// how a well-known-path guess is told apart from a real link-tag hit.
#[serde(default)]
pub title: String,
#[serde(default, rename = "type")]
pub kind: String,
}
/// `POST /v1/feeds` response.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct CreatedFeed {
pub feed_id: i64,
}
/// Feed metadata joined onto every entry we persist (§3.1). /// Feed metadata joined onto every entry we persist (§3.1).
#[derive(Debug, Clone, PartialEq)] #[derive(Debug, Clone, PartialEq)]
pub struct FeedMeta { pub struct FeedMeta {
@@ -247,14 +313,7 @@ impl MinifluxClient {
let url = self.url(path); let url = self.url(path);
let body = self let body = self
.retry .retry
.run( .run(&format!("GET {path}"), retryable, || async {
&format!("GET {path}"),
|e: &MinifluxError| match e {
MinifluxError::Http(e) => is_retryable(e),
MinifluxError::Status { status, .. } => *status >= 500 || *status == 429,
_ => false,
},
|| async {
let resp = self let resp = self
.http .http
.get(&url) .get(&url)
@@ -266,15 +325,10 @@ impl MinifluxClient {
let status = resp.status(); let status = resp.status();
let text = resp.text().await?; let text = resp.text().await?;
if !status.is_success() { if !status.is_success() {
return Err(MinifluxError::Status { return Err(MinifluxError::status(status.as_u16(), path, &text));
status: status.as_u16(),
path: path.to_string(),
body: text.chars().take(300).collect(),
});
} }
Ok(text) Ok(text)
}, })
)
.await?; .await?;
serde_json::from_str(&body).map_err(|source| MinifluxError::Decode { serde_json::from_str(&body).map_err(|source| MinifluxError::Decode {
path: path.to_string(), path: path.to_string(),
@@ -282,6 +336,74 @@ impl MinifluxClient {
}) })
} }
/// One `POST path` with a JSON body, no retry.
async fn post_once<B: Serialize>(&self, path: &str, body: &B) -> Result<String> {
let resp = self
.http
.post(self.url(path))
.header("X-Auth-Token", &self.api_key)
.header("Accept", "application/json")
.json(body)
.send()
.await?;
let status = resp.status();
let text = resp.text().await?;
if !status.is_success() {
return Err(MinifluxError::status(status.as_u16(), path, &text));
}
Ok(text)
}
/// POST `path` with the auth header, mirroring [`Self::get_json`].
///
/// `retry` is false for calls that are not safe to repeat: a retried
/// `POST /v1/feeds` whose response was merely lost would subscribe twice.
async fn post_json<B: Serialize, T: DeserializeOwned>(
&self,
path: &str,
body: &B,
retry: bool,
) -> Result<T> {
let text = if retry {
self.retry
.run(&format!("POST {path}"), retryable, || {
self.post_once(path, body)
})
.await?
} else {
self.post_once(path, body).await?
};
serde_json::from_str(&text).map_err(|source| MinifluxError::Decode {
path: path.to_string(),
source,
})
}
/// `POST /v1/discover` — the feed(s) Miniflux can find behind a page URL
/// (feed discovery plan §4 step 2). Retried: it changes nothing.
pub async fn discover(&self, url: &str) -> Result<Vec<Discovered>> {
self.post_json("/discover", &serde_json::json!({ "url": url }), true)
.await
}
/// `GET /v1/categories` — the Add form's category picker.
pub async fn categories(&self) -> Result<Vec<MinifluxCategory>> {
self.get_json("/categories", &[]).await
}
/// `POST /v1/feeds` — subscribe to `feed_url` in `category_id`, returning
/// the new feed id. Never retried (a repeat can double-subscribe).
pub async fn create_feed(&self, feed_url: &str, category_id: i64) -> Result<i64> {
let created: CreatedFeed = self
.post_json(
"/feeds",
&serde_json::json!({ "feed_url": feed_url, "category_id": category_id }),
false,
)
.await?;
Ok(created.feed_id)
}
/// `GET /v1/feeds` — once per run (§3.1). /// `GET /v1/feeds` — once per run (§3.1).
pub async fn feeds(&self) -> Result<Vec<MinifluxFeed>> { pub async fn feeds(&self) -> Result<Vec<MinifluxFeed>> {
self.get_json("/feeds", &[]).await self.get_json("/feeds", &[]).await
@@ -483,6 +605,93 @@ mod tests {
assert_eq!(entries[1].category, None); assert_eq!(entries[1].category, None);
} }
/// Observed live for an article URL whose page carries `<link rel=alternate>`
/// tags: the same feed offered twice under one title (plan §2).
const DISCOVER_LINK_TAGS_JSON: &str = r#"[
{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.xml","type":"atom"},
{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.json","type":"json"}
]"#;
/// Observed live for a site that answers 200 for any path: nine unverified
/// well-known-path guesses, each with `title == url` (plan §2).
const DISCOVER_GUESSES_JSON: &str = r#"[
{"title":"https://zombo.com/atom.xml","url":"https://zombo.com/atom.xml","type":"atom"},
{"title":"https://zombo.com/feed.atom","url":"https://zombo.com/feed.atom","type":"atom"},
{"title":"https://zombo.com/feed.xml","url":"https://zombo.com/feed.xml","type":"atom"},
{"title":"https://zombo.com/feed/","url":"https://zombo.com/feed/","type":"atom"},
{"title":"https://zombo.com/index.rss","url":"https://zombo.com/index.rss","type":"rss"},
{"title":"https://zombo.com/index.xml","url":"https://zombo.com/index.xml","type":"rss"},
{"title":"https://zombo.com/rss.xml","url":"https://zombo.com/rss.xml","type":"rss"},
{"title":"https://zombo.com/rss/","url":"https://zombo.com/rss/","type":"rss"},
{"title":"https://zombo.com/rss/feed.xml","url":"https://zombo.com/rss/feed.xml","type":"rss"}
]"#;
const CATEGORIES_JSON: &str = r#"[
{"id": 1, "user_id": 1, "title": "Tech", "hide_globally": false},
{"id": 4, "user_id": 1, "title": "Long reads", "hide_globally": true}
]"#;
#[test]
fn deserializes_discover_results() {
let hits: Vec<Discovered> = serde_json::from_str(DISCOVER_LINK_TAGS_JSON).unwrap();
assert_eq!(hits.len(), 2);
assert_eq!(hits[0].title, "blog.philz.dev");
assert_eq!(hits[0].url, "https://blog.philz.dev/feed/feed.xml");
assert_eq!(hits[0].kind, "atom");
assert_eq!(hits[1].kind, "json");
let guesses: Vec<Discovered> = serde_json::from_str(DISCOVER_GUESSES_JSON).unwrap();
assert_eq!(guesses.len(), 9);
assert!(guesses.iter().all(|g| g.title == g.url));
}
#[test]
fn deserializes_categories_and_created_feed() {
let categories: Vec<MinifluxCategory> = serde_json::from_str(CATEGORIES_JSON).unwrap();
assert_eq!(categories.len(), 2);
assert_eq!(categories[1].id, 4);
assert_eq!(categories[1].title, "Long reads");
let created: CreatedFeed = serde_json::from_str(r#"{"feed_id": 123}"#).unwrap();
assert_eq!(created.feed_id, 123);
}
#[test]
fn status_errors_carry_the_error_message() {
let duplicate = MinifluxError::status(
400,
"/feeds",
r#"{"error_message":"This feed already exists."}"#,
);
assert!(duplicate.is_duplicate_feed());
assert_eq!(
duplicate.to_string(),
"miniflux returned 400 for /feeds: This feed already exists."
);
let fetcher = MinifluxError::status(
502,
"/discover",
r#"{"error_message":"fetcher: bad gateway (502 status code)"}"#,
);
assert!(!fetcher.is_duplicate_feed());
assert!(retryable(&fetcher));
match fetcher {
MinifluxError::Status { message, .. } => {
assert_eq!(message, "fetcher: bad gateway (502 status code)");
}
other => panic!("unexpected error {other:?}"),
}
// A non-JSON body falls back to the truncated body itself.
let html = MinifluxError::status(404, "/discover", "<html>nope</html>");
assert!(!retryable(&html));
assert_eq!(
html.to_string(),
"miniflux returned 404 for /discover: <html>nope</html>"
);
}
#[test] #[test]
fn requires_an_api_key() { fn requires_an_api_key() {
let http = crate::http::build_client(crate::http::DEFAULT_TIMEOUT).unwrap(); let http = crate::http::build_client(crate::http::DEFAULT_TIMEOUT).unwrap();
+27 -1
View File
@@ -46,7 +46,7 @@ use crate::types::{
Article, ArticleId, Artifact, BehindThePaper, Candidate, Colophon, Edition, Issue, IssueMeta, Article, ArticleId, Artifact, BehindThePaper, Candidate, Colophon, Edition, Issue, IssueMeta,
Lineup, Models, TokenUsage, reading_minutes, Lineup, Models, TokenUsage, reading_minutes,
}; };
use crate::{comments, dedupe, epub, http, miniflux, publish, social, world}; use crate::{comments, dedupe, discovery, epub, http, miniflux, publish, social, world};
/// One `generate` invocation's inputs — the CLI flags, already parsed (§2). /// One `generate` invocation's inputs — the CLI flags, already parsed (§2).
#[derive(Debug, Clone, Default)] #[derive(Debug, Clone, Default)]
@@ -449,6 +449,32 @@ async fn run_stages(
report.counts.social_hits = enricher.enrich_all(&mut articles).await as i64; report.counts.social_hits = enricher.enrich_all(&mut articles).await as i64;
report.timings.record("social", elapsed_ms(stage)); report.timings.record("social", elapsed_ms(stage));
// --- Stage 5b: feed discovery (feed discovery plan §4) — best effort ---
// Runs here because it needs the real article ids stage 4 minted and the
// subscription map stage 1 already loaded, and because `articles` is moved
// into hygiene next.
if config.discovery.enabled {
let stage = Timestamp::now();
match discovery::run(
db,
&client,
&http,
&config.discovery,
&articles,
&feeds,
Timestamp::now(),
)
.await
{
Ok(summary) => {
tracing::info!(%summary, "feed discovery");
report.counts.feed_candidates_new = summary.candidates_new as i64;
}
Err(error) => report.warn(format!("feed discovery failed: {error:#}")),
}
report.timings.record("discovery", elapsed_ms(stage));
}
// --- Stage 6: hygiene, embeddings, and cheap signals (§8.1, §9) --- // --- Stage 6: hygiene, embeddings, and cheap signals (§8.1, §9) ---
let stage = Timestamp::now(); let stage = Timestamp::now();
let mut personalized = admit::hygiene( let mut personalized = admit::hygiene(
+4
View File
@@ -67,6 +67,10 @@ pub struct StageCounts {
pub excerpt_only: i64, pub excerpt_only: i64,
/// Social lookups that returned a hit (§3.4). /// Social lookups that returned a hit (§3.4).
pub social_hits: i64, pub social_hits: i64,
/// Feed candidates recorded for the first time by the discovery stage
/// (feed discovery plan §4 step 3).
#[serde(default)]
pub feed_candidates_new: i64,
/// Articles passing hygiene and eligible for personalized signals. /// Articles passing hygiene and eligible for personalized signals.
pub eligible: i64, pub eligible: i64,
/// Eligible articles with a valid embedding. /// Eligible articles with a valid embedding.
+639
View File
@@ -0,0 +1,639 @@
//! Dashboard: feed candidates (`/dashboard/feeds`, feed discovery plan §4
//! step 5).
//!
//! The discovery stage records the feeds behind aggregator-only articles in
//! `feed_candidates`; this page ranks the undecided ones by how likely the
//! operator is to enjoy them ([`discovery::score`]) and offers **Add** (which
//! subscribes through `POST /v1/feeds`) and **Dismiss**.
//!
//! Ranking is global, so the candidate view loads every undecided row, scores
//! it in Rust and paginates afterwards (plan §4 step 4). The decided views are
//! ordinary SQL pages.
use std::cmp::Ordering;
use std::collections::{HashMap, HashSet};
use askama::Template;
use axum::Router;
use axum::extract::{Extension, Form, Path, Query, State};
use axum::response::{IntoResponse, Redirect, Response};
use axum::routing::{get, post};
use axum_login::tower_sessions::Session;
use jiff::Timestamp;
use serde::Deserialize;
use sqlx::Row as _;
use super::jobs::set_flash;
use super::{Pager, allow_listed, db_err, dynamic_query, fmt_stored_time, page_number};
use crate::config::Config;
use crate::db::Db;
use crate::discovery::{self, ArticleEvidence, Candidate};
use crate::miniflux::{MinifluxCategory, MinifluxClient};
use crate::server::AppState;
use crate::types::ArticleId;
use crate::web::session::{AuthSession, Viewer};
use crate::web::{Html, Page, Pagination, WebError, take_flash};
const CANDIDATES_PER_PAGE: u32 = 50;
/// `feed_candidates.status` values, allow-listed for `?status=`.
const STATUSES: [&str; 3] = ["candidate", "added", "dismissed"];
/// Articles named in the "Why" column.
const WHY_ARTICLES: usize = 3;
/// `kv` key remembering the category the last Add used (plan §3).
const LAST_CATEGORY_KEY: &str = "feed_discovery_last_category";
const PATH: &str = "/dashboard/feeds";
/// Routes contributed by this page group (merged by `dashboard::router`).
pub fn routes() -> Router<AppState> {
Router::new()
.route(PATH, get(index))
.route("/dashboard/feeds/{id}/add", post(add))
.route("/dashboard/feeds/{id}/dismiss", post(dismiss))
}
#[derive(Debug, Default, Deserialize)]
struct FeedsQuery {
status: Option<String>,
page: Option<u32>,
}
#[derive(Debug, Deserialize)]
struct AddForm {
category_id: i64,
}
/// One linked article, as named in the "Why" column.
#[derive(Debug)]
struct WhyArticle {
id: ArticleId,
title: String,
}
/// One table row.
#[derive(Debug)]
struct FeedRow {
id: i64,
/// The shrunk mean on 0–100, rounded.
score: i64,
feed_url: String,
/// The candidate's title, or its feed URL when it has none.
label: String,
host: String,
interests: Vec<String>,
articles: Vec<WhyArticle>,
article_count: usize,
first_seen: String,
last_seen: String,
/// `last_seen` as a short date for the column; the full first/last
/// timestamps sit in the cell's tooltip.
seen: String,
decided_at: String,
/// `{miniflux.base_url}/feeds/{id}` for a candidate we subscribed to.
miniflux_href: Option<String>,
}
#[derive(Template)]
#[template(path = "dashboard/feeds.html")]
struct FeedsTemplate {
page: Page,
status: String,
rows: Vec<FeedRow>,
candidate_count: i64,
added_count: i64,
dismissed_count: i64,
categories: Vec<MinifluxCategory>,
selected_category: i64,
/// Why the category picker is empty, when it is; Add is disabled then.
categories_error: Option<String>,
pager: Pager,
}
async fn index(
State(state): State<AppState>,
auth: AuthSession,
Extension(session): Extension<Session>,
Query(query): Query<FeedsQuery>,
) -> Result<Response, WebError> {
let viewer = auth.user().await.map(Viewer::from);
let config = state.config();
let db = &state.db;
let status = allow_listed(query.status.as_deref(), &STATUSES).unwrap_or("candidate");
let page_no = page_number(query.page);
// Candidates are ranked globally, so they are all loaded, scored, sorted
// and only then paginated (plan §4 step 4).
let (candidates, total, ranked_evidence) = if status == "candidate" {
let all = discovery::all_candidates(db).await?;
let evidence = load_evidence(db, &config, &all).await?;
let mut ranked: Vec<(Candidate, f64)> = all
.into_iter()
.map(|candidate| {
let score = discovery::score(&evidence_of(&candidate, &evidence));
(candidate, score)
})
.collect();
ranked.sort_by(|a, b| {
b.1.partial_cmp(&a.1)
.unwrap_or(Ordering::Equal)
.then_with(|| b.0.article_ids.len().cmp(&a.0.article_ids.len()))
.then_with(|| b.0.last_seen.cmp(&a.0.last_seen))
});
let total = ranked.len() as i64;
let offset = (page_no.saturating_sub(1) as usize) * CANDIDATES_PER_PAGE as usize;
let page_rows: Vec<Candidate> = ranked
.into_iter()
.skip(offset)
.take(CANDIDATES_PER_PAGE as usize)
.map(|(candidate, _)| candidate)
.collect();
(page_rows, total, Some(evidence))
} else {
let (candidates, total) = discovery::list(db, status, page_no, CANDIDATES_PER_PAGE).await?;
(candidates, total, None)
};
let evidence = match ranked_evidence {
Some(evidence) => evidence,
None => load_evidence(db, &config, &candidates).await?,
};
let titles = article_titles(db, &candidates).await?;
let base_url = config.miniflux.base_url.trim_end_matches('/').to_string();
let rows: Vec<FeedRow> = candidates
.iter()
.map(|candidate| row(candidate, &evidence, &titles, &config, &base_url))
.collect();
// The picker is only ever used by the candidate view, so only it pays for
// the categories call (plan §3).
let (categories, categories_error) = if status == "candidate" {
categories(&config).await
} else {
(Vec::new(), None)
};
let selected_category = selected_category(db, &categories).await?;
let pager = Pager::new(
Pagination {
page: page_no,
per_page: CANDIDATES_PER_PAGE,
total,
},
PATH,
&[("status", Some(status.to_string()))],
);
let mut page = Page::new("Feeds", viewer, "feeds");
page.flash = take_flash(&session).await?;
Ok(Html(FeedsTemplate {
page,
status: status.to_string(),
rows,
candidate_count: discovery::count(db, "candidate").await?,
added_count: discovery::count(db, "added").await?,
dismissed_count: discovery::count(db, "dismissed").await?,
categories,
selected_category,
categories_error,
pager,
})
.into_response())
}
async fn add(
State(state): State<AppState>,
Extension(session): Extension<Session>,
Path(id): Path<i64>,
Form(form): Form<AddForm>,
) -> Result<Response, WebError> {
let config = state.config();
let db = &state.db;
let now = Timestamp::now();
let Some(candidate) = discovery::candidate(db, id).await? else {
set_flash(&session, "error", format!("No feed candidate {id}.")).await?;
return Ok(Redirect::to(PATH).into_response());
};
let label = candidate
.title
.clone()
.unwrap_or_else(|| candidate.feed_url.clone());
let Some(client) = client(&config) else {
set_flash(
&session,
"error",
"Miniflux API key is not configured".into(),
)
.await?;
return Ok(Redirect::to(PATH).into_response());
};
match client
.create_feed(&candidate.feed_url, form.category_id)
.await
{
Ok(feed_id) => {
discovery::set_status(db, id, "added", Some(feed_id), now).await?;
db.kv_set(LAST_CATEGORY_KEY, &form.category_id.to_string())
.await?;
set_flash(&session, "success", format!("Added {label}.")).await?;
}
// Already subscribed by other means: the operator's intent is met, so
// the row is decided too (plan §2).
Err(error) if error.is_duplicate_feed() => {
discovery::set_status(db, id, "added", None, now).await?;
set_flash(&session, "success", format!("{label}: {error}")).await?;
}
Err(error) => {
set_flash(&session, "error", format!("{label}: {error}")).await?;
}
}
Ok(Redirect::to(PATH).into_response())
}
async fn dismiss(
State(state): State<AppState>,
Extension(session): Extension<Session>,
Path(id): Path<i64>,
) -> Result<Response, WebError> {
let db = &state.db;
let Some(candidate) = discovery::candidate(db, id).await? else {
set_flash(&session, "error", format!("No feed candidate {id}.")).await?;
return Ok(Redirect::to(PATH).into_response());
};
discovery::set_status(db, id, "dismissed", None, Timestamp::now()).await?;
let label = candidate.title.unwrap_or(candidate.feed_url);
set_flash(&session, "success", format!("Dismissed {label}.")).await?;
Ok(Redirect::to(PATH).into_response())
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
/// A Miniflux client from `[miniflux]`, or `None` when there is no API key.
fn client(config: &Config) -> Option<MinifluxClient> {
let http = crate::http::build_client(crate::http::DEFAULT_TIMEOUT).ok()?;
MinifluxClient::new(&config.miniflux, http).ok()
}
/// `GET /v1/categories` for the Add picker; the error is shown as a notice on
/// the page (not a flash: nothing was submitted) and disables Add.
async fn categories(config: &Config) -> (Vec<MinifluxCategory>, Option<String>) {
let Some(client) = client(config) else {
return (
Vec::new(),
Some("Miniflux API key is not configured, so feeds cannot be added.".into()),
);
};
match client.categories().await {
Ok(categories) => (categories, None),
Err(error) => (
Vec::new(),
Some(format!("Could not load Miniflux categories: {error}")),
),
}
}
/// The category the last Add used, when it still exists; else the first one.
async fn selected_category(db: &Db, categories: &[MinifluxCategory]) -> Result<i64, WebError> {
let last = db
.kv_get(LAST_CATEGORY_KEY)
.await?
.and_then(|value| value.trim().parse::<i64>().ok())
.filter(|id| categories.iter().any(|category| category.id == *id));
Ok(last
.or_else(|| categories.first().map(|category| category.id))
.unwrap_or_default())
}
fn union_ids(candidates: &[Candidate]) -> Vec<ArticleId> {
let mut seen: HashSet<ArticleId> = HashSet::new();
candidates
.iter()
.flat_map(|candidate| candidate.article_ids.iter().copied())
.filter(|id| seen.insert(*id))
.collect()
}
/// Ranking evidence for every article linked to `candidates`, in one query.
async fn load_evidence(
db: &Db,
config: &Config,
candidates: &[Candidate],
) -> Result<HashMap<ArticleId, ArticleEvidence>, WebError> {
Ok(discovery::load_evidence(
db,
&union_ids(candidates),
config.curation.ranking.rating_lookback_days,
)
.await?)
}
/// Titles for every linked article, including the ones with no evidence (which
/// the "Why" column still lists, last).
async fn article_titles(
db: &Db,
candidates: &[Candidate],
) -> Result<HashMap<ArticleId, String>, WebError> {
let ids = union_ids(candidates);
let mut titles = HashMap::new();
for chunk in ids.chunks(500) {
let placeholders = vec!["?"; chunk.len()].join(", ");
let mut query = dynamic_query(format!(
"SELECT id, COALESCE(title, '') AS title FROM articles WHERE id IN ({placeholders})"
));
for id in chunk {
query = query.bind(*id);
}
for row in query.fetch_all(db.pool()).await.map_err(db_err)? {
titles.insert(row.get("id"), row.get("title"));
}
}
Ok(titles)
}
/// `Sep 7` for the current year, `Sep 7, 2025` otherwise, in the configured
/// time zone; the raw string when it does not parse.
fn compact_date(raw: &str, config: &Config) -> String {
let (Ok(timestamp), Ok(tz)) = (raw.parse::<Timestamp>(), config.tz()) else {
return raw.to_string();
};
let zoned = timestamp.to_zoned(tz.clone());
let format = if zoned.year() == Timestamp::now().to_zoned(tz).year() {
"%b %-d"
} else {
"%b %-d, %Y"
};
zoned.strftime(format).to_string()
}
fn evidence_of(
candidate: &Candidate,
evidence: &HashMap<ArticleId, ArticleEvidence>,
) -> Vec<ArticleEvidence> {
candidate
.article_ids
.iter()
.filter_map(|id| evidence.get(id).cloned())
.collect()
}
fn row(
candidate: &Candidate,
evidence: &HashMap<ArticleId, ArticleEvidence>,
titles: &HashMap<ArticleId, String>,
config: &Config,
base_url: &str,
) -> FeedRow {
let mut scored = evidence_of(candidate, evidence);
let score = discovery::score(&scored);
let interests = discovery::why(&scored);
// Best evidence first; the articles nothing is known about come last, in
// title order.
scored.sort_by(|a, b| {
b.value
.partial_cmp(&a.value)
.unwrap_or(Ordering::Equal)
.then_with(|| a.title.cmp(&b.title))
});
let mut unscored: Vec<WhyArticle> = candidate
.article_ids
.iter()
.filter(|id| !evidence.contains_key(id))
.map(|id| WhyArticle {
id: *id,
title: titles.get(id).cloned().unwrap_or_default(),
})
.collect();
unscored.sort_by(|a, b| a.title.cmp(&b.title));
let articles: Vec<WhyArticle> = scored
.into_iter()
.map(|item| WhyArticle {
id: item.article_id,
title: item.title,
})
.chain(unscored)
.take(WHY_ARTICLES)
.collect();
FeedRow {
id: candidate.id,
score: score.round() as i64,
feed_url: candidate.feed_url.clone(),
label: candidate
.title
.clone()
.unwrap_or_else(|| candidate.feed_url.clone()),
host: candidate.host.clone(),
interests,
articles,
article_count: candidate.article_ids.len(),
first_seen: fmt_stored_time(Some(&candidate.first_seen), config),
last_seen: fmt_stored_time(Some(&candidate.last_seen), config),
seen: compact_date(&candidate.last_seen, config),
decided_at: fmt_stored_time(candidate.decided_at.as_deref(), config),
miniflux_href: candidate
.miniflux_feed_id
.map(|feed_id| format!("{base_url}/feeds/{feed_id}")),
}
}
#[cfg(test)]
mod tests {
use axum::Json;
use axum::body::Body;
use axum::http::{Method, Request, StatusCode, header};
use serde_json::json;
use tower::ServiceExt;
use super::*;
use crate::server::router;
use crate::web::dashboard::tests::{
Seed, app_with_users, assert_admin_only, get, login_cookie, response_text, seed,
};
/// A candidate on `host` linked to `article_ids`.
async fn candidate(db: &Db, title: &str, host: &str, article_ids: &[ArticleId]) -> i64 {
let id = discovery::upsert_candidate(
db,
&format!("https://{host}/feed.xml"),
host,
Some(title),
"2026-09-06T04:00:00Z".parse().unwrap(),
)
.await
.unwrap();
for article_id in article_ids {
discovery::link_article(db, id, *article_id).await.unwrap();
}
id
}
async fn status_of(db: &Db, id: i64) -> (String, Option<i64>, Option<String>) {
let candidate = discovery::candidate(db, id).await.unwrap().unwrap();
(
candidate.status,
candidate.miniflux_feed_id,
candidate.decided_at,
)
}
/// A Miniflux stand-in answering `POST /v1/feeds` with `body`.
async fn miniflux(status: StatusCode, body: serde_json::Value) -> String {
let handler = move || {
let body = body.clone();
async move { (status, Json(body)) }
};
let app = Router::new().route("/v1/feeds", axum::routing::post(handler));
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
let _ = axum::serve(listener, app).await;
});
format!("http://{addr}")
}
async fn app_with_miniflux(seed: &Seed, base_url: String) -> axum::Router {
crate::web::users::add(&seed.db, "reader", "correct horse battery", false)
.await
.unwrap();
crate::web::users::add(&seed.db, "admin", "correct horse battery", true)
.await
.unwrap();
let mut config = Config::default();
config.miniflux.base_url = base_url;
config.miniflux.api_key = Some("test-key-never-logged".into());
router(AppState::new(seed.db.clone(), config, None))
}
async fn post_form(app: &axum::Router, uri: &str, cookie: &str) -> Response {
app.clone()
.oneshot(
Request::builder()
.method(Method::POST)
.uri(uri)
.header(header::COOKIE, cookie)
.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded")
.header("sec-fetch-site", "same-origin")
.body(Body::from("category_id=3"))
.unwrap(),
)
.await
.unwrap()
}
#[tokio::test]
async fn feeds_page_is_admin_only_and_empty_without_candidates() {
let seed = seed().await;
let app = app_with_users(&seed.db).await;
let body = assert_admin_only(&app, "/dashboard/feeds").await;
assert!(body.contains("<h1>Feeds</h1>"), "{body}");
assert!(body.contains("No feed candidates"), "{body}");
assert!(body.contains("Candidates (0)"), "{body}");
// No API key in the test config, so Add is disabled with a notice.
assert!(
body.contains("Miniflux API key is not configured, so feeds cannot be added."),
"{body}"
);
}
#[tokio::test]
async fn candidates_are_listed_best_first() {
let seed = seed().await;
// Article 1 is rated `loved` (value 1.0); article 5 only ever reached
// `triaged`, so it scores on its blend alone.
candidate(&seed.db, "Weak Blog", "weak.example", &[5]).await;
candidate(&seed.db, "Strong Blog", "strong.example", &[1]).await;
let app = app_with_users(&seed.db).await;
let body = assert_admin_only(&app, "/dashboard/feeds").await;
assert!(body.contains("Candidates (2)"), "{body}");
assert!(body.contains("Strong Blog"), "{body}");
assert!(body.contains("Weak Blog"), "{body}");
assert!(body.contains("https://strong.example/feed.xml"), "{body}");
// The rated article is named in the Why column and links to it.
assert!(body.contains("/dashboard/articles/1"), "{body}");
assert!(body.contains("Gaussian Splatting"), "{body}");
let strong = body.find("Strong Blog").unwrap();
let weak = body.find("Weak Blog").unwrap();
assert!(strong < weak, "{body}");
}
#[tokio::test]
async fn dismiss_decides_the_row_and_redirects() {
let seed = seed().await;
let id = candidate(&seed.db, "Noise", "noise.example", &[2]).await;
let app = app_with_users(&seed.db).await;
let admin = login_cookie(&app, "admin", "correct horse battery").await;
let response = post_form(&app, &format!("/dashboard/feeds/{id}/dismiss"), &admin).await;
assert_eq!(response.status(), StatusCode::SEE_OTHER);
assert_eq!(
response.headers().get(header::LOCATION).unwrap(),
"/dashboard/feeds"
);
let (status, feed_id, decided_at) = status_of(&seed.db, id).await;
assert_eq!(status, "dismissed");
assert_eq!(feed_id, None);
assert!(decided_at.is_some());
let body = response_text(get(&app, "/dashboard/feeds", Some(&admin)).await).await;
assert!(body.contains("Dismissed Noise."), "{body}");
assert!(body.contains("Candidates (0)"), "{body}");
assert!(body.contains("Dismissed (1)"), "{body}");
}
#[tokio::test]
async fn add_subscribes_and_records_the_miniflux_feed_id() {
let seed = seed().await;
let id = candidate(&seed.db, "Strong Blog", "strong.example", &[1]).await;
let base_url = miniflux(StatusCode::CREATED, json!({"feed_id": 77})).await;
let app = app_with_miniflux(&seed, base_url).await;
let admin = login_cookie(&app, "admin", "correct horse battery").await;
let response = post_form(&app, &format!("/dashboard/feeds/{id}/add"), &admin).await;
assert_eq!(response.status(), StatusCode::SEE_OTHER);
let (status, feed_id, decided_at) = status_of(&seed.db, id).await;
assert_eq!(status, "added");
assert_eq!(feed_id, Some(77));
assert!(decided_at.is_some());
assert_eq!(
seed.db.kv_get(LAST_CATEGORY_KEY).await.unwrap().as_deref(),
Some("3")
);
let body = response_text(get(&app, "/dashboard/feeds", Some(&admin)).await).await;
assert!(body.contains("Added Strong Blog."), "{body}");
let added =
response_text(get(&app, "/dashboard/feeds?status=added", Some(&admin)).await).await;
assert!(added.contains("/feeds/77"), "{added}");
}
#[tokio::test]
async fn a_duplicate_subscription_still_decides_the_row() {
let seed = seed().await;
let id = candidate(&seed.db, "Known Blog", "known.example", &[1]).await;
let base_url = miniflux(
StatusCode::BAD_REQUEST,
json!({"error_message": "This feed already exists."}),
)
.await;
let app = app_with_miniflux(&seed, base_url).await;
let admin = login_cookie(&app, "admin", "correct horse battery").await;
let response = post_form(&app, &format!("/dashboard/feeds/{id}/add"), &admin).await;
assert_eq!(response.status(), StatusCode::SEE_OTHER);
let (status, feed_id, decided_at) = status_of(&seed.db, id).await;
assert_eq!(status, "added");
assert_eq!(feed_id, None);
assert!(decided_at.is_some());
let body = response_text(get(&app, "/dashboard/feeds", Some(&admin)).await).await;
assert!(body.contains("This feed already exists."), "{body}");
}
}
+5
View File
@@ -10,6 +10,7 @@
//! `_signals_table.html`, the allow-list check and the LIKE-pattern escaper. //! `_signals_table.html`, the allow-list check and the LIKE-pattern escaper.
pub mod articles; pub mod articles;
pub mod feeds;
pub mod jobs; pub mod jobs;
pub mod profile; pub mod profile;
pub mod ratings; pub mod ratings;
@@ -44,6 +45,7 @@ pub fn router() -> Router<AppState> {
.merge(runs::routes()) .merge(runs::routes())
.merge(articles::routes()) .merge(articles::routes())
.merge(ratings::routes()) .merge(ratings::routes())
.merge(feeds::routes())
.merge(profile::routes()) .merge(profile::routes())
.merge(settings::routes()) .merge(settings::routes())
.merge(jobs::routes()) .merge(jobs::routes())
@@ -422,6 +424,7 @@ struct OverviewTemplate {
ratings: Vec<LabelCount>, ratings: Vec<LabelCount>,
ratings_total: i64, ratings_total: i64,
access_requests: i64, access_requests: i64,
feed_candidates: i64,
unrated: Vec<UnratedPick>, unrated: Vec<UnratedPick>,
active_jobs: Vec<JobLine>, active_jobs: Vec<JobLine>,
finished_jobs: Vec<JobLine>, finished_jobs: Vec<JobLine>,
@@ -448,6 +451,7 @@ async fn overview(
.fetch_one(db.pool()) .fetch_one(db.pool())
.await .await
.map_err(db_err)?; .map_err(db_err)?;
let feed_candidates = crate::discovery::count(db, "candidate").await?;
let unrated = unrated_picks(db).await?; let unrated = unrated_picks(db).await?;
let (active_jobs, finished_jobs) = jobs_summary(db, &config).await?; let (active_jobs, finished_jobs) = jobs_summary(db, &config).await?;
let sparklines = overview_sparklines(db).await?; let sparklines = overview_sparklines(db).await?;
@@ -466,6 +470,7 @@ async fn overview(
ratings, ratings,
ratings_total, ratings_total,
access_requests, access_requests,
feed_candidates,
unrated, unrated,
active_jobs, active_jobs,
finished_jobs, finished_jobs,
+5
View File
@@ -277,6 +277,7 @@ const GROUP_ORDER: &[&str] = &[
"miniflux", "miniflux",
"bookorbit", "bookorbit",
"mail", "mail",
"discovery",
]; ];
/// Help text per key, seeded from the README configuration table and the /// Help text per key, seeded from the README configuration table and the
@@ -401,6 +402,9 @@ pub const SETTINGS_HELP: &[(&str, &str)] = &[
("mail.smtp_pass", "SMTP password. Environment only; requires a server restart."), ("mail.smtp_pass", "SMTP password. Environment only; requires a server restart."),
("mail.from", "Sender mailbox as an address or Name <address>. Requires a server restart."), ("mail.from", "Sender mailbox as an address or Name <address>. Requires a server restart."),
("mail.notify_to", "Recipient for new access-request notifications. Requires a server restart."), ("mail.notify_to", "Recipient for new access-request notifications. Requires a server restart."),
("discovery.enabled", "Run the feed discovery stage during generate."),
("discovery.max_lookups_per_run", "How many not-yet-checked hosts one run may look up in Miniflux; each host is re-checked at most every 90 days."),
("discovery.skip_hosts", "Hosts never looked up, one per line. Matches the host or any subdomain of it."),
]; ];
/// `DAILY_EPUB_` + the path upper-cased with `.` → `__` (§13.1 item 2). /// `DAILY_EPUB_` + the path upper-cased with `.` → `__` (§13.1 item 2).
@@ -1725,6 +1729,7 @@ mod tests {
"miniflux", "miniflux",
"bookorbit", "bookorbit",
"mail", "mail",
"discovery",
] ]
); );
let anthropic = groups let anthropic = groups
+1 -1
View File
@@ -1,2 +1,2 @@
/*! tailwindcss v4.3.3 | MIT License | https://tailwindcss.com */ /*! tailwindcss v4.3.3 | MIT License | https://tailwindcss.com */
@layer properties{@supports (((-webkit-hyphens:none)) and (not (margin-trim:inline))) or ((-moz-orient:inline) and (not (color:rgb(from red r g b)))){*,:before,:after,::backdrop{--tw-translate-x:0;--tw-translate-y:0;--tw-translate-z:0;--tw-rotate-x:initial;--tw-rotate-y:initial;--tw-rotate-z:initial;--tw-skew-x:initial;--tw-skew-y:initial;--tw-space-y-reverse:0;--tw-divide-y-reverse:0;--tw-border-style:solid;--tw-leading:initial;--tw-font-weight:initial;--tw-tracking:initial;--tw-ordinal:initial;--tw-slashed-zero:initial;--tw-numeric-figure:initial;--tw-numeric-spacing:initial;--tw-numeric-fraction:initial;--tw-shadow:0 0 #0000;--tw-shadow-color:initial;--tw-shadow-alpha:100%;--tw-inset-shadow:0 0 #0000;--tw-inset-shadow-color:initial;--tw-inset-shadow-alpha:100%;--tw-ring-color:initial;--tw-ring-shadow:0 0 #0000;--tw-inset-ring-color:initial;--tw-inset-ring-shadow:0 0 #0000;--tw-ring-inset:initial;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-offset-shadow:0 0 #0000;--tw-outline-style:solid;--tw-blur:initial;--tw-brightness:initial;--tw-contrast:initial;--tw-grayscale:initial;--tw-hue-rotate:initial;--tw-invert:initial;--tw-opacity:initial;--tw-saturate:initial;--tw-sepia:initial;--tw-drop-shadow:initial;--tw-drop-shadow-color:initial;--tw-drop-shadow-alpha:100%;--tw-drop-shadow-size:initial;--tw-duration:initial}}}@layer theme{:root,:host{--font-sans:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--font-serif:"Newsreader", "Newsreader Fallback Georgia", "Newsreader Fallback Times", "Newsreader Fallback Noto Serif", "Newsreader Fallback DejaVu Serif", "Iowan Old Style", "Palatino Linotype", Georgia, serif;--font-mono:ui-monospace, SFMono-Regular, Menlo, monospace;--spacing:.25rem;--container-md:28rem;--container-lg:32rem;--container-2xl:42rem;--container-7xl:80rem;--text-xs:.75rem;--text-xs--line-height:calc(1 / .75);--text-sm:.875rem;--text-sm--line-height:calc(1.25 / .875);--text-base:1rem;--text-base--line-height:calc(1.5 / 1);--text-lg:1.125rem;--text-lg--line-height:calc(1.75 / 1.125);--text-xl:1.25rem;--text-xl--line-height:calc(1.75 / 1.25);--text-2xl:1.5rem;--text-2xl--line-height:calc(2 / 1.5);--text-3xl:1.875rem;--text-3xl--line-height:calc(2.25 / 1.875);--text-4xl:2.25rem;--text-4xl--line-height:calc(2.5 / 2.25);--text-5xl:3rem;--text-5xl--line-height:1;--font-weight-normal:400;--font-weight-medium:500;--font-weight-semibold:600;--tracking-normal:0em;--leading-tight:1.25;--leading-snug:1.375;--leading-normal:1.5;--leading-relaxed:1.625;--radius-sm:.25rem;--default-transition-duration:.15s;--default-transition-timing-function:cubic-bezier(.4, 0, .2, 1);--default-font-family:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--default-mono-font-family:ui-monospace, SFMono-Regular, Menlo, monospace;--color-paper:var(--paper);--color-paper-2:var(--paper-2);--color-ink:var(--ink)}}@layer base{*,:after,:before,::backdrop{box-sizing:border-box;border:0 solid;margin:0;padding:0}::file-selector-button{box-sizing:border-box;border:0 solid;margin:0;padding:0}html,:host{-webkit-text-size-adjust:100%;tab-size:4;line-height:1.5;font-family:var(--default-font-family,-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Noto Sans", Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji");font-feature-settings:var(--default-font-feature-settings,normal);font-variation-settings:var(--default-font-variation-settings,normal);-webkit-tap-highlight-color:transparent}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,samp,pre{font-family:var(--default-mono-font-family,ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace);font-feature-settings:var(--default-mono-font-feature-settings,normal);font-variation-settings:var(--default-mono-font-variation-settings,normal);font-size:1em}small{font-size:80%}sub,sup{vertical-align:baseline;font-size:75%;line-height:0;position:relative}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}:-moz-focusring:where(:not(iframe)){outline:auto}progress{vertical-align:baseline}summary{display:list-item}ol,ul,menu{list-style:none}img,svg,video,canvas,audio,iframe,embed,object{vertical-align:middle;display:block}img,video{max-width:100%;height:auto}button,input,select,optgroup,textarea{font:inherit;font-feature-settings:inherit;font-variation-settings:inherit;letter-spacing:inherit;color:inherit;opacity:1;background-color:#0000;border-radius:0}::file-selector-button{font:inherit;font-feature-settings:inherit;font-vaLine truncated @layer properties{@supports (((-webkit-hyphens:none)) and (not (margin-trim:inline))) or ((-moz-orient:inline) and (not (color:rgb(from red r g b)))){*,:before,:after,::backdrop{--tw-translate-x:0;--tw-translate-y:0;--tw-translate-z:0;--tw-rotate-x:initial;--tw-rotate-y:initial;--tw-rotate-z:initial;--tw-skew-x:initial;--tw-skew-y:initial;--tw-space-y-reverse:0;--tw-divide-y-reverse:0;--tw-border-style:solid;--tw-leading:initial;--tw-font-weight:initial;--tw-tracking:initial;--tw-ordinal:initial;--tw-slashed-zero:initial;--tw-numeric-figure:initial;--tw-numeric-spacing:initial;--tw-numeric-fraction:initial;--tw-shadow:0 0 #0000;--tw-shadow-color:initial;--tw-shadow-alpha:100%;--tw-inset-shadow:0 0 #0000;--tw-inset-shadow-color:initial;--tw-inset-shadow-alpha:100%;--tw-ring-color:initial;--tw-ring-shadow:0 0 #0000;--tw-inset-ring-color:initial;--tw-inset-ring-shadow:0 0 #0000;--tw-ring-inset:initial;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-offset-shadow:0 0 #0000;--tw-outline-style:solid;--tw-blur:initial;--tw-brightness:initial;--tw-contrast:initial;--tw-grayscale:initial;--tw-hue-rotate:initial;--tw-invert:initial;--tw-opacity:initial;--tw-saturate:initial;--tw-sepia:initial;--tw-drop-shadow:initial;--tw-drop-shadow-color:initial;--tw-drop-shadow-alpha:100%;--tw-drop-shadow-size:initial;--tw-duration:initial}}}@layer theme{:root,:host{--font-sans:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--font-serif:"Newsreader", "Newsreader Fallback Georgia", "Newsreader Fallback Times", "Newsreader Fallback Noto Serif", "Newsreader Fallback DejaVu Serif", "Iowan Old Style", "Palatino Linotype", Georgia, serif;--font-mono:ui-monospace, SFMono-Regular, Menlo, monospace;--spacing:.25rem;--container-md:28rem;--container-lg:32rem;--container-2xl:42rem;--container-7xl:80rem;--text-xs:.75rem;--text-xs--line-height:calc(1 / .75);--text-sm:.875rem;--text-sm--line-height:calc(1.25 / .875);--text-base:1rem;--text-base--line-height:calc(1.5 / 1);--text-lg:1.125rem;--text-lg--line-height:calc(1.75 / 1.125);--text-xl:1.25rem;--text-xl--line-height:calc(1.75 / 1.25);--text-2xl:1.5rem;--text-2xl--line-height:calc(2 / 1.5);--text-3xl:1.875rem;--text-3xl--line-height:calc(2.25 / 1.875);--text-4xl:2.25rem;--text-4xl--line-height:calc(2.5 / 2.25);--text-5xl:3rem;--text-5xl--line-height:1;--font-weight-normal:400;--font-weight-medium:500;--font-weight-semibold:600;--tracking-normal:0em;--leading-tight:1.25;--leading-snug:1.375;--leading-normal:1.5;--leading-relaxed:1.625;--radius-sm:.25rem;--default-transition-duration:.15s;--default-transition-timing-function:cubic-bezier(.4, 0, .2, 1);--default-font-family:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--default-mono-font-family:ui-monospace, SFMono-Regular, Menlo, monospace;--color-paper:var(--paper);--color-paper-2:var(--paper-2);--color-ink:var(--ink)}}@layer base{*,:after,:before,::backdrop{box-sizing:border-box;border:0 solid;margin:0;padding:0}::file-selector-button{box-sizing:border-box;border:0 solid;margin:0;padding:0}html,:host{-webkit-text-size-adjust:100%;tab-size:4;line-height:1.5;font-family:var(--default-font-family,-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Noto Sans", Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji");font-feature-settings:var(--default-font-feature-settings,normal);font-variation-settings:var(--default-font-variation-settings,normal);-webkit-tap-highlight-color:transparent}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,samp,pre{font-family:var(--default-mono-font-family,ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace);font-feature-settings:var(--default-mono-font-feature-settings,normal);font-variation-settings:var(--default-mono-font-variation-settings,normal);font-size:1em}small{font-size:80%}sub,sup{vertical-align:baseline;font-size:75%;line-height:0;position:relative}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}:-moz-focusring:where(:not(iframe)){outline:auto}progress{vertical-align:baseline}summary{display:list-item}ol,ul,menu{list-style:none}img,svg,video,canvas,audio,iframe,embed,object{vertical-align:middle;display:block}img,video{max-width:100%;height:auto}button,input,select,optgroup,textarea{font:inherit;font-feature-settings:inherit;font-variation-settings:inherit;letter-spacing:inherit;color:inherit;opacity:1;background-color:#0000;border-radius:0}::file-selector-button{font:inherit;font-feature-settings:inherit;font-vaLine truncated
+21
View File
@@ -0,0 +1,21 @@
{% extends "layout.html" %}{% block content %}<section class="dashboard">
<header class="page-head"><div>
<h1>Feeds</h1>
<p class="page-desc">Feeds behind the stories that reached us through an aggregator, ranked by how the articles they carried actually fared. Add subscribes in Miniflux; Dismiss hides the feed for good.</p>
</div><div class="page-actions"><a class="btn" href="/dashboard/articles">Articles</a><a class="btn" href="/dashboard/runs">Runs</a></div></header>
<nav class="tabs" aria-label="Candidate status"><a href="/dashboard/feeds"{% if status == "candidate" %} class="active" aria-current="page"{% endif %}>Candidates ({{ candidate_count }})</a> <a href="/dashboard/feeds?status=added"{% if status == "added" %} class="active" aria-current="page"{% endif %}>Added ({{ added_count }})</a> <a href="/dashboard/feeds?status=dismissed"{% if status == "dismissed" %} class="active" aria-current="page"{% endif %}>Dismissed ({{ dismissed_count }})</a></nav>
{% if let Some(error) = categories_error %}<p class="notice">{{ error }}</p>{% endif %}
{% include "dashboard/_pager.html" %}
<table class="table-fixed">
<thead><tr><th class="num w-14">Score</th><th>Feed</th><th>Why</th><th class="num w-20">Articles</th><th class="w-20">Seen</th><th class="w-60 lg:w-84"><span class="sr-only">Actions</span></th></tr></thead>
<tbody>{% for row in rows %}<tr>
<td class="num tabular-nums">{{ row.score }}</td>
<td class="cell-wrap"><a class="line-clamp-2" href="{{ row.feed_url }}" rel="noopener" target="_blank" title="{{ row.feed_url }}">{{ row.label }}</a><span class="muted block truncate text-xs" title="{{ row.host }}">{{ row.host }}</span></td>
<td class="cell-wrap">{% for interest in row.interests %}<span class="badge">{{ interest }}</span> {% endfor %}{% for article in row.articles %}<div class="line-clamp-1 text-xs"><a href="/dashboard/articles/{{ article.id }}" title="{{ article.title }}">{{ article.title }}</a></div>{% endfor %}</td>
<td class="num">{{ row.article_count }}</td>
<td class="cell-tight text-xs text-muted"><span title="First seen {{ row.first_seen }} · last seen {{ row.last_seen }}">{{ row.seen }}</span></td>
<td>{% if status == "candidate" %}<div class="flex flex-wrap items-start gap-2"><form class="form-inline flex-nowrap" method="post" action="/dashboard/feeds/{{ row.id }}/add"><select class="min-w-0 max-w-36 truncate" name="category_id" aria-label="Category for {{ row.label }}"{% if categories.is_empty() %} disabled{% endif %}>{% for category in categories %}<option value="{{ category.id }}"{% if category.id == selected_category %} selected{% endif %}>{{ category.title }}</option>{% endfor %}</select><button class="btn-primary" type="submit"{% if categories.is_empty() %} disabled title="Miniflux categories are unavailable"{% endif %}>Add</button></form><form method="post" action="/dashboard/feeds/{{ row.id }}/dismiss"><button class="btn" type="submit">Dismiss</button></form></div>{% else %}<span class="muted text-xs">{{ row.decided_at }}{% if let Some(href) = row.miniflux_href %} · <a href="{{ href }}" rel="noopener" target="_blank">in Miniflux</a>{% endif %}</span>{% endif %}</td>
</tr>{% endfor %}{% if rows.is_empty() %}<tr><td colspan="6" class="text-muted">No feed candidates with this status. The discovery stage adds them during a run; <code>daily-epub feeds discover</code> backfills from recent articles.</td></tr>{% endif %}</tbody>
</table>
{% include "dashboard/_pager.html" %}
</section>{% endblock %}
@@ -10,6 +10,7 @@
<div class="tile"><span class="tile-label">Verdicts, 7 days</span><span class="tile-num">{{ ratings_total }}</span><span class="tile-delta"><a href="/dashboard/ratings">rating history</a></span></div> <div class="tile"><span class="tile-label">Verdicts, 7 days</span><span class="tile-num">{{ ratings_total }}</span><span class="tile-delta"><a href="/dashboard/ratings">rating history</a></span></div>
<div class="tile"><span class="tile-label">Unrated picks</span><span class="tile-num">{{ unrated.len() }}</span><span class="tile-delta">from the last three issues</span></div> <div class="tile"><span class="tile-label">Unrated picks</span><span class="tile-num">{{ unrated.len() }}</span><span class="tile-delta">from the last three issues</span></div>
<div class="tile"><span class="tile-label">Active jobs</span><span class="tile-num">{{ active_jobs.len() }}</span><span class="tile-delta"><a href="/dashboard/jobs">all jobs</a></span></div> <div class="tile"><span class="tile-label">Active jobs</span><span class="tile-num">{{ active_jobs.len() }}</span><span class="tile-delta"><a href="/dashboard/jobs">all jobs</a></span></div>
<div class="tile"><span class="tile-label">Feed candidates</span><span class="tile-num">{{ feed_candidates }}</span><span class="tile-delta"><a href="/dashboard/feeds">review feeds</a></span></div>
<div class="tile"><span class="tile-label">Access requests</span><span class="tile-num">{{ access_requests }}</span><span class="tile-delta"><a href="/dashboard/users">review requests</a></span></div> <div class="tile"><span class="tile-label">Access requests</span><span class="tile-num">{{ access_requests }}</span><span class="tile-delta"><a href="/dashboard/users">review requests</a></span></div>
</div> </div>
+1
View File
@@ -47,6 +47,7 @@
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "runs" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/runs"{% if page.active_nav == "runs" %} aria-current="page"{% endif %}>Runs</a> <a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "runs" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/runs"{% if page.active_nav == "runs" %} aria-current="page"{% endif %}>Runs</a>
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "articles" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/articles"{% if page.active_nav == "articles" %} aria-current="page"{% endif %}>Articles</a> <a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "articles" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/articles"{% if page.active_nav == "articles" %} aria-current="page"{% endif %}>Articles</a>
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "ratings" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/ratings"{% if page.active_nav == "ratings" %} aria-current="page"{% endif %}>Ratings</a> <a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "ratings" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/ratings"{% if page.active_nav == "ratings" %} aria-current="page"{% endif %}>Ratings</a>
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "feeds" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/feeds"{% if page.active_nav == "feeds" %} aria-current="page"{% endif %}>Feeds</a>
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "profile" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/profile"{% if page.active_nav == "profile" %} aria-current="page"{% endif %}>Profile</a> <a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "profile" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/profile"{% if page.active_nav == "profile" %} aria-current="page"{% endif %}>Profile</a>
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "stats" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/stats"{% if page.active_nav == "stats" %} aria-current="page"{% endif %}>Stats</a> <a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "stats" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/stats"{% if page.active_nav == "stats" %} aria-current="page"{% endif %}>Stats</a>
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "jobs" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/jobs"{% if page.active_nav == "jobs" %} aria-current="page"{% endif %}>Jobs</a> <a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "jobs" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/jobs"{% if page.active_nav == "jobs" %} aria-current="page"{% endif %}>Jobs</a>