Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 thallada
|
72b32dc69b | ||
|
thallada
|
56d2d74a97 | ||
 thalladaandClaude Fable 5.1
|
81cbbb407f | ||
|
thallada
|
c0f086e0da | ||
|
thalladaandClaude Fable 5.1
|
2bbb4809b7 | ||
|
thalladaandClaude Fable 5.1
|
c971fd9317 | ||
|
thalladaandClaude Fable 5.1
|
4a2fc6d1c4 |
No files matched your search
@@ -133,6 +133,7 @@ daily-epub stats [--days 14] # the evaluation framework, one fact per line
|
||||
daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes]
|
||||
daily-epub features prune # stale embeddings, old telemetry and assessments
|
||||
daily-epub backfill-social [--days 7] # re-poll social scores for recent articles
|
||||
daily-epub feeds discover [--days 14] [--limit 50] # seed feed candidates from recent aggregator articles
|
||||
daily-epub db migrate # run migrations (also automatic on every start)
|
||||
daily-epub config check # validate the config, print the resolved roles, keys and paths
|
||||
daily-epub users add USER [--admin] [--password-stdin]
|
||||
@@ -145,6 +146,13 @@ daily-epub users logout USER # revoke all of USER's sessions
|
||||
daily-epub job run NAME # systemd job-unit entry point; normally not run by hand
|
||||
```
|
||||
|
||||
On the server, run these through the `justfile` (`just --list`): every server
|
||||
recipe wraps the binary in `systemd-run` as the `daily-epub` user with
|
||||
`/etc/daily-epub/env` loaded, exactly as the units do, so API keys resolve.
|
||||
`just de <subcommand...>` runs anything; `just feeds-discover 30`,
|
||||
`just config-check`, `just deploy` and `just logs-generate` cover the common cases.
|
||||
Plain `sudo -u daily-epub daily-epub ...` does **not** read the env file.
|
||||
|
||||
`--dry-run` does everything except deliver: it still ingests, persists entries and
|
||||
articles, curates and **builds both EPUBs into `--out`**, but it does not copy to
|
||||
BookOrbit, does not run the retention sweep, does not write the `issues` row and
|
||||
@@ -245,6 +253,14 @@ verdict and optional note. Imports run in the background and show per-URL
|
||||
status in the page plus live logs on the job page. If dashboard jobs are
|
||||
disabled, queueing still works; run `daily-epub job run import-ratings` by hand.
|
||||
|
||||
The Feeds page lists feeds discovered behind articles that reached the paper
|
||||
only through an aggregator (Hacker News, Lobsters, Reddit, Scour), ranked by
|
||||
the same per-article signals the paper selects on, so the strongest leads sit
|
||||
at the top. Each row offers **Add**, which subscribes the feed in a chosen
|
||||
Miniflux category, and **Dismiss**. Discovery runs as a best-effort stage
|
||||
during `generate`; `daily-epub feeds discover --days 14 --limit 50` seeds the
|
||||
page from articles already in the database.
|
||||
|
||||
The Settings page derives its fields from `Config`, rewrites `config.toml` in
|
||||
place with `toml_edit`, preserves comments/order and file permissions, validates
|
||||
before an atomic rename, and records attributed history. It re-reads hand edits
|
||||
@@ -407,6 +423,9 @@ prints what resolved.
|
||||
| `mail.smtp_pass` | — | **`DAILY_EPUB_MAIL__SMTP_PASS`**, environment only. |
|
||||
| `mail.from` | `""` | Sender mailbox, either a bare address or `Name <address>`. |
|
||||
| `mail.notify_to` | unset | Recipient for access-request notifications. |
|
||||
| `discovery.enabled` | `true` | Run the feed discovery stage during `generate`. |
|
||||
| `discovery.max_lookups_per_run` | `30` | Hosts one run may look up in Miniflux. Each host is re-checked at most every 90 days. |
|
||||
| `discovery.skip_hosts` | aggregators, code hosts, social networks | Hosts never looked up. A host matches itself or any subdomain of it. |
|
||||
| `xtc.enabled` | `true` | Set `false` to skip the converter entirely. |
|
||||
| `xtc.command` | `node` | Converter executable. |
|
||||
| `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends `<input.epub> -o <output> -f <format>` (plus `-c <settings>`). |
|
||||
|
||||
@@ -237,3 +237,20 @@ smtp_user = ""
|
||||
# smtp_pass: environment only (DAILY_EPUB_MAIL__SMTP_PASS)
|
||||
from = "" # e.g. The Daily EPUB <daily@hallada.net>
|
||||
# notify_to = "operator@example.com" # recipient for access-request notifications
|
||||
|
||||
[discovery]
|
||||
enabled = true # run the feed discovery stage during generate
|
||||
max_lookups_per_run = 30 # hosts looked up in Miniflux per run; each host is re-checked at most every 90 days
|
||||
skip_hosts = [ # never looked up: the host itself or any subdomain of it
|
||||
"news.ycombinator.com",
|
||||
"lobste.rs",
|
||||
"reddit.com",
|
||||
"github.com",
|
||||
"gist.github.com",
|
||||
"x.com",
|
||||
"twitter.com",
|
||||
"youtube.com",
|
||||
"en.wikipedia.org",
|
||||
"arxiv.org",
|
||||
"docs.google.com",
|
||||
]
|
||||
@@ -0,0 +1,157 @@
|
||||
# Feed discovery: propose new Miniflux subscriptions from aggregator hits
|
||||
|
||||
**Date:** 2026-09-07
|
||||
**Repository:** `thallada/the-daily-epub`
|
||||
**Status:** small implementation plan, ready to execute
|
||||
**Builds on:** `docs/plans/2026-09-03-web-dashboard.md` (the admin dashboard this adds a page to), `docs/plans/2026-09-02-personalized-curation-v2.md` (the per-article signals this reuses)
|
||||
|
||||
Written for a fresh implementation agent. Facts about this repo were checked against `main` (`e69940f`) on 2026-09-07. Facts about the Miniflux API were checked against `https://miniflux.app/docs/api.html` the same day; anything marked *assumption* must be re-checked against the live instance before relying on it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Most of the paper's articles arrive through aggregators (Hacker News, Lobsters, Reddit, Scour), not through the author's own feed. Every such article is a lead on a feed the operator does not subscribe to yet. This feature:
|
||||
|
||||
1. During `generate`, after articles are persisted, asks Miniflux to discover the feed(s) behind each aggregator-only article and records the ones the operator is not already subscribed to as **feed candidates**.
|
||||
2. Adds an admin dashboard page, **Feeds** (`/dashboard/feeds`), listing candidates ranked by how likely the operator is to enjoy the feed, computed **algorithmically from telemetry the pipeline already persists** (utility / preliminary blend, stage reached, explicit ratings). No LLM calls, no new embeddings.
|
||||
3. Each row has **Add** (with a Miniflux category picker; calls `POST /v1/feeds`) and **Dismiss**.
|
||||
|
||||
Over time the subscribed set grows and the pool of direct-feed articles each issue draws from grows with it.
|
||||
|
||||
Out of scope (deliberately, see §7): auto-subscribing, feed previews/health checks, Scour integration, OPML, per-feed LLM judgement, un-dismissing from the UI.
|
||||
|
||||
## 2. Verified facts
|
||||
|
||||
### About this repo
|
||||
|
||||
- **Pipeline order** (`src/pipeline.rs:354` `run_stages`): ingest → dedupe → extraction → `persist_articles` (stage 4, ~line 443, mints real article ids) → social enrichment (stage 5, best effort, ~line 448) → hygiene/embeddings/signals (stage 6) → … Failure policy is documented at the top of the file: *best effort* stages log, push a warning onto `report` (status `degraded`) and continue. The new stage is best effort.
|
||||
- **How an article arrived** is stored per article in `articles.sources_json` as `Vec<SourceRef>` (`src/types.rs:64`), each with a `SourceKind` (`Scour | HnFrontpage | Lobsters | Reddit | Feed`, `src/types.rs:49`). Classification is `dedupe::classify_source_with_feed` (`src/dedupe.rs:177`). `Article::came_via(kind)` exists (`src/types.rs:147`). "Aggregator-only" therefore means `!article.sources.iter().any(|s| s.kind == SourceKind::Feed)`.
|
||||
- **Ingest already loads the whole subscription list once per run**: `MinifluxClient::ingest_window` returns `(Vec<Entry>, HashMap<FeedId, FeedMeta>)` where `FeedMeta` has `feed_url` and `site_url` (`src/miniflux.rs:105`). That map is the "already subscribed" set; no extra call needed in the pipeline.
|
||||
- **Miniflux client** (`src/miniflux.rs`) is currently GET-only: one private `get_json` helper with retry (`RetryPolicy`, retry on network/5xx/429), `X-Auth-Token` auth, `MinifluxError::{Http, Status, Decode, MissingApiKey}`. Module doc says "Reads only … read state is never mutated" — still true for the pipeline; the dashboard **Add** action is the only writer and only creates feeds.
|
||||
- **Per-article ranking evidence is persisted** in `candidate_runs` (`migrations/0002_curation_v2.sql:59`): `(run_id, article_id) PK, stage, excluded_reason, signals_json, utility REAL, rank_utility, …`. `signals_json` decodes to `curate::telemetry::SignalsJson` (`src/curate/telemetry.rs:114`) whose `blend()` returns the preliminary blend on 0–100 and whose `top_interests: Vec<TopInterest{name, z, cos}>` names the matched interests. `utility` (0–100) is set only for articles that reached the deep set (`admitted`/`assessed`/`shortlisted`/`selected`). Stage names: `dashboard::STAGES` (`src/web/dashboard/mod.rs`). `runs.status` distinguishes finished runs; dry runs are not flagged in `runs` (a dry run still writes telemetry, which is fine to use).
|
||||
- **Ratings**: `Db::current_ratings(lookback_days) -> Vec<RatedArticle>` (`src/db.rs:708`) returns one current verdict per article with `value` (`loved` 1.0, `good` 0.35, `not_for_me` −1.0) — the same numbers `signals::RatedExample` uses.
|
||||
- **Dashboard conventions**: one submodule per page group under `src/web/dashboard/`, each exposing `routes() -> Router<AppState>` merged in `dashboard::router()` (`src/web/dashboard/mod.rs:41`). Admin gating is applied by the caller. POST-then-redirect with a session flash is the established pattern (`users::approve`, `src/web/dashboard/users.rs:122`, uses `set_flash(&session, "error"|"ok", text)` then `Redirect::to`). Nav tabs are hardcoded in `src/web/templates/layout.html:47–54` keyed on `page.active_nav`. Overview tiles live in `src/web/templates/dashboard/overview.html`. Templates are askama; shared pager is `dashboard/_pager.html`.
|
||||
- **`AppState`** (`src/server.rs:79`) holds `db`, `config: Arc<RwLock<Arc<Config>>>` (read via `state.config()`), `web: Arc<WebState>`, `mailer`. Handlers construct outbound clients from config as needed (`bookorbit.rs` does this for OPDS); reuse whatever shared `reqwest::Client` `WebState` already carries, otherwise `http::build_client(http::DEFAULT_TIMEOUT)`.
|
||||
- **Config**: sections are `#[serde(deny_unknown_fields, default)]` structs on `Config` with `Default` impls; env override is `DAILY_EPUB_<SECTION>__<KEY>`. The settings dashboard has a **hardcoded section list**, help text and a test asserting the list (`src/web/dashboard/settings.rs`, see the BookOrbit plan §2 for the exact spots) — a new section must be added there or the settings page/test breaks. `config.example.toml` and the README config table document every key.
|
||||
- **Migrations**: `sqlx::migrate!("./migrations")`; latest is `0009_request_username.sql`, so the new file is `0010_feed_discovery.sql`.
|
||||
- **Jobs / CLI**: catalogue jobs are in `src/jobs.rs` and run through `main.rs` (`Job::ImportRatings => imports::run(...)`). A one-off backfill only needs a CLI subcommand, not a job (§4 step 6).
|
||||
|
||||
### About Miniflux (docs, 2026-09-07)
|
||||
|
||||
- **Discover**: `POST /v1/discover` with body `{"url": "<page or site url>"}` → `[{"url": "<feed url>", "title": "…", "type": "rss|atom|json"}]`. Optional body fields (`username`, `password`, `user_agent`, `fetch_via_proxy`) are not needed. **Observed live on 2026-09-07** (operator ran it against several hosts):
|
||||
- When the page has no `<link rel="alternate">`, Miniflux falls back to probing well-known paths on the site root (`/atom.xml`, `/feed.atom`, `/feed.xml`, `/feed/`, `/index.rss`, `/index.xml`, `/rss.xml`, `/rss/`, `/rss/feed.xml`) and **returns every path that answered 200 without checking that the body is a feed**. A site that serves its SPA shell for any path (zombo.com, immich, BookOrbit) yields up to nine bogus results, each with `title == url`. **Discover results are leads, not facts; every one must be validated before it is stored (§3, §4 step 3).**
|
||||
- When the target site fails, the response is a 4xx/5xx with `{"error_message": "fetcher: bad gateway (502 status code)"}`, `"fetcher: unexpected status code (412 status code)"`, or `"resource not found"` (nothing found / 404). Treat all of these as "host checked, zero candidates" and move on; never retry within the run.
|
||||
- Real `<link rel="alternate">` hits work from an **article** URL. Observed for `https://blog.philz.dev/blog/language-server-db/`:
|
||||
`[{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.xml","type":"atom"},{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.json","type":"json"}]`.
|
||||
The same content is offered twice (Atom and JSON Feed) under one title, so link-tag hits need deduping by title too (§3). Miniflux substitutes the URL for the title when the tag has none, so `title == url` marks a result as a well-known-path guess in practice.
|
||||
- **Create feed duplicate**: observed `HTTP 400` with `{"error_message":"This feed already exists."}`. The flash shows `error_message` verbatim; on that exact message the row is also flipped to `added` (without a `miniflux_feed_id`), since the operator subscribed by other means.
|
||||
- **Create feed**: `POST /v1/feeds` with `{"feed_url": "…", "category_id": <int>}` → `{"feed_id": <int>}`. Errors come back as 4xx/5xx with a JSON `error_message`; a duplicate subscription is a 4xx (*assumption:* 400 with "This feed already exists"). Surface `error_message` in the flash rather than special-casing.
|
||||
- **Categories**: `GET /v1/categories` → `[{"id", "user_id", "title", "hide_globally"}]`.
|
||||
- **Feeds**: `GET /v1/feeds` elements carry `feed_url` and `site_url` (already modelled as `MinifluxFeed`).
|
||||
- API keys are full-access; the existing key can create feeds. Nothing to provision.
|
||||
|
||||
## 3. Design decisions (settled)
|
||||
|
||||
| Topic | Decision | Why |
|
||||
|---|---|---|
|
||||
| Where discovery runs | A best-effort **pipeline stage right after social enrichment** (needs real article ids, before hygiene). Plus a CLI backfill (`daily-epub feeds discover --days N`) that runs the same function over recent `articles`, for seeding. | The user asked for "a step in article processing". The backfill is the same function behind a flag; without it the page is empty until tomorrow. |
|
||||
| Which articles | Aggregator-only articles: no `SourceKind::Feed` in `sources`. Skip when the canonical URL's host is in `discovery.skip_hosts` or matches the host of any subscribed feed's `site_url`/`feed_url`. | Articles that already came through a direct feed prove nothing. Host match against subscriptions catches "same site, different feed URL" cheaply. |
|
||||
| Discovery mechanism | **Miniflux `POST /v1/discover`** with the article URL, followed by **our own validation fetch** of each returned URL. No new crate. | Miniflux already does `<link rel=alternate>` + well-known paths + YouTube etc. and runs on the same host, but its well-known-path guesses are unverified (§2). Validating is ~40 lines; replacing discover with a home-built finder is far more. |
|
||||
| Validation | `GET` each discovered URL with the shared client (10 s timeout, read at most 256 KB). Accept only if the body, after an optional BOM, whitespace and `<?xml …?>` declaration, starts with `<rss`, `<feed`, or `<rdf:RDF`, or is JSON containing `"https://jsonfeed.org/version/`. Take the feed's own `<title>` from that body (first `<title>…</title>` in the buffer, unescaped, trimmed) as the candidate title when Miniflux gave `title == url`. Any other body, a non-2xx status or an error rejects the URL. | Content-type alone is unreliable (SPAs return `text/html` for everything, some real feeds return `text/xml` or `application/octet-stream`). Sniffing the root element is cheap and decisive. |
|
||||
| Duplicate results | After validation: group link-tag hits (`title != url`) by `title`, keeping one per group and preferring `type` `rss`/`atom` over `json`; then at most **one** `title == url` guess (the first in Miniflux's order); at most 3 per host overall. | blog.philz.dev offers the same feed as Atom and JSON Feed under one title; WordPress answers `/feed/` and `/rss/`, Hugo `/index.xml` and `/feed.xml`, all with the same content. One is enough. Real tag/category feeds arrive with distinct link-tag titles and are kept. |
|
||||
| Per-host memo | Table `feed_discovery_hosts(host PK, checked_at, candidates)`. A host is looked up **once**; re-checked only after `RECHECK_DAYS = 90` (const). New articles on an already-checked host are just **linked** to that host's existing candidates (no network). | Keeps per-run lookups bounded and lets a candidate accumulate evidence for free. |
|
||||
| Per-run cap | `discovery.max_lookups_per_run` (default 30), concurrency 4, 10 s timeout per lookup. Order: articles with the highest social score first. | A run already takes ~12 min; discovery must not add more than ~1 min. |
|
||||
| Multiple feeds per host | Drop results whose URL or title contains `comment` before validating. Each kept result is one candidate keyed by canonicalized `feed_url` (unique). | Comment feeds are noise; tag/category feeds are a real choice the operator can make on the page. |
|
||||
| Ranking | Pure function over the candidate's linked articles (§4 step 4): per-article evidence `s_i ∈ [−1, 1]` = explicit rating value if rated; else `0.85` if `stage = selected`; else `utility/100` if present; else `blend()/100` if present; else no evidence. Feed score = shrunk mean `(Σ s_i + prior·k) / (n + k)` with `prior = 0.3`, `k = 2`, shown as 0–100. Sort: score desc, then `n` desc, then `last_seen` desc. | Reuses exactly the signals the paper selects on (interest, knn, feed, social, heuristic, triage, quality, fit) with zero new cost. Shrinkage stops a single 95-utility article outranking three 80s. Constants live in one place; not configurable (YAGNI). |
|
||||
| "Why" column | The three most frequent `top_interests[].name` across the linked articles, plus the linked article titles (top 3 by `s_i`) linking to `/dashboard/articles/{id}`. | Enough to decide at a glance; all read from `signals_json`, no extra work. |
|
||||
| Add | Row form: `<select name="category_id">` + **Add** → `POST /dashboard/feeds/{id}/add` → Miniflux `POST /v1/feeds` → set `status = 'added'`, `miniflux_feed_id`, `decided_at`; flash "Added *title* to *category*". The select defaults to the last category used (`kv` key `feed_discovery_last_category`). | The operator will click Add many times in a row; remembering the category saves a click each time. |
|
||||
| Categories | `GET /v1/categories` **on page render**, one call. If it fails: flash the error and render the table with Add disabled. | Categories change rarely but caching them is more code than a call. |
|
||||
| Dismiss | `POST /dashboard/feeds/{id}/dismiss` → `status = 'dismissed'`, `decided_at`. No restore button. | Dismissals are cheap to undo with SQL if ever needed. |
|
||||
| Reconciliation | At each run's discovery stage, any `candidate` whose `feed_url` (or host) now appears in the subscribed feed map becomes `added` (with `miniflux_feed_id`). | Feeds added directly in Miniflux disappear from the list without manual bookkeeping. |
|
||||
| Page filter | `?status=candidate|added|dismissed` (default `candidate`), allow-listed; paginated with `_pager.html`, 50 rows. | One query param buys an audit view of what was added/dismissed. |
|
||||
| Config | `[discovery] enabled = true, max_lookups_per_run = 30, skip_hosts = [...]`. | Exactly the knobs that need turning: kill switch, run-time bound, noise list. |
|
||||
|
||||
## 4. Implementation steps
|
||||
|
||||
1. **Migration** `migrations/0010_feed_discovery.sql`:
|
||||
```sql
|
||||
CREATE TABLE feed_candidates (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
feed_url TEXT NOT NULL UNIQUE,
|
||||
host TEXT NOT NULL,
|
||||
title TEXT,
|
||||
status TEXT NOT NULL CHECK (status IN ('candidate', 'added', 'dismissed')),
|
||||
first_seen TEXT NOT NULL,
|
||||
last_seen TEXT NOT NULL,
|
||||
miniflux_feed_id INTEGER,
|
||||
decided_at TEXT
|
||||
);
|
||||
CREATE INDEX idx_feed_candidates_status_host ON feed_candidates(status, host);
|
||||
|
||||
CREATE TABLE feed_candidate_articles (
|
||||
candidate_id INTEGER NOT NULL REFERENCES feed_candidates(id) ON DELETE CASCADE,
|
||||
article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (candidate_id, article_id)
|
||||
);
|
||||
|
||||
CREATE TABLE feed_discovery_hosts (
|
||||
host TEXT PRIMARY KEY,
|
||||
checked_at TEXT NOT NULL,
|
||||
candidates INTEGER NOT NULL
|
||||
);
|
||||
```
|
||||
`Db` methods (in a new `src/discovery.rs`, not `db.rs`, following `imports.rs`): `hosts_checked(hosts) -> HashMap<host, (checked_at, candidates)>`, `mark_host_checked`, `upsert_candidate(feed_url, host, title, now) -> id` (bumps `last_seen`, never demotes `added`/`dismissed`), `link_article(candidate_id, article_id)`, `candidates_for_host(host)`, `list(status, page)`, `set_status(id, status, miniflux_feed_id, now)`, `reconcile_added(subscribed: &[(feed_url, site_host)], now)`, `count(status)`.
|
||||
|
||||
2. **Miniflux client** (`src/miniflux.rs`): add a private `post_json<B: Serialize, T: DeserializeOwned>(path, body)` mirroring `get_json` (same retry predicate; **do not retry** `create_feed` on 5xx — a retried create can double-subscribe; pass `RetryPolicy::none()` or a flag). Wire types `Discovered { url, title, type }`, `Category { id, title }`, `CreatedFeed { feed_id }`. Methods: `discover(url) -> Vec<Discovered>`, `categories() -> Vec<Category>`, `create_feed(feed_url, category_id) -> i64`. Make `MinifluxError::Status` keep the `error_message` string when the body is JSON so the flash reads "This feed already exists" rather than raw JSON. Update the module doc's "reads only" sentence. Tests: deserialize fixtures for all three.
|
||||
|
||||
3. **Discovery stage** (`src/discovery.rs` + wiring in `src/pipeline.rs` after stage 5, timing key `"discovery"`, report count `feed_candidates_new`, warning on error):
|
||||
```text
|
||||
select aggregator-only articles → drop skip_hosts / subscribed hosts
|
||||
→ reconcile_added(subscribed)
|
||||
→ for each distinct host: if checked within RECHECK_DAYS → link article to its candidates
|
||||
else queue (cap max_lookups_per_run, highest social score first)
|
||||
→ discover() ×4 concurrent (an error_message response = host checked, 0 candidates)
|
||||
→ drop "comment" results and already-subscribed feed_urls
|
||||
→ validate each remaining URL (GET, ≤256 KB, feed-root sniff, pull <title>)
|
||||
→ one link-tag hit per title (xml over json) + at most one guessed URL, max 3
|
||||
→ upsert_candidate + link_article + mark_host_checked
|
||||
```
|
||||
The function signature is `pub async fn run(db, miniflux: &MinifluxClient, http: &reqwest::Client, cfg: &DiscoveryConfig, articles: &[Article], subscribed: &HashMap<FeedId, FeedMeta>, now) -> Result<Summary>` so the pipeline and the CLI share it. Host = `url::Url::host_str()` lowercased with a leading `www.` stripped; compare subscriptions the same way. Budget: one discover call plus up to nine validation GETs per host, so with the 30-host cap the stage is bounded at ~300 small requests, 4 in flight; log the count. Pure helpers (`aggregator_only`, `host_of`, `keep_result`, `sniff_feed(&[u8]) -> Option<FeedKind>`, `feed_title(&[u8])`, `dedupe_guesses`) get unit tests, including the observed zombo.com response (nine `title == url` guesses that all fail the sniff → zero candidates), the observed blog.philz.dev response (Atom + JSON under one title → exactly the Atom candidate), and an SPA shell body (`<!doctype html>…`) rejected by `sniff_feed`.
|
||||
|
||||
4. **Ranking** (`src/discovery.rs`, pure): `pub fn score(evidence: &[ArticleEvidence]) -> f64` and `pub fn why(evidence) -> Vec<String>` per §3. Evidence loading is one query per page render: for the page's candidates, `SELECT article_id, stage, utility, signals_json FROM candidate_runs WHERE article_id IN (…) ORDER BY run_id DESC` keeping the latest row per article, joined with `current_ratings`. The page holds ≤ 50 candidates × a handful of articles each, so ranking happens in Rust after loading **all** `candidate`-status rows (sorting must be global, not per page). If the candidate count ever exceeds ~2 000 this needs a stored score; not now.
|
||||
Unit tests: rated article dominates; shrinkage (one 0.95 article scores below three 0.8s); no evidence → prior; `why` picks the modal interests.
|
||||
|
||||
5. **Dashboard page** `src/web/dashboard/feeds.rs` + `src/web/templates/dashboard/feeds.html`:
|
||||
- Routes: `GET /dashboard/feeds`, `POST /dashboard/feeds/{id}/add`, `POST /dashboard/feeds/{id}/dismiss`. Merge into `dashboard::router()`. Nav tab **Feeds** in `layout.html` (`active_nav = "feeds"`). Overview tile "Feed candidates → review" linking to the page.
|
||||
- Table columns: Score (0–100, tabular numerals), Feed (title → `feed_url`, host beneath), Why (interest names + up to 3 article titles linking to the article page), Articles (n), Seen (first/last via `format_time`), Actions (category select + Add; Dismiss). `?status=` filter links at the top. Added rows show the category and a link to the Miniflux feed page (`{base_url}/feeds/{miniflux_feed_id}`); dismissed rows show only the date.
|
||||
- Add handler: read `category_id` from the form and pass it through (Miniflux validates it), call `create_feed`. On `Ok` set status + `kv` last category + flash ok. On a 400 whose `error_message` is `This feed already exists.` set status `added` and flash that message as info. On any other `Err` flash `error_message` (or the transport error) and leave the row as `candidate`. Requires `state.config().miniflux.api_key`; if the client fails to build, flash "Miniflux API key is not configured".
|
||||
- Server test in the existing dashboard test harness: page renders with an empty table; add/dismiss flip `status` (point `miniflux.base_url` at a `tokio` listener that answers the create call, as the BookOrbit tests do for OPDS; note the loopback-bind caveat in `memory/codex-sandbox-quirk.md`).
|
||||
|
||||
6. **CLI backfill** (`src/main.rs`): `daily-epub feeds discover [--days 14] [--limit 200]` — loads `articles` with `first_seen` in the window, fetches the feed map (`MinifluxClient::feed_map`), calls `discovery::run` with `max_lookups_per_run = limit`, prints the summary. Takes the pipeline run lock (it writes the same tables the run does). No job-catalogue entry.
|
||||
|
||||
7. **Config + docs**: `DiscoveryConfig { enabled: bool, max_lookups_per_run: usize, skip_hosts: Vec<String> }` with defaults `true, 30, ["news.ycombinator.com", "lobste.rs", "reddit.com", "github.com", "gist.github.com", "x.com", "twitter.com", "youtube.com", "en.wikipedia.org", "arxiv.org", "docs.google.com"]`. Register the section in the settings dashboard (list, help text, test). Document in `config.example.toml` and the README table. Add a short "Feeds" paragraph to the dashboard section of the README.
|
||||
|
||||
Order matters only for 1 → 2 → 3 → 4 → 5; 6 and 7 can be done alongside. Expect roughly 900 lines including tests.
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- `cargo test` green; new pure-function tests in `discovery.rs`, wire-type tests in `miniflux.rs`, one handler test for the page.
|
||||
- Live, on the server after deploy: run `daily-epub feeds discover --days 14 --limit 50`, open `/dashboard/feeds`, confirm the top rows look like feeds the operator would actually want, click **Add** on one, confirm it appears in Miniflux under the chosen category and the row flips to *added*. Run `generate --dry-run` and check the report has a `discovery` timing under ~60 s and no warning.
|
||||
- Check the `journalctl` line for how many hosts were skipped as already subscribed; if it is near zero the `www.`/host normalisation is wrong.
|
||||
- Check the line for discover results rejected by validation. A healthy day should reject a good share of them (SPA catch-alls); zero rejections means the sniff is too lenient, and near-total rejection means it is too strict (look for feeds served with a leading comment or DOCTYPE).
|
||||
|
||||
## 6. Rollout notes
|
||||
|
||||
- Migrations run automatically on the first `daily-epub` command after deploy.
|
||||
- No new secrets: the existing `DAILY_EPUB_MINIFLUX__API_KEY` can create feeds.
|
||||
- Seed the page once with the CLI backfill so it is useful on day one.
|
||||
- Every Miniflux behaviour the design depends on was observed live on 2026-09-07 (§2); the recorded responses are the test fixtures. Nothing remains to confirm before implementation.
|
||||
|
||||
## 7. Non-goals (do not add)
|
||||
|
||||
- Auto-subscribing without a click, or any LLM judgement of feeds.
|
||||
- Fetching or previewing feed contents, feed health checks, unread counts.
|
||||
- Un-dismiss / undo buttons, per-row notes, bulk actions, sorting controls (the list is already sorted by the only order that matters).
|
||||
- A stored/cached score column, a scheduled job, OPML export, Scour "recommended feeds" integration (ideas.md item, separate).
|
||||
- Configurable ranking constants.
|
||||
@@ -0,0 +1,230 @@
|
||||
# Operator and developer shortcuts for The Daily EPUB.
|
||||
#
|
||||
# just list recipes
|
||||
# just de <args> run any daily-epub subcommand on the server as the service user
|
||||
#
|
||||
# Server recipes run the binary the way the systemd units do: same user, working
|
||||
# directory and EnvironmentFile (`/etc/daily-epub/env`, where the API keys live),
|
||||
# via a transient `systemd-run` unit. Plain `sudo -u daily-epub` never loads that
|
||||
# file, so `miniflux api key is not configured` from a hand-run command means the
|
||||
# environment, not the config.
|
||||
|
||||
set positional-arguments
|
||||
|
||||
bin := "/usr/local/bin/daily-epub"
|
||||
config := "/etc/daily-epub/config.toml"
|
||||
env_file := "/etc/daily-epub/env"
|
||||
work_dir := "/var/lib/daily-epub"
|
||||
user := "daily-epub"
|
||||
|
||||
# The systemd-run prefix every server recipe uses.
|
||||
run := "sudo systemd-run --quiet --wait --pty --collect --uid=" + user + " --gid=" + user \
|
||||
+ " -p WorkingDirectory=" + work_dir + " -p EnvironmentFile=" + env_file \
|
||||
+ " " + bin + " --config " + config
|
||||
|
||||
default:
|
||||
@just --list --unsorted
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server: any subcommand
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Run any daily-epub subcommand as the service user with the env file loaded.
|
||||
de *args:
|
||||
{{run}} "$@"
|
||||
|
||||
# Validate the config as `generate` would; one fact per line.
|
||||
config-check:
|
||||
{{run}} config check
|
||||
|
||||
# Run pending migrations (also automatic on every start).
|
||||
db-migrate:
|
||||
{{run}} db migrate
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server: issues
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Build and publish an issue by hand (same as the timer, but in the foreground).
|
||||
generate *args:
|
||||
{{run}} generate "$@"
|
||||
|
||||
# Build everything, publish nothing, record no issue.
|
||||
dry-run *args:
|
||||
{{run}} generate --dry-run "$@"
|
||||
|
||||
# Why an article was (or was not) in the paper, from persisted telemetry.
|
||||
explain *args:
|
||||
{{run}} explain "$@"
|
||||
|
||||
# The weekly numbers: issues, ratings, retriever yield, cost, timing.
|
||||
stats *args:
|
||||
{{run}} stats "$@"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server: feeds, ratings, profile, features
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Seed /dashboard/feeds from recent aggregator-only articles.
|
||||
feeds-discover days="14" limit="50":
|
||||
{{run}} feeds discover --days {{days}} --limit {{limit}}
|
||||
|
||||
# List current explicit verdicts.
|
||||
ratings-list *args:
|
||||
{{run}} ratings list "$@"
|
||||
|
||||
# Set a verdict: `just ratings-set <article-id> <loved|good|not_for_me> [--note ...]`.
|
||||
ratings-set *args:
|
||||
{{run}} ratings set "$@"
|
||||
|
||||
# Clear a verdict: `just ratings-clear <article-id>`.
|
||||
ratings-clear *args:
|
||||
{{run}} ratings clear "$@"
|
||||
|
||||
# Regenerate the taste profile from ratings history.
|
||||
profile-rebuild:
|
||||
{{run}} profile rebuild
|
||||
|
||||
# Embed rated and published articles, then interests, into the cache.
|
||||
features-backfill *args:
|
||||
{{run}} features backfill "$@"
|
||||
|
||||
# Drop stale embeddings, old telemetry and old assessments per the retention config.
|
||||
features-prune:
|
||||
{{run}} features prune
|
||||
|
||||
# Re-poll social scores for recent entries.
|
||||
backfill-social days="7":
|
||||
{{run}} backfill-social --days {{days}}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server: users and jobs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# List dashboard users and open-session counts.
|
||||
users-list:
|
||||
{{run}} users list
|
||||
|
||||
# Add a user: `just users-add NAME [--admin]` (prompts for the password).
|
||||
users-add *args:
|
||||
{{run}} users add "$@"
|
||||
|
||||
# Change a password and revoke sessions: `just users-passwd NAME`.
|
||||
users-passwd *args:
|
||||
{{run}} users passwd "$@"
|
||||
|
||||
# Any other `users` subcommand: `just users role NAME admin`, `just users disable NAME`, ...
|
||||
users *args:
|
||||
{{run}} users "$@"
|
||||
|
||||
# Run one catalogue job in-process (generate, dry-run, profile-rebuild, features-backfill, backfill-social, features-prune, import-ratings).
|
||||
job name:
|
||||
{{run}} job run {{name}}
|
||||
|
||||
# Start a catalogue job as its systemd unit, the way the dashboard does.
|
||||
job-unit name:
|
||||
sudo systemctl start daily-epub-job@{{name}}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server: services and logs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Status of the web server, the generate timer and its last run.
|
||||
status:
|
||||
systemctl status --no-pager daily-epub.service daily-epub-generate.timer daily-epub-generate.service || true
|
||||
systemctl list-timers --no-pager daily-epub-generate.timer
|
||||
|
||||
# Restart the web server (after installing a new binary or unit).
|
||||
restart:
|
||||
sudo systemctl restart daily-epub.service
|
||||
|
||||
# Follow the web server log.
|
||||
logs:
|
||||
journalctl -u daily-epub.service -f
|
||||
|
||||
# Last N lines of the most recent generate run.
|
||||
logs-generate lines="120":
|
||||
journalctl -u daily-epub-generate.service -n {{lines}} --no-pager
|
||||
|
||||
# Follow one job unit's log: `just logs-job features-prune`.
|
||||
logs-job name:
|
||||
journalctl -u daily-epub-job@{{name}} -f
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deploy (run from a checkout on the server)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Release build, install the binary and restart the web server.
|
||||
deploy: build-release
|
||||
sudo install -m0755 target/release/daily-epub {{bin}}
|
||||
sudo systemctl restart daily-epub.service
|
||||
systemctl is-active daily-epub.service
|
||||
|
||||
# Install the systemd units and the polkit rule, then reload systemd.
|
||||
install-units:
|
||||
sudo install -m0644 systemd/daily-epub.service systemd/daily-epub-generate.service \
|
||||
systemd/daily-epub-generate.timer systemd/daily-epub-job@.service /etc/systemd/system/
|
||||
sudo install -m0644 systemd/50-daily-epub.rules /etc/polkit-1/rules.d/
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Edit the environment file (API keys and other secrets).
|
||||
edit-env:
|
||||
sudo -e {{env_file}}
|
||||
|
||||
# Edit the config file as the service user would see it.
|
||||
edit-config:
|
||||
sudo -e {{config}}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Development (local checkout)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
build:
|
||||
cargo build
|
||||
|
||||
build-release:
|
||||
cargo build --release
|
||||
|
||||
# Everything CI expects: format check, clippy, tests, and the committed CSS is current.
|
||||
check: fmt-check clippy test css-check
|
||||
|
||||
test *args:
|
||||
cargo test "$@"
|
||||
|
||||
clippy:
|
||||
cargo clippy --all-targets
|
||||
|
||||
fmt:
|
||||
cargo fmt
|
||||
|
||||
fmt-check:
|
||||
cargo fmt --check
|
||||
|
||||
# Rebuild src/web/static/app.css from the Tailwind source (commit the result).
|
||||
css:
|
||||
npm run css
|
||||
|
||||
# Fail if the committed app.css is out of date.
|
||||
css-check:
|
||||
npm run css:check
|
||||
|
||||
# Rebuild app.css on every template change.
|
||||
css-watch:
|
||||
npm run css:watch
|
||||
|
||||
# Seed a throwaway dev database under ./dev (admin/adminpassword123, reader/readerpassword123).
|
||||
dev-seed:
|
||||
cargo run --example seed_dev_db -- ./dev
|
||||
|
||||
# Serve the dev database on http://127.0.0.1:3599.
|
||||
dev-serve:
|
||||
DAILY_EPUB_SERVER__HMAC_SECRET=$(head -c 48 /dev/urandom | base64) \
|
||||
cargo run -- --config ./dev/config.toml serve
|
||||
|
||||
# Run any subcommand against the local dev config: `just dev config check`.
|
||||
dev *args:
|
||||
cargo run -- --config ./dev/config.toml "$@"
|
||||
|
||||
# Replay extraction + images over published EPUBs: `just image-audit ~/books/*.epub`.
|
||||
image-audit *args:
|
||||
cargo run --release --example image_audit -- --cache /tmp/pagecache "$@"
|
||||
@@ -0,0 +1,27 @@
|
||||
-- Feed discovery: proposed Miniflux subscriptions found behind aggregator-only
|
||||
-- articles (feed discovery plan §4 step 1).
|
||||
|
||||
CREATE TABLE feed_candidates (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
feed_url TEXT NOT NULL UNIQUE,
|
||||
host TEXT NOT NULL,
|
||||
title TEXT,
|
||||
status TEXT NOT NULL CHECK (status IN ('candidate', 'added', 'dismissed')),
|
||||
first_seen TEXT NOT NULL,
|
||||
last_seen TEXT NOT NULL,
|
||||
miniflux_feed_id INTEGER,
|
||||
decided_at TEXT
|
||||
);
|
||||
CREATE INDEX idx_feed_candidates_status_host ON feed_candidates(status, host);
|
||||
|
||||
CREATE TABLE feed_candidate_articles (
|
||||
candidate_id INTEGER NOT NULL REFERENCES feed_candidates(id) ON DELETE CASCADE,
|
||||
article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (candidate_id, article_id)
|
||||
);
|
||||
|
||||
CREATE TABLE feed_discovery_hosts (
|
||||
host TEXT PRIMARY KEY,
|
||||
checked_at TEXT NOT NULL,
|
||||
candidates INTEGER NOT NULL
|
||||
);
|
||||
@@ -84,6 +84,8 @@ pub struct Config {
|
||||
pub server: ServerConfig,
|
||||
pub bookorbit: BookorbitConfig,
|
||||
pub mail: MailConfig,
|
||||
/// `[discovery]` — propose new feed subscriptions from aggregator hits.
|
||||
pub discovery: DiscoveryConfig,
|
||||
}
|
||||
|
||||
impl Default for Config {
|
||||
@@ -110,6 +112,7 @@ impl Default for Config {
|
||||
server: ServerConfig::default(),
|
||||
bookorbit: BookorbitConfig::default(),
|
||||
mail: MailConfig::default(),
|
||||
discovery: DiscoveryConfig::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -823,6 +826,44 @@ impl Default for MailConfig {
|
||||
}
|
||||
}
|
||||
|
||||
/// `[discovery]` — feed discovery from aggregator-only articles (feed
|
||||
/// discovery plan §3).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields, default)]
|
||||
pub struct DiscoveryConfig {
|
||||
/// Whether the discovery stage runs during `generate`.
|
||||
pub enabled: bool,
|
||||
/// How many not-yet-checked hosts one run may look up in Miniflux.
|
||||
pub max_lookups_per_run: usize,
|
||||
/// Hosts never looked up (aggregators, code hosts, social networks).
|
||||
pub skip_hosts: Vec<String>,
|
||||
}
|
||||
|
||||
impl Default for DiscoveryConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: true,
|
||||
max_lookups_per_run: 30,
|
||||
skip_hosts: [
|
||||
"news.ycombinator.com",
|
||||
"lobste.rs",
|
||||
"reddit.com",
|
||||
"github.com",
|
||||
"gist.github.com",
|
||||
"x.com",
|
||||
"twitter.com",
|
||||
"youtube.com",
|
||||
"en.wikipedia.org",
|
||||
"arxiv.org",
|
||||
"docs.google.com",
|
||||
]
|
||||
.into_iter()
|
||||
.map(String::from)
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl MailConfig {
|
||||
/// Whether mail is enabled with all fields required for SMTP delivery.
|
||||
pub fn is_active(&self) -> bool {
|
||||
|
||||
+1577
File diff suppressed because it is too large.
Load diff
@@ -19,6 +19,7 @@ pub mod config;
|
||||
pub mod curate;
|
||||
pub mod db;
|
||||
pub mod dedupe;
|
||||
pub mod discovery;
|
||||
pub mod epub;
|
||||
pub mod extract;
|
||||
pub mod html;
|
||||
|
||||
+60
-1
@@ -17,7 +17,7 @@ use daily_epub::db::Db;
|
||||
use daily_epub::pipeline::{self, GenerateOptions, GenerateOutcome};
|
||||
use daily_epub::report::{RunReport, VOYAGE_PROVIDER};
|
||||
use daily_epub::types::{ArticleId, Vote};
|
||||
use daily_epub::{curate, http, imports, jobs, lock, rate, server, social};
|
||||
use daily_epub::{curate, discovery, http, imports, jobs, lock, rate, server, social};
|
||||
|
||||
/// A personalized daily newspaper, delivered as an EPUB.
|
||||
#[derive(Debug, Parser)]
|
||||
@@ -61,6 +61,9 @@ enum Command {
|
||||
/// Manage dashboard users without taking the pipeline run lock.
|
||||
#[command(subcommand)]
|
||||
Users(UsersCommand),
|
||||
/// Feed subscription candidates (feed discovery plan §4 step 6).
|
||||
#[command(subcommand)]
|
||||
Feeds(FeedsCommand),
|
||||
/// Operator jobs (what `daily-epub-job@<name>.service` runs).
|
||||
#[command(subcommand)]
|
||||
Job(JobCommand),
|
||||
@@ -76,6 +79,23 @@ enum JobCommand {
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, Subcommand)]
|
||||
enum FeedsCommand {
|
||||
/// Look for feeds behind recent aggregator-only articles and record the
|
||||
/// ones not already subscribed as candidates for `/dashboard/feeds`.
|
||||
Discover(FeedsDiscoverArgs),
|
||||
}
|
||||
|
||||
#[derive(Debug, clap::Args)]
|
||||
struct FeedsDiscoverArgs {
|
||||
/// How far back to look for articles.
|
||||
#[arg(long, default_value_t = 14)]
|
||||
days: i64,
|
||||
/// How many not-yet-checked hosts to look up.
|
||||
#[arg(long, default_value_t = 200)]
|
||||
limit: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Subcommand)]
|
||||
enum UsersCommand {
|
||||
/// Add a user.
|
||||
@@ -381,6 +401,10 @@ async fn main() -> Result<()> {
|
||||
let db = Db::open_and_migrate(&config.database_path).await?;
|
||||
cmd_users(&db, command).await?;
|
||||
}
|
||||
Command::Feeds(FeedsCommand::Discover(args)) => {
|
||||
let db = Db::open_and_migrate(&config.database_path).await?;
|
||||
println!("{}", cmd_feeds_discover(&config, &db, args).await?);
|
||||
}
|
||||
Command::Job(JobCommand::Run { name }) => {
|
||||
let Some(job) = jobs::Job::parse(&name) else {
|
||||
eprintln!("unknown job {name:?}; the catalogue is:");
|
||||
@@ -411,6 +435,8 @@ async fn main() -> Result<()> {
|
||||
fn lock_holder(command: &Command) -> Option<&'static str> {
|
||||
match command {
|
||||
Command::Generate(_) => Some("generate"),
|
||||
// Writes the same tables the discovery stage of a run writes.
|
||||
Command::Feeds(FeedsCommand::Discover(_)) => Some("feeds discover"),
|
||||
Command::Profile(ProfileCommand::Rebuild) => Some("profile rebuild"),
|
||||
Command::Features(FeaturesCommand::Backfill(_)) => Some("features backfill"),
|
||||
Command::BackfillSocial(_) => Some("backfill-social"),
|
||||
@@ -430,6 +456,39 @@ fn lock_holder(command: &Command) -> Option<&'static str> {
|
||||
}
|
||||
}
|
||||
|
||||
/// `daily-epub feeds discover` — the same stage `generate` runs, over the last
|
||||
/// `--days` of articles, so the dashboard page is useful before tomorrow's run.
|
||||
async fn cmd_feeds_discover(config: &Config, db: &Db, args: FeedsDiscoverArgs) -> Result<String> {
|
||||
let since = jiff::Timestamp::now()
|
||||
.checked_sub(jiff::Span::new().hours(args.days.max(0).saturating_mul(24)))
|
||||
.unwrap_or(jiff::Timestamp::UNIX_EPOCH);
|
||||
let articles = discovery::articles_since(db, since).await?;
|
||||
let http = http::build_client(http::DEFAULT_TIMEOUT).context("building http client")?;
|
||||
let client = daily_epub::miniflux::MinifluxClient::new(&config.miniflux, http.clone())
|
||||
.context("constructing the miniflux client")?;
|
||||
let feeds = client
|
||||
.feed_map()
|
||||
.await
|
||||
.context("loading the miniflux feed list")?;
|
||||
let mut cfg = config.discovery.clone();
|
||||
cfg.max_lookups_per_run = args.limit;
|
||||
let summary = discovery::run(
|
||||
db,
|
||||
&client,
|
||||
&http,
|
||||
&cfg,
|
||||
&articles,
|
||||
&feeds,
|
||||
jiff::Timestamp::now(),
|
||||
)
|
||||
.await?;
|
||||
Ok(format!(
|
||||
"{} articles since {}: {summary}",
|
||||
articles.len(),
|
||||
since
|
||||
))
|
||||
}
|
||||
|
||||
async fn cmd_users(db: &Db, command: UsersCommand) -> Result<()> {
|
||||
use daily_epub::web::users;
|
||||
match command {
|
||||
|
||||
+230
-21
@@ -1,13 +1,16 @@
|
||||
//! Miniflux API client (spec §3.1).
|
||||
//!
|
||||
//! Reads only: entries are fetched with `published_after` inside the lookback
|
||||
//! window **regardless of read/unread status**, and read state is never mutated
|
||||
//! so normal reader usage is undisturbed.
|
||||
//! Almost reads only: entries are fetched with `published_after` inside the
|
||||
//! lookback window **regardless of read/unread status**, and read state is never
|
||||
//! mutated so normal reader usage is undisturbed. The single writing call is
|
||||
//! [`MinifluxClient::create_feed`], which the dashboard's feed-discovery page
|
||||
//! uses to subscribe to a candidate the operator picked (feed discovery plan
|
||||
//! §4 step 2); the pipeline never calls it.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use jiff::Timestamp;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde::{Deserialize, Serialize, de::DeserializeOwned};
|
||||
|
||||
use crate::config::MinifluxConfig;
|
||||
use crate::http::{RetryPolicy, is_retryable};
|
||||
@@ -24,11 +27,13 @@ pub enum MinifluxError {
|
||||
MissingApiKey,
|
||||
#[error("miniflux request failed: {0}")]
|
||||
Http(#[from] reqwest::Error),
|
||||
#[error("miniflux returned {status} for {path}: {body}")]
|
||||
#[error("miniflux returned {status} for {path}: {message}")]
|
||||
Status {
|
||||
status: u16,
|
||||
path: String,
|
||||
body: String,
|
||||
/// `error_message` when the body was Miniflux's JSON error envelope,
|
||||
/// else the truncated body — what a dashboard flash should show.
|
||||
message: String,
|
||||
},
|
||||
#[error("could not parse miniflux response for {path}: {source}")]
|
||||
Decode {
|
||||
@@ -40,6 +45,46 @@ pub enum MinifluxError {
|
||||
|
||||
type Result<T> = std::result::Result<T, MinifluxError>;
|
||||
|
||||
impl MinifluxError {
|
||||
/// Build a [`MinifluxError::Status`], pulling `error_message` out of
|
||||
/// Miniflux's JSON error envelope when the body has one (§4 step 2).
|
||||
fn status(status: u16, path: &str, body: &str) -> Self {
|
||||
#[derive(Deserialize)]
|
||||
struct ErrorBody {
|
||||
error_message: String,
|
||||
}
|
||||
let body: String = body.chars().take(300).collect();
|
||||
let message = serde_json::from_str::<ErrorBody>(&body)
|
||||
.map(|e| e.error_message)
|
||||
.unwrap_or_else(|_| body.clone());
|
||||
MinifluxError::Status {
|
||||
status,
|
||||
path: path.to_string(),
|
||||
message,
|
||||
}
|
||||
}
|
||||
|
||||
/// True for the `400 This feed already exists.` a duplicate `POST /v1/feeds`
|
||||
/// returns — the dashboard flips the row to `added` on it instead of
|
||||
/// treating it as a failure.
|
||||
pub fn is_duplicate_feed(&self) -> bool {
|
||||
matches!(
|
||||
self,
|
||||
MinifluxError::Status { status: 400, message, .. }
|
||||
if message.trim().eq_ignore_ascii_case("This feed already exists.")
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// Network failures and 5xx/429 responses are worth another attempt (§3.1).
|
||||
fn retryable(error: &MinifluxError) -> bool {
|
||||
match error {
|
||||
MinifluxError::Http(e) => is_retryable(e),
|
||||
MinifluxError::Status { status, .. } => *status >= 500 || *status == 429,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Wire types (only the fields §3.1 lists as used)
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -106,6 +151,27 @@ pub struct EntriesResponse {
|
||||
pub entries: Vec<MinifluxEntry>,
|
||||
}
|
||||
|
||||
/// `POST /v1/discover` element — a *lead*, not a verified feed: Miniflux
|
||||
/// returns well-known paths that merely answered 200 (plan §2), so every one is
|
||||
/// validated before it becomes a candidate.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Discovered {
|
||||
#[serde(default)]
|
||||
pub url: String,
|
||||
/// Miniflux substitutes the URL when the `<link>` tag has no title, which is
|
||||
/// how a well-known-path guess is told apart from a real link-tag hit.
|
||||
#[serde(default)]
|
||||
pub title: String,
|
||||
#[serde(default, rename = "type")]
|
||||
pub kind: String,
|
||||
}
|
||||
|
||||
/// `POST /v1/feeds` response.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct CreatedFeed {
|
||||
pub feed_id: i64,
|
||||
}
|
||||
|
||||
/// Feed metadata joined onto every entry we persist (§3.1).
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct FeedMeta {
|
||||
@@ -247,14 +313,7 @@ impl MinifluxClient {
|
||||
let url = self.url(path);
|
||||
let body = self
|
||||
.retry
|
||||
.run(
|
||||
&format!("GET {path}"),
|
||||
|e: &MinifluxError| match e {
|
||||
MinifluxError::Http(e) => is_retryable(e),
|
||||
MinifluxError::Status { status, .. } => *status >= 500 || *status == 429,
|
||||
_ => false,
|
||||
},
|
||||
|| async {
|
||||
.run(&format!("GET {path}"), retryable, || async {
|
||||
let resp = self
|
||||
.http
|
||||
.get(&url)
|
||||
@@ -266,15 +325,10 @@ impl MinifluxClient {
|
||||
let status = resp.status();
|
||||
let text = resp.text().await?;
|
||||
if !status.is_success() {
|
||||
return Err(MinifluxError::Status {
|
||||
status: status.as_u16(),
|
||||
path: path.to_string(),
|
||||
body: text.chars().take(300).collect(),
|
||||
});
|
||||
return Err(MinifluxError::status(status.as_u16(), path, &text));
|
||||
}
|
||||
Ok(text)
|
||||
},
|
||||
)
|
||||
})
|
||||
.await?;
|
||||
serde_json::from_str(&body).map_err(|source| MinifluxError::Decode {
|
||||
path: path.to_string(),
|
||||
@@ -282,6 +336,74 @@ impl MinifluxClient {
|
||||
})
|
||||
}
|
||||
|
||||
/// One `POST path` with a JSON body, no retry.
|
||||
async fn post_once<B: Serialize>(&self, path: &str, body: &B) -> Result<String> {
|
||||
let resp = self
|
||||
.http
|
||||
.post(self.url(path))
|
||||
.header("X-Auth-Token", &self.api_key)
|
||||
.header("Accept", "application/json")
|
||||
.json(body)
|
||||
.send()
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let text = resp.text().await?;
|
||||
if !status.is_success() {
|
||||
return Err(MinifluxError::status(status.as_u16(), path, &text));
|
||||
}
|
||||
Ok(text)
|
||||
}
|
||||
|
||||
/// POST `path` with the auth header, mirroring [`Self::get_json`].
|
||||
///
|
||||
/// `retry` is false for calls that are not safe to repeat: a retried
|
||||
/// `POST /v1/feeds` whose response was merely lost would subscribe twice.
|
||||
async fn post_json<B: Serialize, T: DeserializeOwned>(
|
||||
&self,
|
||||
path: &str,
|
||||
body: &B,
|
||||
retry: bool,
|
||||
) -> Result<T> {
|
||||
let text = if retry {
|
||||
self.retry
|
||||
.run(&format!("POST {path}"), retryable, || {
|
||||
self.post_once(path, body)
|
||||
})
|
||||
.await?
|
||||
} else {
|
||||
self.post_once(path, body).await?
|
||||
};
|
||||
serde_json::from_str(&text).map_err(|source| MinifluxError::Decode {
|
||||
path: path.to_string(),
|
||||
source,
|
||||
})
|
||||
}
|
||||
|
||||
/// `POST /v1/discover` — the feed(s) Miniflux can find behind a page URL
|
||||
/// (feed discovery plan §4 step 2). Retried: it changes nothing.
|
||||
pub async fn discover(&self, url: &str) -> Result<Vec<Discovered>> {
|
||||
self.post_json("/discover", &serde_json::json!({ "url": url }), true)
|
||||
.await
|
||||
}
|
||||
|
||||
/// `GET /v1/categories` — the Add form's category picker.
|
||||
pub async fn categories(&self) -> Result<Vec<MinifluxCategory>> {
|
||||
self.get_json("/categories", &[]).await
|
||||
}
|
||||
|
||||
/// `POST /v1/feeds` — subscribe to `feed_url` in `category_id`, returning
|
||||
/// the new feed id. Never retried (a repeat can double-subscribe).
|
||||
pub async fn create_feed(&self, feed_url: &str, category_id: i64) -> Result<i64> {
|
||||
let created: CreatedFeed = self
|
||||
.post_json(
|
||||
"/feeds",
|
||||
&serde_json::json!({ "feed_url": feed_url, "category_id": category_id }),
|
||||
false,
|
||||
)
|
||||
.await?;
|
||||
Ok(created.feed_id)
|
||||
}
|
||||
|
||||
/// `GET /v1/feeds` — once per run (§3.1).
|
||||
pub async fn feeds(&self) -> Result<Vec<MinifluxFeed>> {
|
||||
self.get_json("/feeds", &[]).await
|
||||
@@ -483,6 +605,93 @@ mod tests {
|
||||
assert_eq!(entries[1].category, None);
|
||||
}
|
||||
|
||||
/// Observed live for an article URL whose page carries `<link rel=alternate>`
|
||||
/// tags: the same feed offered twice under one title (plan §2).
|
||||
const DISCOVER_LINK_TAGS_JSON: &str = r#"[
|
||||
{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.xml","type":"atom"},
|
||||
{"title":"blog.philz.dev","url":"https://blog.philz.dev/feed/feed.json","type":"json"}
|
||||
]"#;
|
||||
|
||||
/// Observed live for a site that answers 200 for any path: nine unverified
|
||||
/// well-known-path guesses, each with `title == url` (plan §2).
|
||||
const DISCOVER_GUESSES_JSON: &str = r#"[
|
||||
{"title":"https://zombo.com/atom.xml","url":"https://zombo.com/atom.xml","type":"atom"},
|
||||
{"title":"https://zombo.com/feed.atom","url":"https://zombo.com/feed.atom","type":"atom"},
|
||||
{"title":"https://zombo.com/feed.xml","url":"https://zombo.com/feed.xml","type":"atom"},
|
||||
{"title":"https://zombo.com/feed/","url":"https://zombo.com/feed/","type":"atom"},
|
||||
{"title":"https://zombo.com/index.rss","url":"https://zombo.com/index.rss","type":"rss"},
|
||||
{"title":"https://zombo.com/index.xml","url":"https://zombo.com/index.xml","type":"rss"},
|
||||
{"title":"https://zombo.com/rss.xml","url":"https://zombo.com/rss.xml","type":"rss"},
|
||||
{"title":"https://zombo.com/rss/","url":"https://zombo.com/rss/","type":"rss"},
|
||||
{"title":"https://zombo.com/rss/feed.xml","url":"https://zombo.com/rss/feed.xml","type":"rss"}
|
||||
]"#;
|
||||
|
||||
const CATEGORIES_JSON: &str = r#"[
|
||||
{"id": 1, "user_id": 1, "title": "Tech", "hide_globally": false},
|
||||
{"id": 4, "user_id": 1, "title": "Long reads", "hide_globally": true}
|
||||
]"#;
|
||||
|
||||
#[test]
|
||||
fn deserializes_discover_results() {
|
||||
let hits: Vec<Discovered> = serde_json::from_str(DISCOVER_LINK_TAGS_JSON).unwrap();
|
||||
assert_eq!(hits.len(), 2);
|
||||
assert_eq!(hits[0].title, "blog.philz.dev");
|
||||
assert_eq!(hits[0].url, "https://blog.philz.dev/feed/feed.xml");
|
||||
assert_eq!(hits[0].kind, "atom");
|
||||
assert_eq!(hits[1].kind, "json");
|
||||
|
||||
let guesses: Vec<Discovered> = serde_json::from_str(DISCOVER_GUESSES_JSON).unwrap();
|
||||
assert_eq!(guesses.len(), 9);
|
||||
assert!(guesses.iter().all(|g| g.title == g.url));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deserializes_categories_and_created_feed() {
|
||||
let categories: Vec<MinifluxCategory> = serde_json::from_str(CATEGORIES_JSON).unwrap();
|
||||
assert_eq!(categories.len(), 2);
|
||||
assert_eq!(categories[1].id, 4);
|
||||
assert_eq!(categories[1].title, "Long reads");
|
||||
|
||||
let created: CreatedFeed = serde_json::from_str(r#"{"feed_id": 123}"#).unwrap();
|
||||
assert_eq!(created.feed_id, 123);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn status_errors_carry_the_error_message() {
|
||||
let duplicate = MinifluxError::status(
|
||||
400,
|
||||
"/feeds",
|
||||
r#"{"error_message":"This feed already exists."}"#,
|
||||
);
|
||||
assert!(duplicate.is_duplicate_feed());
|
||||
assert_eq!(
|
||||
duplicate.to_string(),
|
||||
"miniflux returned 400 for /feeds: This feed already exists."
|
||||
);
|
||||
|
||||
let fetcher = MinifluxError::status(
|
||||
502,
|
||||
"/discover",
|
||||
r#"{"error_message":"fetcher: bad gateway (502 status code)"}"#,
|
||||
);
|
||||
assert!(!fetcher.is_duplicate_feed());
|
||||
assert!(retryable(&fetcher));
|
||||
match fetcher {
|
||||
MinifluxError::Status { message, .. } => {
|
||||
assert_eq!(message, "fetcher: bad gateway (502 status code)");
|
||||
}
|
||||
other => panic!("unexpected error {other:?}"),
|
||||
}
|
||||
|
||||
// A non-JSON body falls back to the truncated body itself.
|
||||
let html = MinifluxError::status(404, "/discover", "<html>nope</html>");
|
||||
assert!(!retryable(&html));
|
||||
assert_eq!(
|
||||
html.to_string(),
|
||||
"miniflux returned 404 for /discover: <html>nope</html>"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn requires_an_api_key() {
|
||||
let http = crate::http::build_client(crate::http::DEFAULT_TIMEOUT).unwrap();
|
||||
|
||||
+27
-1
@@ -46,7 +46,7 @@ use crate::types::{
|
||||
Article, ArticleId, Artifact, BehindThePaper, Candidate, Colophon, Edition, Issue, IssueMeta,
|
||||
Lineup, Models, TokenUsage, reading_minutes,
|
||||
};
|
||||
use crate::{comments, dedupe, epub, http, miniflux, publish, social, world};
|
||||
use crate::{comments, dedupe, discovery, epub, http, miniflux, publish, social, world};
|
||||
|
||||
/// One `generate` invocation's inputs — the CLI flags, already parsed (§2).
|
||||
#[derive(Debug, Clone, Default)]
|
||||
@@ -449,6 +449,32 @@ async fn run_stages(
|
||||
report.counts.social_hits = enricher.enrich_all(&mut articles).await as i64;
|
||||
report.timings.record("social", elapsed_ms(stage));
|
||||
|
||||
// --- Stage 5b: feed discovery (feed discovery plan §4) — best effort ---
|
||||
// Runs here because it needs the real article ids stage 4 minted and the
|
||||
// subscription map stage 1 already loaded, and because `articles` is moved
|
||||
// into hygiene next.
|
||||
if config.discovery.enabled {
|
||||
let stage = Timestamp::now();
|
||||
match discovery::run(
|
||||
db,
|
||||
&client,
|
||||
&http,
|
||||
&config.discovery,
|
||||
&articles,
|
||||
&feeds,
|
||||
Timestamp::now(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(summary) => {
|
||||
tracing::info!(%summary, "feed discovery");
|
||||
report.counts.feed_candidates_new = summary.candidates_new as i64;
|
||||
}
|
||||
Err(error) => report.warn(format!("feed discovery failed: {error:#}")),
|
||||
}
|
||||
report.timings.record("discovery", elapsed_ms(stage));
|
||||
}
|
||||
|
||||
// --- Stage 6: hygiene, embeddings, and cheap signals (§8.1, §9) ---
|
||||
let stage = Timestamp::now();
|
||||
let mut personalized = admit::hygiene(
|
||||
|
||||
@@ -67,6 +67,10 @@ pub struct StageCounts {
|
||||
pub excerpt_only: i64,
|
||||
/// Social lookups that returned a hit (§3.4).
|
||||
pub social_hits: i64,
|
||||
/// Feed candidates recorded for the first time by the discovery stage
|
||||
/// (feed discovery plan §4 step 3).
|
||||
#[serde(default)]
|
||||
pub feed_candidates_new: i64,
|
||||
/// Articles passing hygiene and eligible for personalized signals.
|
||||
pub eligible: i64,
|
||||
/// Eligible articles with a valid embedding.
|
||||
|
||||
@@ -0,0 +1,639 @@
|
||||
//! Dashboard: feed candidates (`/dashboard/feeds`, feed discovery plan §4
|
||||
//! step 5).
|
||||
//!
|
||||
//! The discovery stage records the feeds behind aggregator-only articles in
|
||||
//! `feed_candidates`; this page ranks the undecided ones by how likely the
|
||||
//! operator is to enjoy them ([`discovery::score`]) and offers **Add** (which
|
||||
//! subscribes through `POST /v1/feeds`) and **Dismiss**.
|
||||
//!
|
||||
//! Ranking is global, so the candidate view loads every undecided row, scores
|
||||
//! it in Rust and paginates afterwards (plan §4 step 4). The decided views are
|
||||
//! ordinary SQL pages.
|
||||
|
||||
use std::cmp::Ordering;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use askama::Template;
|
||||
use axum::Router;
|
||||
use axum::extract::{Extension, Form, Path, Query, State};
|
||||
use axum::response::{IntoResponse, Redirect, Response};
|
||||
use axum::routing::{get, post};
|
||||
use axum_login::tower_sessions::Session;
|
||||
use jiff::Timestamp;
|
||||
use serde::Deserialize;
|
||||
use sqlx::Row as _;
|
||||
|
||||
use super::jobs::set_flash;
|
||||
use super::{Pager, allow_listed, db_err, dynamic_query, fmt_stored_time, page_number};
|
||||
use crate::config::Config;
|
||||
use crate::db::Db;
|
||||
use crate::discovery::{self, ArticleEvidence, Candidate};
|
||||
use crate::miniflux::{MinifluxCategory, MinifluxClient};
|
||||
use crate::server::AppState;
|
||||
use crate::types::ArticleId;
|
||||
use crate::web::session::{AuthSession, Viewer};
|
||||
use crate::web::{Html, Page, Pagination, WebError, take_flash};
|
||||
|
||||
const CANDIDATES_PER_PAGE: u32 = 50;
|
||||
/// `feed_candidates.status` values, allow-listed for `?status=`.
|
||||
const STATUSES: [&str; 3] = ["candidate", "added", "dismissed"];
|
||||
/// Articles named in the "Why" column.
|
||||
const WHY_ARTICLES: usize = 3;
|
||||
/// `kv` key remembering the category the last Add used (plan §3).
|
||||
const LAST_CATEGORY_KEY: &str = "feed_discovery_last_category";
|
||||
const PATH: &str = "/dashboard/feeds";
|
||||
|
||||
/// Routes contributed by this page group (merged by `dashboard::router`).
|
||||
pub fn routes() -> Router<AppState> {
|
||||
Router::new()
|
||||
.route(PATH, get(index))
|
||||
.route("/dashboard/feeds/{id}/add", post(add))
|
||||
.route("/dashboard/feeds/{id}/dismiss", post(dismiss))
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, Deserialize)]
|
||||
struct FeedsQuery {
|
||||
status: Option<String>,
|
||||
page: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct AddForm {
|
||||
category_id: i64,
|
||||
}
|
||||
|
||||
/// One linked article, as named in the "Why" column.
|
||||
#[derive(Debug)]
|
||||
struct WhyArticle {
|
||||
id: ArticleId,
|
||||
title: String,
|
||||
}
|
||||
|
||||
/// One table row.
|
||||
#[derive(Debug)]
|
||||
struct FeedRow {
|
||||
id: i64,
|
||||
/// The shrunk mean on 0–100, rounded.
|
||||
score: i64,
|
||||
feed_url: String,
|
||||
/// The candidate's title, or its feed URL when it has none.
|
||||
label: String,
|
||||
host: String,
|
||||
interests: Vec<String>,
|
||||
articles: Vec<WhyArticle>,
|
||||
article_count: usize,
|
||||
first_seen: String,
|
||||
last_seen: String,
|
||||
/// `last_seen` as a short date for the column; the full first/last
|
||||
/// timestamps sit in the cell's tooltip.
|
||||
seen: String,
|
||||
decided_at: String,
|
||||
/// `{miniflux.base_url}/feeds/{id}` for a candidate we subscribed to.
|
||||
miniflux_href: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Template)]
|
||||
#[template(path = "dashboard/feeds.html")]
|
||||
struct FeedsTemplate {
|
||||
page: Page,
|
||||
status: String,
|
||||
rows: Vec<FeedRow>,
|
||||
candidate_count: i64,
|
||||
added_count: i64,
|
||||
dismissed_count: i64,
|
||||
categories: Vec<MinifluxCategory>,
|
||||
selected_category: i64,
|
||||
/// Why the category picker is empty, when it is; Add is disabled then.
|
||||
categories_error: Option<String>,
|
||||
pager: Pager,
|
||||
}
|
||||
|
||||
async fn index(
|
||||
State(state): State<AppState>,
|
||||
auth: AuthSession,
|
||||
Extension(session): Extension<Session>,
|
||||
Query(query): Query<FeedsQuery>,
|
||||
) -> Result<Response, WebError> {
|
||||
let viewer = auth.user().await.map(Viewer::from);
|
||||
let config = state.config();
|
||||
let db = &state.db;
|
||||
let status = allow_listed(query.status.as_deref(), &STATUSES).unwrap_or("candidate");
|
||||
let page_no = page_number(query.page);
|
||||
|
||||
// Candidates are ranked globally, so they are all loaded, scored, sorted
|
||||
// and only then paginated (plan §4 step 4).
|
||||
let (candidates, total, ranked_evidence) = if status == "candidate" {
|
||||
let all = discovery::all_candidates(db).await?;
|
||||
let evidence = load_evidence(db, &config, &all).await?;
|
||||
let mut ranked: Vec<(Candidate, f64)> = all
|
||||
.into_iter()
|
||||
.map(|candidate| {
|
||||
let score = discovery::score(&evidence_of(&candidate, &evidence));
|
||||
(candidate, score)
|
||||
})
|
||||
.collect();
|
||||
ranked.sort_by(|a, b| {
|
||||
b.1.partial_cmp(&a.1)
|
||||
.unwrap_or(Ordering::Equal)
|
||||
.then_with(|| b.0.article_ids.len().cmp(&a.0.article_ids.len()))
|
||||
.then_with(|| b.0.last_seen.cmp(&a.0.last_seen))
|
||||
});
|
||||
let total = ranked.len() as i64;
|
||||
let offset = (page_no.saturating_sub(1) as usize) * CANDIDATES_PER_PAGE as usize;
|
||||
let page_rows: Vec<Candidate> = ranked
|
||||
.into_iter()
|
||||
.skip(offset)
|
||||
.take(CANDIDATES_PER_PAGE as usize)
|
||||
.map(|(candidate, _)| candidate)
|
||||
.collect();
|
||||
(page_rows, total, Some(evidence))
|
||||
} else {
|
||||
let (candidates, total) = discovery::list(db, status, page_no, CANDIDATES_PER_PAGE).await?;
|
||||
(candidates, total, None)
|
||||
};
|
||||
|
||||
let evidence = match ranked_evidence {
|
||||
Some(evidence) => evidence,
|
||||
None => load_evidence(db, &config, &candidates).await?,
|
||||
};
|
||||
let titles = article_titles(db, &candidates).await?;
|
||||
let base_url = config.miniflux.base_url.trim_end_matches('/').to_string();
|
||||
let rows: Vec<FeedRow> = candidates
|
||||
.iter()
|
||||
.map(|candidate| row(candidate, &evidence, &titles, &config, &base_url))
|
||||
.collect();
|
||||
|
||||
// The picker is only ever used by the candidate view, so only it pays for
|
||||
// the categories call (plan §3).
|
||||
let (categories, categories_error) = if status == "candidate" {
|
||||
categories(&config).await
|
||||
} else {
|
||||
(Vec::new(), None)
|
||||
};
|
||||
let selected_category = selected_category(db, &categories).await?;
|
||||
|
||||
let pager = Pager::new(
|
||||
Pagination {
|
||||
page: page_no,
|
||||
per_page: CANDIDATES_PER_PAGE,
|
||||
total,
|
||||
},
|
||||
PATH,
|
||||
&[("status", Some(status.to_string()))],
|
||||
);
|
||||
let mut page = Page::new("Feeds", viewer, "feeds");
|
||||
page.flash = take_flash(&session).await?;
|
||||
Ok(Html(FeedsTemplate {
|
||||
page,
|
||||
status: status.to_string(),
|
||||
rows,
|
||||
candidate_count: discovery::count(db, "candidate").await?,
|
||||
added_count: discovery::count(db, "added").await?,
|
||||
dismissed_count: discovery::count(db, "dismissed").await?,
|
||||
categories,
|
||||
selected_category,
|
||||
categories_error,
|
||||
pager,
|
||||
})
|
||||
.into_response())
|
||||
}
|
||||
|
||||
async fn add(
|
||||
State(state): State<AppState>,
|
||||
Extension(session): Extension<Session>,
|
||||
Path(id): Path<i64>,
|
||||
Form(form): Form<AddForm>,
|
||||
) -> Result<Response, WebError> {
|
||||
let config = state.config();
|
||||
let db = &state.db;
|
||||
let now = Timestamp::now();
|
||||
let Some(candidate) = discovery::candidate(db, id).await? else {
|
||||
set_flash(&session, "error", format!("No feed candidate {id}.")).await?;
|
||||
return Ok(Redirect::to(PATH).into_response());
|
||||
};
|
||||
let label = candidate
|
||||
.title
|
||||
.clone()
|
||||
.unwrap_or_else(|| candidate.feed_url.clone());
|
||||
let Some(client) = client(&config) else {
|
||||
set_flash(
|
||||
&session,
|
||||
"error",
|
||||
"Miniflux API key is not configured".into(),
|
||||
)
|
||||
.await?;
|
||||
return Ok(Redirect::to(PATH).into_response());
|
||||
};
|
||||
|
||||
match client
|
||||
.create_feed(&candidate.feed_url, form.category_id)
|
||||
.await
|
||||
{
|
||||
Ok(feed_id) => {
|
||||
discovery::set_status(db, id, "added", Some(feed_id), now).await?;
|
||||
db.kv_set(LAST_CATEGORY_KEY, &form.category_id.to_string())
|
||||
.await?;
|
||||
set_flash(&session, "success", format!("Added {label}.")).await?;
|
||||
}
|
||||
// Already subscribed by other means: the operator's intent is met, so
|
||||
// the row is decided too (plan §2).
|
||||
Err(error) if error.is_duplicate_feed() => {
|
||||
discovery::set_status(db, id, "added", None, now).await?;
|
||||
set_flash(&session, "success", format!("{label}: {error}")).await?;
|
||||
}
|
||||
Err(error) => {
|
||||
set_flash(&session, "error", format!("{label}: {error}")).await?;
|
||||
}
|
||||
}
|
||||
Ok(Redirect::to(PATH).into_response())
|
||||
}
|
||||
|
||||
async fn dismiss(
|
||||
State(state): State<AppState>,
|
||||
Extension(session): Extension<Session>,
|
||||
Path(id): Path<i64>,
|
||||
) -> Result<Response, WebError> {
|
||||
let db = &state.db;
|
||||
let Some(candidate) = discovery::candidate(db, id).await? else {
|
||||
set_flash(&session, "error", format!("No feed candidate {id}.")).await?;
|
||||
return Ok(Redirect::to(PATH).into_response());
|
||||
};
|
||||
discovery::set_status(db, id, "dismissed", None, Timestamp::now()).await?;
|
||||
let label = candidate.title.unwrap_or(candidate.feed_url);
|
||||
set_flash(&session, "success", format!("Dismissed {label}.")).await?;
|
||||
Ok(Redirect::to(PATH).into_response())
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// A Miniflux client from `[miniflux]`, or `None` when there is no API key.
|
||||
fn client(config: &Config) -> Option<MinifluxClient> {
|
||||
let http = crate::http::build_client(crate::http::DEFAULT_TIMEOUT).ok()?;
|
||||
MinifluxClient::new(&config.miniflux, http).ok()
|
||||
}
|
||||
|
||||
/// `GET /v1/categories` for the Add picker; the error is shown as a notice on
|
||||
/// the page (not a flash: nothing was submitted) and disables Add.
|
||||
async fn categories(config: &Config) -> (Vec<MinifluxCategory>, Option<String>) {
|
||||
let Some(client) = client(config) else {
|
||||
return (
|
||||
Vec::new(),
|
||||
Some("Miniflux API key is not configured, so feeds cannot be added.".into()),
|
||||
);
|
||||
};
|
||||
match client.categories().await {
|
||||
Ok(categories) => (categories, None),
|
||||
Err(error) => (
|
||||
Vec::new(),
|
||||
Some(format!("Could not load Miniflux categories: {error}")),
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The category the last Add used, when it still exists; else the first one.
|
||||
async fn selected_category(db: &Db, categories: &[MinifluxCategory]) -> Result<i64, WebError> {
|
||||
let last = db
|
||||
.kv_get(LAST_CATEGORY_KEY)
|
||||
.await?
|
||||
.and_then(|value| value.trim().parse::<i64>().ok())
|
||||
.filter(|id| categories.iter().any(|category| category.id == *id));
|
||||
Ok(last
|
||||
.or_else(|| categories.first().map(|category| category.id))
|
||||
.unwrap_or_default())
|
||||
}
|
||||
|
||||
fn union_ids(candidates: &[Candidate]) -> Vec<ArticleId> {
|
||||
let mut seen: HashSet<ArticleId> = HashSet::new();
|
||||
candidates
|
||||
.iter()
|
||||
.flat_map(|candidate| candidate.article_ids.iter().copied())
|
||||
.filter(|id| seen.insert(*id))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Ranking evidence for every article linked to `candidates`, in one query.
|
||||
async fn load_evidence(
|
||||
db: &Db,
|
||||
config: &Config,
|
||||
candidates: &[Candidate],
|
||||
) -> Result<HashMap<ArticleId, ArticleEvidence>, WebError> {
|
||||
Ok(discovery::load_evidence(
|
||||
db,
|
||||
&union_ids(candidates),
|
||||
config.curation.ranking.rating_lookback_days,
|
||||
)
|
||||
.await?)
|
||||
}
|
||||
|
||||
/// Titles for every linked article, including the ones with no evidence (which
|
||||
/// the "Why" column still lists, last).
|
||||
async fn article_titles(
|
||||
db: &Db,
|
||||
candidates: &[Candidate],
|
||||
) -> Result<HashMap<ArticleId, String>, WebError> {
|
||||
let ids = union_ids(candidates);
|
||||
let mut titles = HashMap::new();
|
||||
for chunk in ids.chunks(500) {
|
||||
let placeholders = vec!["?"; chunk.len()].join(", ");
|
||||
let mut query = dynamic_query(format!(
|
||||
"SELECT id, COALESCE(title, '') AS title FROM articles WHERE id IN ({placeholders})"
|
||||
));
|
||||
for id in chunk {
|
||||
query = query.bind(*id);
|
||||
}
|
||||
for row in query.fetch_all(db.pool()).await.map_err(db_err)? {
|
||||
titles.insert(row.get("id"), row.get("title"));
|
||||
}
|
||||
}
|
||||
Ok(titles)
|
||||
}
|
||||
|
||||
/// `Sep 7` for the current year, `Sep 7, 2025` otherwise, in the configured
|
||||
/// time zone; the raw string when it does not parse.
|
||||
fn compact_date(raw: &str, config: &Config) -> String {
|
||||
let (Ok(timestamp), Ok(tz)) = (raw.parse::<Timestamp>(), config.tz()) else {
|
||||
return raw.to_string();
|
||||
};
|
||||
let zoned = timestamp.to_zoned(tz.clone());
|
||||
let format = if zoned.year() == Timestamp::now().to_zoned(tz).year() {
|
||||
"%b %-d"
|
||||
} else {
|
||||
"%b %-d, %Y"
|
||||
};
|
||||
zoned.strftime(format).to_string()
|
||||
}
|
||||
|
||||
fn evidence_of(
|
||||
candidate: &Candidate,
|
||||
evidence: &HashMap<ArticleId, ArticleEvidence>,
|
||||
) -> Vec<ArticleEvidence> {
|
||||
candidate
|
||||
.article_ids
|
||||
.iter()
|
||||
.filter_map(|id| evidence.get(id).cloned())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn row(
|
||||
candidate: &Candidate,
|
||||
evidence: &HashMap<ArticleId, ArticleEvidence>,
|
||||
titles: &HashMap<ArticleId, String>,
|
||||
config: &Config,
|
||||
base_url: &str,
|
||||
) -> FeedRow {
|
||||
let mut scored = evidence_of(candidate, evidence);
|
||||
let score = discovery::score(&scored);
|
||||
let interests = discovery::why(&scored);
|
||||
|
||||
// Best evidence first; the articles nothing is known about come last, in
|
||||
// title order.
|
||||
scored.sort_by(|a, b| {
|
||||
b.value
|
||||
.partial_cmp(&a.value)
|
||||
.unwrap_or(Ordering::Equal)
|
||||
.then_with(|| a.title.cmp(&b.title))
|
||||
});
|
||||
let mut unscored: Vec<WhyArticle> = candidate
|
||||
.article_ids
|
||||
.iter()
|
||||
.filter(|id| !evidence.contains_key(id))
|
||||
.map(|id| WhyArticle {
|
||||
id: *id,
|
||||
title: titles.get(id).cloned().unwrap_or_default(),
|
||||
})
|
||||
.collect();
|
||||
unscored.sort_by(|a, b| a.title.cmp(&b.title));
|
||||
let articles: Vec<WhyArticle> = scored
|
||||
.into_iter()
|
||||
.map(|item| WhyArticle {
|
||||
id: item.article_id,
|
||||
title: item.title,
|
||||
})
|
||||
.chain(unscored)
|
||||
.take(WHY_ARTICLES)
|
||||
.collect();
|
||||
|
||||
FeedRow {
|
||||
id: candidate.id,
|
||||
score: score.round() as i64,
|
||||
feed_url: candidate.feed_url.clone(),
|
||||
label: candidate
|
||||
.title
|
||||
.clone()
|
||||
.unwrap_or_else(|| candidate.feed_url.clone()),
|
||||
host: candidate.host.clone(),
|
||||
interests,
|
||||
articles,
|
||||
article_count: candidate.article_ids.len(),
|
||||
first_seen: fmt_stored_time(Some(&candidate.first_seen), config),
|
||||
last_seen: fmt_stored_time(Some(&candidate.last_seen), config),
|
||||
seen: compact_date(&candidate.last_seen, config),
|
||||
decided_at: fmt_stored_time(candidate.decided_at.as_deref(), config),
|
||||
miniflux_href: candidate
|
||||
.miniflux_feed_id
|
||||
.map(|feed_id| format!("{base_url}/feeds/{feed_id}")),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use axum::Json;
|
||||
use axum::body::Body;
|
||||
use axum::http::{Method, Request, StatusCode, header};
|
||||
use serde_json::json;
|
||||
use tower::ServiceExt;
|
||||
|
||||
use super::*;
|
||||
use crate::server::router;
|
||||
use crate::web::dashboard::tests::{
|
||||
Seed, app_with_users, assert_admin_only, get, login_cookie, response_text, seed,
|
||||
};
|
||||
|
||||
/// A candidate on `host` linked to `article_ids`.
|
||||
async fn candidate(db: &Db, title: &str, host: &str, article_ids: &[ArticleId]) -> i64 {
|
||||
let id = discovery::upsert_candidate(
|
||||
db,
|
||||
&format!("https://{host}/feed.xml"),
|
||||
host,
|
||||
Some(title),
|
||||
"2026-09-06T04:00:00Z".parse().unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
for article_id in article_ids {
|
||||
discovery::link_article(db, id, *article_id).await.unwrap();
|
||||
}
|
||||
id
|
||||
}
|
||||
|
||||
async fn status_of(db: &Db, id: i64) -> (String, Option<i64>, Option<String>) {
|
||||
let candidate = discovery::candidate(db, id).await.unwrap().unwrap();
|
||||
(
|
||||
candidate.status,
|
||||
candidate.miniflux_feed_id,
|
||||
candidate.decided_at,
|
||||
)
|
||||
}
|
||||
|
||||
/// A Miniflux stand-in answering `POST /v1/feeds` with `body`.
|
||||
async fn miniflux(status: StatusCode, body: serde_json::Value) -> String {
|
||||
let handler = move || {
|
||||
let body = body.clone();
|
||||
async move { (status, Json(body)) }
|
||||
};
|
||||
let app = Router::new().route("/v1/feeds", axum::routing::post(handler));
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
let _ = axum::serve(listener, app).await;
|
||||
});
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
async fn app_with_miniflux(seed: &Seed, base_url: String) -> axum::Router {
|
||||
crate::web::users::add(&seed.db, "reader", "correct horse battery", false)
|
||||
.await
|
||||
.unwrap();
|
||||
crate::web::users::add(&seed.db, "admin", "correct horse battery", true)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut config = Config::default();
|
||||
config.miniflux.base_url = base_url;
|
||||
config.miniflux.api_key = Some("test-key-never-logged".into());
|
||||
router(AppState::new(seed.db.clone(), config, None))
|
||||
}
|
||||
|
||||
async fn post_form(app: &axum::Router, uri: &str, cookie: &str) -> Response {
|
||||
app.clone()
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.method(Method::POST)
|
||||
.uri(uri)
|
||||
.header(header::COOKIE, cookie)
|
||||
.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded")
|
||||
.header("sec-fetch-site", "same-origin")
|
||||
.body(Body::from("category_id=3"))
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn feeds_page_is_admin_only_and_empty_without_candidates() {
|
||||
let seed = seed().await;
|
||||
let app = app_with_users(&seed.db).await;
|
||||
|
||||
let body = assert_admin_only(&app, "/dashboard/feeds").await;
|
||||
|
||||
assert!(body.contains("<h1>Feeds</h1>"), "{body}");
|
||||
assert!(body.contains("No feed candidates"), "{body}");
|
||||
assert!(body.contains("Candidates (0)"), "{body}");
|
||||
// No API key in the test config, so Add is disabled with a notice.
|
||||
assert!(
|
||||
body.contains("Miniflux API key is not configured, so feeds cannot be added."),
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn candidates_are_listed_best_first() {
|
||||
let seed = seed().await;
|
||||
// Article 1 is rated `loved` (value 1.0); article 5 only ever reached
|
||||
// `triaged`, so it scores on its blend alone.
|
||||
candidate(&seed.db, "Weak Blog", "weak.example", &[5]).await;
|
||||
candidate(&seed.db, "Strong Blog", "strong.example", &[1]).await;
|
||||
let app = app_with_users(&seed.db).await;
|
||||
|
||||
let body = assert_admin_only(&app, "/dashboard/feeds").await;
|
||||
|
||||
assert!(body.contains("Candidates (2)"), "{body}");
|
||||
assert!(body.contains("Strong Blog"), "{body}");
|
||||
assert!(body.contains("Weak Blog"), "{body}");
|
||||
assert!(body.contains("https://strong.example/feed.xml"), "{body}");
|
||||
// The rated article is named in the Why column and links to it.
|
||||
assert!(body.contains("/dashboard/articles/1"), "{body}");
|
||||
assert!(body.contains("Gaussian Splatting"), "{body}");
|
||||
let strong = body.find("Strong Blog").unwrap();
|
||||
let weak = body.find("Weak Blog").unwrap();
|
||||
assert!(strong < weak, "{body}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dismiss_decides_the_row_and_redirects() {
|
||||
let seed = seed().await;
|
||||
let id = candidate(&seed.db, "Noise", "noise.example", &[2]).await;
|
||||
let app = app_with_users(&seed.db).await;
|
||||
let admin = login_cookie(&app, "admin", "correct horse battery").await;
|
||||
|
||||
let response = post_form(&app, &format!("/dashboard/feeds/{id}/dismiss"), &admin).await;
|
||||
assert_eq!(response.status(), StatusCode::SEE_OTHER);
|
||||
assert_eq!(
|
||||
response.headers().get(header::LOCATION).unwrap(),
|
||||
"/dashboard/feeds"
|
||||
);
|
||||
|
||||
let (status, feed_id, decided_at) = status_of(&seed.db, id).await;
|
||||
assert_eq!(status, "dismissed");
|
||||
assert_eq!(feed_id, None);
|
||||
assert!(decided_at.is_some());
|
||||
|
||||
let body = response_text(get(&app, "/dashboard/feeds", Some(&admin)).await).await;
|
||||
assert!(body.contains("Dismissed Noise."), "{body}");
|
||||
assert!(body.contains("Candidates (0)"), "{body}");
|
||||
assert!(body.contains("Dismissed (1)"), "{body}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn add_subscribes_and_records_the_miniflux_feed_id() {
|
||||
let seed = seed().await;
|
||||
let id = candidate(&seed.db, "Strong Blog", "strong.example", &[1]).await;
|
||||
let base_url = miniflux(StatusCode::CREATED, json!({"feed_id": 77})).await;
|
||||
let app = app_with_miniflux(&seed, base_url).await;
|
||||
let admin = login_cookie(&app, "admin", "correct horse battery").await;
|
||||
|
||||
let response = post_form(&app, &format!("/dashboard/feeds/{id}/add"), &admin).await;
|
||||
assert_eq!(response.status(), StatusCode::SEE_OTHER);
|
||||
|
||||
let (status, feed_id, decided_at) = status_of(&seed.db, id).await;
|
||||
assert_eq!(status, "added");
|
||||
assert_eq!(feed_id, Some(77));
|
||||
assert!(decided_at.is_some());
|
||||
assert_eq!(
|
||||
seed.db.kv_get(LAST_CATEGORY_KEY).await.unwrap().as_deref(),
|
||||
Some("3")
|
||||
);
|
||||
|
||||
let body = response_text(get(&app, "/dashboard/feeds", Some(&admin)).await).await;
|
||||
assert!(body.contains("Added Strong Blog."), "{body}");
|
||||
let added =
|
||||
response_text(get(&app, "/dashboard/feeds?status=added", Some(&admin)).await).await;
|
||||
assert!(added.contains("/feeds/77"), "{added}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_duplicate_subscription_still_decides_the_row() {
|
||||
let seed = seed().await;
|
||||
let id = candidate(&seed.db, "Known Blog", "known.example", &[1]).await;
|
||||
let base_url = miniflux(
|
||||
StatusCode::BAD_REQUEST,
|
||||
json!({"error_message": "This feed already exists."}),
|
||||
)
|
||||
.await;
|
||||
let app = app_with_miniflux(&seed, base_url).await;
|
||||
let admin = login_cookie(&app, "admin", "correct horse battery").await;
|
||||
|
||||
let response = post_form(&app, &format!("/dashboard/feeds/{id}/add"), &admin).await;
|
||||
assert_eq!(response.status(), StatusCode::SEE_OTHER);
|
||||
|
||||
let (status, feed_id, decided_at) = status_of(&seed.db, id).await;
|
||||
assert_eq!(status, "added");
|
||||
assert_eq!(feed_id, None);
|
||||
assert!(decided_at.is_some());
|
||||
|
||||
let body = response_text(get(&app, "/dashboard/feeds", Some(&admin)).await).await;
|
||||
assert!(body.contains("This feed already exists."), "{body}");
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@
|
||||
//! `_signals_table.html`, the allow-list check and the LIKE-pattern escaper.
|
||||
|
||||
pub mod articles;
|
||||
pub mod feeds;
|
||||
pub mod jobs;
|
||||
pub mod profile;
|
||||
pub mod ratings;
|
||||
@@ -44,6 +45,7 @@ pub fn router() -> Router<AppState> {
|
||||
.merge(runs::routes())
|
||||
.merge(articles::routes())
|
||||
.merge(ratings::routes())
|
||||
.merge(feeds::routes())
|
||||
.merge(profile::routes())
|
||||
.merge(settings::routes())
|
||||
.merge(jobs::routes())
|
||||
@@ -422,6 +424,7 @@ struct OverviewTemplate {
|
||||
ratings: Vec<LabelCount>,
|
||||
ratings_total: i64,
|
||||
access_requests: i64,
|
||||
feed_candidates: i64,
|
||||
unrated: Vec<UnratedPick>,
|
||||
active_jobs: Vec<JobLine>,
|
||||
finished_jobs: Vec<JobLine>,
|
||||
@@ -448,6 +451,7 @@ async fn overview(
|
||||
.fetch_one(db.pool())
|
||||
.await
|
||||
.map_err(db_err)?;
|
||||
let feed_candidates = crate::discovery::count(db, "candidate").await?;
|
||||
let unrated = unrated_picks(db).await?;
|
||||
let (active_jobs, finished_jobs) = jobs_summary(db, &config).await?;
|
||||
let sparklines = overview_sparklines(db).await?;
|
||||
@@ -466,6 +470,7 @@ async fn overview(
|
||||
ratings,
|
||||
ratings_total,
|
||||
access_requests,
|
||||
feed_candidates,
|
||||
unrated,
|
||||
active_jobs,
|
||||
finished_jobs,
|
||||
|
||||
@@ -277,6 +277,7 @@ const GROUP_ORDER: &[&str] = &[
|
||||
"miniflux",
|
||||
"bookorbit",
|
||||
"mail",
|
||||
"discovery",
|
||||
];
|
||||
|
||||
/// Help text per key, seeded from the README configuration table and the
|
||||
@@ -401,6 +402,9 @@ pub const SETTINGS_HELP: &[(&str, &str)] = &[
|
||||
("mail.smtp_pass", "SMTP password. Environment only; requires a server restart."),
|
||||
("mail.from", "Sender mailbox as an address or Name <address>. Requires a server restart."),
|
||||
("mail.notify_to", "Recipient for new access-request notifications. Requires a server restart."),
|
||||
("discovery.enabled", "Run the feed discovery stage during generate."),
|
||||
("discovery.max_lookups_per_run", "How many not-yet-checked hosts one run may look up in Miniflux; each host is re-checked at most every 90 days."),
|
||||
("discovery.skip_hosts", "Hosts never looked up, one per line. Matches the host or any subdomain of it."),
|
||||
];
|
||||
|
||||
/// `DAILY_EPUB_` + the path upper-cased with `.` → `__` (§13.1 item 2).
|
||||
@@ -1725,6 +1729,7 @@ mod tests {
|
||||
"miniflux",
|
||||
"bookorbit",
|
||||
"mail",
|
||||
"discovery",
|
||||
]
|
||||
);
|
||||
let anthropic = groups
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
/*! tailwindcss v4.3.3 | MIT License | https://tailwindcss.com */
|
||||
@layer properties{@supports (((-webkit-hyphens:none)) and (not (margin-trim:inline))) or ((-moz-orient:inline) and (not (color:rgb(from red r g b)))){*,:before,:after,::backdrop{--tw-translate-x:0;--tw-translate-y:0;--tw-translate-z:0;--tw-rotate-x:initial;--tw-rotate-y:initial;--tw-rotate-z:initial;--tw-skew-x:initial;--tw-skew-y:initial;--tw-space-y-reverse:0;--tw-divide-y-reverse:0;--tw-border-style:solid;--tw-leading:initial;--tw-font-weight:initial;--tw-tracking:initial;--tw-ordinal:initial;--tw-slashed-zero:initial;--tw-numeric-figure:initial;--tw-numeric-spacing:initial;--tw-numeric-fraction:initial;--tw-shadow:0 0 #0000;--tw-shadow-color:initial;--tw-shadow-alpha:100%;--tw-inset-shadow:0 0 #0000;--tw-inset-shadow-color:initial;--tw-inset-shadow-alpha:100%;--tw-ring-color:initial;--tw-ring-shadow:0 0 #0000;--tw-inset-ring-color:initial;--tw-inset-ring-shadow:0 0 #0000;--tw-ring-inset:initial;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-offset-shadow:0 0 #0000;--tw-outline-style:solid;--tw-blur:initial;--tw-brightness:initial;--tw-contrast:initial;--tw-grayscale:initial;--tw-hue-rotate:initial;--tw-invert:initial;--tw-opacity:initial;--tw-saturate:initial;--tw-sepia:initial;--tw-drop-shadow:initial;--tw-drop-shadow-color:initial;--tw-drop-shadow-alpha:100%;--tw-drop-shadow-size:initial;--tw-duration:initial}}}@layer theme{:root,:host{--font-sans:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--font-serif:"Newsreader", "Newsreader Fallback Georgia", "Newsreader Fallback Times", "Newsreader Fallback Noto Serif", "Newsreader Fallback DejaVu Serif", "Iowan Old Style", "Palatino Linotype", Georgia, serif;--font-mono:ui-monospace, SFMono-Regular, Menlo, monospace;--spacing:.25rem;--container-md:28rem;--container-lg:32rem;--container-2xl:42rem;--container-7xl:80rem;--text-xs:.75rem;--text-xs--line-height:calc(1 / .75);--text-sm:.875rem;--text-sm--line-height:calc(1.25 / .875);--text-base:1rem;--text-base--line-height:calc(1.5 / 1);--text-lg:1.125rem;--text-lg--line-height:calc(1.75 / 1.125);--text-xl:1.25rem;--text-xl--line-height:calc(1.75 / 1.25);--text-2xl:1.5rem;--text-2xl--line-height:calc(2 / 1.5);--text-3xl:1.875rem;--text-3xl--line-height:calc(2.25 / 1.875);--text-4xl:2.25rem;--text-4xl--line-height:calc(2.5 / 2.25);--text-5xl:3rem;--text-5xl--line-height:1;--font-weight-normal:400;--font-weight-medium:500;--font-weight-semibold:600;--tracking-normal:0em;--leading-tight:1.25;--leading-snug:1.375;--leading-normal:1.5;--leading-relaxed:1.625;--radius-sm:.25rem;--default-transition-duration:.15s;--default-transition-timing-function:cubic-bezier(.4, 0, .2, 1);--default-font-family:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--default-mono-font-family:ui-monospace, SFMono-Regular, Menlo, monospace;--color-paper:var(--paper);--color-paper-2:var(--paper-2);--color-ink:var(--ink)}}@layer base{*,:after,:before,::backdrop{box-sizing:border-box;border:0 solid;margin:0;padding:0}::file-selector-button{box-sizing:border-box;border:0 solid;margin:0;padding:0}html,:host{-webkit-text-size-adjust:100%;tab-size:4;line-height:1.5;font-family:var(--default-font-family,-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Noto Sans", Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji");font-feature-settings:var(--default-font-feature-settings,normal);font-variation-settings:var(--default-font-variation-settings,normal);-webkit-tap-highlight-color:transparent}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,samp,pre{font-family:var(--default-mono-font-family,ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace);font-feature-settings:var(--default-mono-font-feature-settings,normal);font-variation-settings:var(--default-mono-font-variation-settings,normal);font-size:1em}small{font-size:80%}sub,sup{vertical-align:baseline;font-size:75%;line-height:0;position:relative}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}:-moz-focusring:where(:not(iframe)){outline:auto}progress{vertical-align:baseline}summary{display:list-item}ol,ul,menu{list-style:none}img,svg,video,canvas,audio,iframe,embed,object{vertical-align:middle;display:block}img,video{max-width:100%;height:auto}button,input,select,optgroup,textarea{font:inherit;font-feature-settings:inherit;font-variation-settings:inherit;letter-spacing:inherit;color:inherit;opacity:1;background-color:#0000;border-radius:0}::file-selector-button{font:inherit;font-feature-settings:inherit;font-vaLine truncated
|
||||
@layer properties{@supports (((-webkit-hyphens:none)) and (not (margin-trim:inline))) or ((-moz-orient:inline) and (not (color:rgb(from red r g b)))){*,:before,:after,::backdrop{--tw-translate-x:0;--tw-translate-y:0;--tw-translate-z:0;--tw-rotate-x:initial;--tw-rotate-y:initial;--tw-rotate-z:initial;--tw-skew-x:initial;--tw-skew-y:initial;--tw-space-y-reverse:0;--tw-divide-y-reverse:0;--tw-border-style:solid;--tw-leading:initial;--tw-font-weight:initial;--tw-tracking:initial;--tw-ordinal:initial;--tw-slashed-zero:initial;--tw-numeric-figure:initial;--tw-numeric-spacing:initial;--tw-numeric-fraction:initial;--tw-shadow:0 0 #0000;--tw-shadow-color:initial;--tw-shadow-alpha:100%;--tw-inset-shadow:0 0 #0000;--tw-inset-shadow-color:initial;--tw-inset-shadow-alpha:100%;--tw-ring-color:initial;--tw-ring-shadow:0 0 #0000;--tw-inset-ring-color:initial;--tw-inset-ring-shadow:0 0 #0000;--tw-ring-inset:initial;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-offset-shadow:0 0 #0000;--tw-outline-style:solid;--tw-blur:initial;--tw-brightness:initial;--tw-contrast:initial;--tw-grayscale:initial;--tw-hue-rotate:initial;--tw-invert:initial;--tw-opacity:initial;--tw-saturate:initial;--tw-sepia:initial;--tw-drop-shadow:initial;--tw-drop-shadow-color:initial;--tw-drop-shadow-alpha:100%;--tw-drop-shadow-size:initial;--tw-duration:initial}}}@layer theme{:root,:host{--font-sans:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--font-serif:"Newsreader", "Newsreader Fallback Georgia", "Newsreader Fallback Times", "Newsreader Fallback Noto Serif", "Newsreader Fallback DejaVu Serif", "Iowan Old Style", "Palatino Linotype", Georgia, serif;--font-mono:ui-monospace, SFMono-Regular, Menlo, monospace;--spacing:.25rem;--container-md:28rem;--container-lg:32rem;--container-2xl:42rem;--container-7xl:80rem;--text-xs:.75rem;--text-xs--line-height:calc(1 / .75);--text-sm:.875rem;--text-sm--line-height:calc(1.25 / .875);--text-base:1rem;--text-base--line-height:calc(1.5 / 1);--text-lg:1.125rem;--text-lg--line-height:calc(1.75 / 1.125);--text-xl:1.25rem;--text-xl--line-height:calc(1.75 / 1.25);--text-2xl:1.5rem;--text-2xl--line-height:calc(2 / 1.5);--text-3xl:1.875rem;--text-3xl--line-height:calc(2.25 / 1.875);--text-4xl:2.25rem;--text-4xl--line-height:calc(2.5 / 2.25);--text-5xl:3rem;--text-5xl--line-height:1;--font-weight-normal:400;--font-weight-medium:500;--font-weight-semibold:600;--tracking-normal:0em;--leading-tight:1.25;--leading-snug:1.375;--leading-normal:1.5;--leading-relaxed:1.625;--radius-sm:.25rem;--default-transition-duration:.15s;--default-transition-timing-function:cubic-bezier(.4, 0, .2, 1);--default-font-family:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--default-mono-font-family:ui-monospace, SFMono-Regular, Menlo, monospace;--color-paper:var(--paper);--color-paper-2:var(--paper-2);--color-ink:var(--ink)}}@layer base{*,:after,:before,::backdrop{box-sizing:border-box;border:0 solid;margin:0;padding:0}::file-selector-button{box-sizing:border-box;border:0 solid;margin:0;padding:0}html,:host{-webkit-text-size-adjust:100%;tab-size:4;line-height:1.5;font-family:var(--default-font-family,-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Noto Sans", Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji");font-feature-settings:var(--default-font-feature-settings,normal);font-variation-settings:var(--default-font-variation-settings,normal);-webkit-tap-highlight-color:transparent}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,samp,pre{font-family:var(--default-mono-font-family,ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace);font-feature-settings:var(--default-mono-font-feature-settings,normal);font-variation-settings:var(--default-mono-font-variation-settings,normal);font-size:1em}small{font-size:80%}sub,sup{vertical-align:baseline;font-size:75%;line-height:0;position:relative}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}:-moz-focusring:where(:not(iframe)){outline:auto}progress{vertical-align:baseline}summary{display:list-item}ol,ul,menu{list-style:none}img,svg,video,canvas,audio,iframe,embed,object{vertical-align:middle;display:block}img,video{max-width:100%;height:auto}button,input,select,optgroup,textarea{font:inherit;font-feature-settings:inherit;font-variation-settings:inherit;letter-spacing:inherit;color:inherit;opacity:1;background-color:#0000;border-radius:0}::file-selector-button{font:inherit;font-feature-settings:inherit;font-vaLine truncated
|
||||
@@ -0,0 +1,21 @@
|
||||
{% extends "layout.html" %}{% block content %}<section class="dashboard">
|
||||
<header class="page-head"><div>
|
||||
<h1>Feeds</h1>
|
||||
<p class="page-desc">Feeds behind the stories that reached us through an aggregator, ranked by how the articles they carried actually fared. Add subscribes in Miniflux; Dismiss hides the feed for good.</p>
|
||||
</div><div class="page-actions"><a class="btn" href="/dashboard/articles">Articles</a><a class="btn" href="/dashboard/runs">Runs</a></div></header>
|
||||
<nav class="tabs" aria-label="Candidate status"><a href="/dashboard/feeds"{% if status == "candidate" %} class="active" aria-current="page"{% endif %}>Candidates ({{ candidate_count }})</a> <a href="/dashboard/feeds?status=added"{% if status == "added" %} class="active" aria-current="page"{% endif %}>Added ({{ added_count }})</a> <a href="/dashboard/feeds?status=dismissed"{% if status == "dismissed" %} class="active" aria-current="page"{% endif %}>Dismissed ({{ dismissed_count }})</a></nav>
|
||||
{% if let Some(error) = categories_error %}<p class="notice">{{ error }}</p>{% endif %}
|
||||
{% include "dashboard/_pager.html" %}
|
||||
<table class="table-fixed">
|
||||
<thead><tr><th class="num w-14">Score</th><th>Feed</th><th>Why</th><th class="num w-20">Articles</th><th class="w-20">Seen</th><th class="w-60 lg:w-84"><span class="sr-only">Actions</span></th></tr></thead>
|
||||
<tbody>{% for row in rows %}<tr>
|
||||
<td class="num tabular-nums">{{ row.score }}</td>
|
||||
<td class="cell-wrap"><a class="line-clamp-2" href="{{ row.feed_url }}" rel="noopener" target="_blank" title="{{ row.feed_url }}">{{ row.label }}</a><span class="muted block truncate text-xs" title="{{ row.host }}">{{ row.host }}</span></td>
|
||||
<td class="cell-wrap">{% for interest in row.interests %}<span class="badge">{{ interest }}</span> {% endfor %}{% for article in row.articles %}<div class="line-clamp-1 text-xs"><a href="/dashboard/articles/{{ article.id }}" title="{{ article.title }}">{{ article.title }}</a></div>{% endfor %}</td>
|
||||
<td class="num">{{ row.article_count }}</td>
|
||||
<td class="cell-tight text-xs text-muted"><span title="First seen {{ row.first_seen }} · last seen {{ row.last_seen }}">{{ row.seen }}</span></td>
|
||||
<td>{% if status == "candidate" %}<div class="flex flex-wrap items-start gap-2"><form class="form-inline flex-nowrap" method="post" action="/dashboard/feeds/{{ row.id }}/add"><select class="min-w-0 max-w-36 truncate" name="category_id" aria-label="Category for {{ row.label }}"{% if categories.is_empty() %} disabled{% endif %}>{% for category in categories %}<option value="{{ category.id }}"{% if category.id == selected_category %} selected{% endif %}>{{ category.title }}</option>{% endfor %}</select><button class="btn-primary" type="submit"{% if categories.is_empty() %} disabled title="Miniflux categories are unavailable"{% endif %}>Add</button></form><form method="post" action="/dashboard/feeds/{{ row.id }}/dismiss"><button class="btn" type="submit">Dismiss</button></form></div>{% else %}<span class="muted text-xs">{{ row.decided_at }}{% if let Some(href) = row.miniflux_href %} · <a href="{{ href }}" rel="noopener" target="_blank">in Miniflux</a>{% endif %}</span>{% endif %}</td>
|
||||
</tr>{% endfor %}{% if rows.is_empty() %}<tr><td colspan="6" class="text-muted">No feed candidates with this status. The discovery stage adds them during a run; <code>daily-epub feeds discover</code> backfills from recent articles.</td></tr>{% endif %}</tbody>
|
||||
</table>
|
||||
{% include "dashboard/_pager.html" %}
|
||||
</section>{% endblock %}
|
||||
@@ -10,6 +10,7 @@
|
||||
<div class="tile"><span class="tile-label">Verdicts, 7 days</span><span class="tile-num">{{ ratings_total }}</span><span class="tile-delta"><a href="/dashboard/ratings">rating history</a></span></div>
|
||||
<div class="tile"><span class="tile-label">Unrated picks</span><span class="tile-num">{{ unrated.len() }}</span><span class="tile-delta">from the last three issues</span></div>
|
||||
<div class="tile"><span class="tile-label">Active jobs</span><span class="tile-num">{{ active_jobs.len() }}</span><span class="tile-delta"><a href="/dashboard/jobs">all jobs</a></span></div>
|
||||
<div class="tile"><span class="tile-label">Feed candidates</span><span class="tile-num">{{ feed_candidates }}</span><span class="tile-delta"><a href="/dashboard/feeds">review feeds</a></span></div>
|
||||
<div class="tile"><span class="tile-label">Access requests</span><span class="tile-num">{{ access_requests }}</span><span class="tile-delta"><a href="/dashboard/users">review requests</a></span></div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -47,6 +47,7 @@
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "runs" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/runs"{% if page.active_nav == "runs" %} aria-current="page"{% endif %}>Runs</a>
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "articles" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/articles"{% if page.active_nav == "articles" %} aria-current="page"{% endif %}>Articles</a>
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "ratings" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/ratings"{% if page.active_nav == "ratings" %} aria-current="page"{% endif %}>Ratings</a>
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "feeds" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/feeds"{% if page.active_nav == "feeds" %} aria-current="page"{% endif %}>Feeds</a>
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "profile" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/profile"{% if page.active_nav == "profile" %} aria-current="page"{% endif %}>Profile</a>
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "stats" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/stats"{% if page.active_nav == "stats" %} aria-current="page"{% endif %}>Stats</a>
|
||||
<a class="flex min-h-10 min-w-10 items-center justify-center border-b-2 no-underline hover:text-ink {% if page.active_nav == "jobs" %}border-accent text-ink{% else %}border-transparent text-muted{% endif %}" href="/dashboard/jobs"{% if page.active_nav == "jobs" %} aria-current="page"{% endif %}>Jobs</a>
|
||||
|
||||
Reference in new issue
Block a user