the-daily-epub/docs/plans/2026-08-15-the-daily-epub.md

# The Daily EPUB — Project Plan


## 1. Overview

**The Daily EPUB** is a Rust service that generates a personalized daily newspaper as an EPUB. Every morning it:

1. Pulls the last ~24h of entries from a self-hosted **Miniflux** instance (300–500 articles/day).
2. Deduplicates, extracts full text, and enriches entries with social-proof signals (HackerNews, Lobsters, Reddit).
3. Applies cheap heuristic pre-filters, then uses **DeepSeek V4 Flash** to score, select, and organize ~15–25 articles into newspaper sections.
4. Generates editorial framing: a front-page "day in brief," section intros, and per-article summaries.
5. Builds two EPUB editions (standard + Xteink X4-optimized), converts the X4 edition to XTC/XTCH.
6. Publishes into a dedicated **BookOrbit** library via watched folder (→ OPDS for KOReader devices) and serves XTC via a minimal built-in OPDS feed.
7. Collects 👍/👎 feedback via rating links inside the EPUB to continuously improve curation.

**Reader profile (bake into curation prompts):** prefers long-form, high-effort, well-written articles on *any* topic; uses social proof (HN/Reddit upvotes+comments) as a quality proxy; wants tech news, light general/US world news (prefers Wikipedia Current Events for world news), Boston-area news, and ultra-niche community news. The full interest list lives in `data/scour-interests.opml` (~220 Scour interests: Rust, systems programming, e-ink, self-hosting, PKM, sci-fi, creative coding, space, running, board games, Boston Tech, etc.) — compile it into the taste profile at build time.

### Existing infrastructure (all on this server)

| Service | Local | External | Notes |
|---|---|---|---|
| Miniflux | `127.0.0.1:8082` | `miniflux.hallada.net` | Installed via PPA. API auth via `X-Auth-Token` header. |
| BookOrbit | `127.0.0.1:3498` | `bookorbit.hallada.net` | NestJS/Vue/Postgres. Supports multiple isolated libraries, per-library watched folders, OPDS at `/api/v1/opds` (Basic auth, `opds_access` permission). |
| The Daily EPUB (new) | `127.0.0.1:<port, e.g. 3499>` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + XTC OPDS + static files. |

### Secrets/config the operator must provide

- `MINIFLUX_API_KEY` (create in Miniflux: Settings → API Keys)
- `DEEPSEEK_API_KEY`
- `DAILY_EPUB_SECRET` (random 32+ bytes; HMAC for rating links)
- BookOrbit: create a "The Daily EPUB" library with its own folder, enable **Watch folders** for it; note the folder path for config.
- Reverse proxy entry for `daily.hallada.net` → `127.0.0.1:3499` (rating links must be reachable from devices on the internet; TLS via existing setup).
- Node.js 18+ and `epub-to-xtc-converter` CLI installed (`npm i -g` per its README) for XTC output.

### Prior art studied

- **feedpaper** (heyjonny.dev / jonashonecker/feedpaper): Feedbin → filter unsuitable feeds → EPUB → manual copy to X4. Lesson: filter out YouTube/link-only/JS-heavy sources early; simplicity works.
- **inkfeed** (adhamsalama/inkfeed): Go backend, Mozilla Readability extraction, MOBI/EPUB export, special handling for Reddit JSON and Google News redirects. Lesson: robust content extraction is the hard part.
- **Calibre news system** (manual.calibre-ebook.com/news.html): recipe model — masthead, per-section feeds, article cleanup hooks, index pages. We mirror its structure: cover → front page → sections → articles.

---

## 2. Architecture

Single Rust binary crate `daily-epub` (workspace not needed yet) with clap subcommands:

```
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
daily-epub serve                # long-running: rating endpoints + XTC OPDS + static
daily-epub profile rebuild      # regenerate taste profile from ratings (also runs weekly inside generate)
daily-epub backfill-social      # re-poll social scores for recent entries (optional helper)
daily-epub db migrate           # run sqlx migrations (also auto-run on start)
```

`generate` is invoked by a systemd timer each morning; `serve` runs as a persistent systemd service. Both share one SQLite database.

### Pipeline (inside `generate`)

```
Miniflux ingest → normalize/dedupe → content extraction → social enrichment
  → heuristic pre-filter (500 → ~120) → LLM scoring (batched) → LLM selection (~120 → 15–25)
  → comment fetching for selected → LLM editorial (summaries, section intros, front page)
  → EPUB build (standard + X4 editions) → XTC conversion → publish (BookOrbit folder, XTC dir, OPDS xml)
  → retention pruning → run report logged + stored
```

Every stage writes to SQLite so the run is resumable/idempotent per date: re-running `generate --date X` replaces that issue.

### Repository layout

```
the-daily-epub/
├── Cargo.toml
├── config.example.toml
├── data/scour-interests.opml          # (already present)
├── docs/plans/
├── migrations/                        # sqlx sqlite migrations
├── systemd/
│   ├── daily-epub.service             # serve
│   ├── daily-epub-generate.service    # oneshot
│   └── daily-epub-generate.timer
├── src/
│   ├── main.rs                        # clap dispatch
│   ├── config.rs                      # figment: TOML + env overrides
│   ├── db.rs                          # sqlx pool, queries
│   ├── miniflux.rs                    # API client
│   ├── extract.rs                     # readability + sanitization + word counts
│   ├── social/
│   │   ├── mod.rs                     # SocialRef model, orchestrator
│   │   ├── hn.rs                      # Algolia search + item tree
│   │   ├── lobsters.rs                # /s/{id}.json
│   │   └── reddit.rs                  # api/info.json + comments .json
│   ├── curate/
│   │   ├── prefilter.rs               # heuristics + feed priors
│   │   ├── llm.rs                     # DeepSeek client (async-openai, custom base)
│   │   ├── score.rs                   # batched scoring stage
│   │   ├── select.rs                  # lineup selection stage
│   │   ├── editorial.rs              # summaries, intros, front page
│   │   └── profile.rs                # taste profile build/rebuild
│   ├── comments.rs                    # comment tree → rendered XHTML
│   ├── epub/
│   │   ├── build.rs                   # epub-builder assembly
│   │   ├── templates/                 # askama XHTML templates + CSS
│   │   ├── images.rs                  # download, resize, grayscale, re-encode
│   │   └── x4.rs                      # X4 edition transforms + XTC CLI invocation
│   ├── publish.rs                     # copy to BookOrbit folder, OPDS xml gen, retention
│   ├── server.rs                      # axum: /r/… ratings, /opds/xtc.xml, /files/…
│   └── report.rs                      # run summary (counts, cost, timings)
└── tests/                             # integration tests with fixture JSON
```

### Crate choices (all popular, well-maintained)

| Concern | Crate |
|---|---|
| async runtime | `tokio` |
| HTTP client | `reqwest` (rustls-tls, gzip, cookies off) |
| HTTP server | `axum` + `tower-http` (trace, fs) |
| serialization | `serde`, `serde_json` |
| DB | `sqlx` (sqlite, runtime-tokio, migrations) |
| CLI | `clap` (derive) |
| config | `figment` (TOML file + `DAILY_EPUB_*` env) |
| errors | `thiserror` (lib-ish modules) + `anyhow` (top level) |
| logging | `tracing` + `tracing-subscriber` (env-filter) |
| time | `jiff` (tz-aware; day boundaries in `America/New_York`) |
| feeds/URLs | `url`; (feed parsing not needed — Miniflux does it) |
| readability | `dom_smoothie` (Rust port of Mozilla Readability; fallback `readability` crate if issues) |
| HTML manipulation | `scraper` (select/rewrite), `ammonia` (sanitize to safe XHTML subset) |
| templating | `askama` (typed XHTML templates) |
| EPUB | `epub-builder` (EPUB3, nav TOC, resources, cover) |
| images | `image` (decode, resize, grayscale, JPEG encode) |
| LLM | `async-openai` with `OpenAIConfig::new().with_api_base("https://api.deepseek.com/v1")` |
| auth tokens | `hmac` + `sha2`, `hex` |
| retry | `backoff` or hand-rolled with `tokio::time` (jittered exponential, max 3) |

---

## 3. Stage details

### 3.1 Miniflux ingestion (`miniflux.rs`)

- Client for `http://127.0.0.1:8082/v1`, header `X-Auth-Token`.
- `GET /v1/entries?order=published_at&direction=desc&published_after=<unix>&limit=250&offset=…` — page through everything published in the window `[now - lookback_hours (default 26h), now]`, **regardless of read/unread status** (never mutate read state; this must not disturb normal reader usage).
- Also `GET /v1/feeds` once per run to map `feed_id → {title, site_url, category.title}`.
- Persist raw entries. Fields used: `id`, `feed_id`, `title`, `url`, `comments_url` (hnrss/lobsters populate this — free social linkage!), `author`, `published_at`, `content` (Miniflux's stored content — full text if the feed's "fetch original content" is on, else the feed summary).
- Watermark per run stored in `kv` table; the window overlap + upsert-by-entry-id makes re-runs safe.

### 3.2 Normalize & dedupe

- **Canonical URL:** lowercase host, strip fragments, strip tracking params (`utm_*`, `ref`, `fbclid`, `gclid`, `s`, `si`), trim trailing `/`, resolve known redirectors (Google News links → target param).
- **Cluster duplicates** (same story via HN frontpage feed + Scour feed + the blog's own feed): primary key = canonical URL; secondary fuzzy pass = normalized title (lowercased, alphanumeric-only) exact match within the window. Merge into one `article` row keeping: the richest content, the union of social refs, and a `sources` list (used as a curation signal — appearing in multiple feeds is itself social proof; specifically flag "came via Scour" and "came via HN frontpage").
- Drop obvious non-articles early: audio/video enclosure-only entries, entries whose URL host is youtube/vimeo/spotify, empty-title entries.

### 3.3 Content extraction (`extract.rs`)

Priority order per article:
1. Miniflux `content` if it looks like full text (word count ≥ 250 or ≥ 80% of a fetched version).
2. Fetch `url` (10s timeout, desktop UA, max 3 MB) → `dom_smoothie` readability → main content HTML.
3. Fallback: feed summary/excerpt with a "(excerpt only — read online)" note; such articles are penalized in pre-filter unless social score is high.

Then: sanitize with `ammonia` (allow: p, h1–h4, ul/ol/li, blockquote, pre, code, em, strong, a, img, figure, figcaption, table basics, hr, br), compute `word_count`, collect image URLs (cap 12/article), detect paywall heuristically (very short text + known paywall domains list) → mark `excerpt_only`.

### 3.4 Social enrichment (`social/`)

For every deduped article (cheap, parallel with a semaphore of ~8, aggressive caching in `social` table):

- **HackerNews** (Algolia, free, generous limits):
  - If `comments_url` is `news.ycombinator.com/item?id=N` → that's the story id.
  - Else `GET https://hn.algolia.com/api/v1/search?query=<canonical_url>&restrictSearchableAttributes=url` → take best hit. Store `points`, `num_comments`, `objectID`.
- **Lobsters:** only when the article arrived via a lobste.rs feed or `comments_url` points at `lobste.rs/s/<id>` (no public URL-search API) → later fetch `https://lobste.rs/s/<id>.json` for score + comments.
- **Reddit:** `GET https://www.reddit.com/api/info.json?url=<canonical_url>` with a descriptive User-Agent (`the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`) → best post by score; store `score`, `num_comments`, `permalink`. Respect ~1 req/sec pacing; on 429 back off and continue (social data is best-effort).
- **X/Twitter:** **not supported** — no free API. Documented limitation; the `source` enum leaves room to add it later.

Composite `social_score = log10(1 + hn_points) + 0.7*log10(1 + reddit_score) + log10(1 + lobsters_score) + 0.5*log10(1 + total_comments)`.

### 3.5 Heuristic pre-filter (`curate/prefilter.rs`) — 300–500 → ~120

Score each article 0–100; keep top `prefilter_keep` (default 120) plus all auto-includes:

- **Auto-include:** articles from feeds in the configured `always_include_feeds` list (the infrequent personal blogs Tyler always reads) skip filtering *and* LLM scoring is still run for section/summary purposes but they can't be dropped.
- `+` word count (long-form preference: 0 pts <300 words, scaling to max at ~2500+)
- `+` social_score (scaled)
- `+` came via Scour (it already matched his interests), `+` came via HN frontpage
- `+` feed prior (see §3.9: per-feed Bayesian upvote rate from ratings history)
- `−` excerpt_only, `−` title looks like link-roundup/release-notes/sponsor post (regex list), `−` domain on a configurable blocklist
- **Dedup vs. history:** exclude anything already included in a previous issue (`issue_articles`), and anything the LLM scored < 3 within the last 7 days (don't re-score churn).

This stage is pure Rust, free, and keeps LLM cost flat as feed volume grows.

### 3.6 LLM curation (`curate/llm.rs`, `score.rs`, `select.rs`) — DeepSeek V4 Flash

Client: `async-openai` against `https://api.deepseek.com/v1`, model id from config (default `deepseek-v4-flash` — **verify exact model id against DeepSeek docs at implementation time**), `response_format: json_object`, temperature 0.3 for scoring / 0.8 for editorial. DeepSeek automatically prefix-caches, so put the (identical, long) system prompt first in every request: cached input is $0.0028/M vs $0.14/M.

**Taste profile (system prompt core, `curate/profile.rs`):** a ~600-word document assembled from: (a) the interest names parsed out of `data/scour-interests.opml`, grouped into themes; (b) hard-coded stated preferences (long-form, effort, any topic if excellent, social proof matters, Boston local, ultra-niche community news, Wikipedia-style neutral world news); (c) a "learned adjustments" section regenerated weekly by an LLM call that summarizes recent 👍/👎 ratings ("consistently downvotes: crypto press releases; consistently upvotes: database internals deep-dives…"). Stored in the DB (`kv`) and versioned.

**Stage A — scoring (batched):** batches of 12 articles per request. Per article send: title, source feed, author, word count, social stats, sources list, and a ~200-word excerpt. Output JSON per article: `{id, score: 0-10, category, rationale (≤20 words), is_paywalled_guess}`. ~120 articles = 10 requests ≈ 90k input (mostly cache-miss article text) + ~4k output ≈ **$0.02**.

**Stage B — lineup selection (single call):** send the top ~40 by combined score (LLM score weighted with social + priors) with their rationales. Output: final 15–25 picks (`target_article_count` config, default 20), each assigned a **section**, an ordering, and one flagged `lead_story`. Sections chosen from a configured palette (LLM may only use these): *Top Stories; Tech & Engineering; Science & Space; AI & Machine Learning; Culture & Essays; Boston & Local; Niche Corner; From the Blogroll* (auto-includes land here by default); *World Briefing* is reserved (§3.8). Empty sections are omitted.

**Stage C — editorial:**
- Per selected article, one call with full text (truncated to ~5k tokens): 2–3 sentence summary written like a newspaper abstract (what it argues, why it's worth reading — not clickbait). 20 calls ≈ 100k input / 3k output ≈ **$0.015**.
- One call for the front page: given the lineup + summaries, write "**From the Editor**" — 250–400 words identifying the day's themes and guiding the read — plus a 2–3 sentence intro per section. Voice: warm, literate, a little playful; never fabricates facts not present in the summaries.

**Cost guardrail:** track token usage per run (returned in API responses) in `runs`; config `max_daily_usd` (default 2.00) — if exceeded mid-run, skip remaining editorial calls and fall back to feed excerpts as summaries, log loudly. Expected steady-state cost: **≈ $0.05–0.30/day**, far under the $5 ceiling, with headroom to feed more/fuller text later.

### 3.7 Comment chapters (`comments.rs`)

For each **selected** article with social refs:

- **HN:** `GET https://hn.algolia.com/api/v1/items/{objectID}` → full tree.
- **Lobsters:** `GET https://lobste.rs/s/{id}.json`.
- **Reddit:** `GET https://www.reddit.com{permalink}.json?limit=100&depth=3&sort=top`.

Rendering (heuristic, no LLM): pick top ~8 top-level threads by score, depth ≤ 3, ≤ 4 children per node, per-comment cap 1,200 chars (ellipsize), whole chapter cap ~4,000 words. Render as nested `<blockquote>`-style indentation with author + points + relative depth styling that reads well on e-ink (no color, border-left indent). Sanitize with `ammonia`. Each discussion becomes its own chapter titled "💬 Discussion: {article title} ({N} comments on {source})", placed immediately after its article and nested under it in the TOC. Multiple sources = one chapter with per-source subsections, ordered HN → Lobsters → Reddit.

### 3.8 World Briefing (Wikipedia Current Events)

Since Tyler prefers Wikipedia's Current Events portal for world news, include it directly rather than curating wire-service articles: fetch the day's portal page (`https://en.wikipedia.org/wiki/Portal:Current_events/{YYYY}_{Month}_{D}` via the MediaWiki REST HTML API), extract the day's bulleted events, strip citations/edit links, keep internal links as plain text, and render as a compact "World Briefing" section chapter with CC BY-SA attribution + link. Config-toggleable (`world_briefing = true`). Failure is non-fatal (skip section).

### 3.9 Feedback loop (`server.rs` + `curate/profile.rs`)

- Each article chapter ends with a footer:
  `Was this a good pick? [ 👍 Yes ] · [ 👎 No ]` + `Read online ↗` (original URL).
- Link format: `https://daily.hallada.net/r/{issue_date}/{article_id}/{up|down}?t={token}` where `token = hex(hmac_sha256(secret, "{issue_date}/{article_id}/{vote}"))[..16]`. GET (KOReader opens links in its built-in browser/prompt; GET is the only thing that works from an e-reader). Idempotent upsert; response is a tiny static HTML page ("Recorded 👍 — thanks!") sized for e-ink browsers.
- Ratings drive: (a) **feed priors** — per-feed `(upvotes+1)/(upvotes+downvotes+2)` beta-smoothed score used in pre-filter; (b) the weekly **learned adjustments** rewrite of the taste profile (§3.6).
- Future (out of v1 scope, schema-ready): embedding-based classifier — `fastembed` (bge-small ONNX) embeddings + `linfa` logistic regression over rated articles as an additional pre-filter signal once ≥ ~200 ratings exist.

### 3.10 EPUB assembly (`epub/`)

Built with `epub-builder` (EPUB3 + nav + NCX fallback), content pages from `askama` templates, all assets embedded (fully offline). Structure:

1. **Cover** — generated PNG: masthead "The Daily EPUB", date ("Friday, August 15, 2026"), issue number (days since first issue), article count. Render simple typographic SVG → rasterize (via `resvg`+`tiny-skia` — small, pure Rust) at 1200×1600 (standard) / 480×800 grayscale (X4).
2. **From the Editor** — front-page brief + issue stats line ("22 articles · ~1h 45m read · 6 sections").
3. **In This Issue** — the introduction chapter: per-section, each article's title, source, reading time, and its 2–3 sentence summary, linked to the chapter.
4. **Sections** — section title page (name + LLM intro), then article chapters: header (title, author, source, date, word count/reading time, social stats line "▲ 342 on HN · 210 comments"), cleaned body with embedded images, footer (rating links + read-online link). Discussion chapter follows when present.
5. **World Briefing** section (when enabled).
6. **Colophon** — generation timestamp, models used, token cost, source feed counts.

TOC: nav depth 2 (sections → articles, discussions nested). Metadata: `dc:title` "The Daily EPUB — 2026-08-15", `dc:creator` "The Daily EPUB", `dc:date`, `dc:language en`, EPUB3 `belongs-to-collection` = "The Daily EPUB" with `group-position` = issue number (BookOrbit/KOReader sort correctly). Deterministic chapter ids (`art-{entry_id}`) so rating links and TOC stay stable across regenerations.

**Images (`epub/images.rs`):** download (10s timeout, 5 MB cap, semaphore 8), re-encode with `image`:
- *Standard edition:* max width 1200px, JPEG q80 (PNG kept for line art/transparency after white-flatten), strip metadata (re-encode does), skip decorative images < 24px, drop SVG/WebP-source images unless decodable, per-issue asset budget ~25 MB.
- *X4 edition (`epub/x4.rs`):* grayscale (Luma8), fit within 480×800, JPEG q70, flatten transparency to white; simplified CSS (no floats/flex/grid, no embedded fonts, larger base font, generous line-height, hyphenation on); cover at native 480×800. (These mirror what `epub-to-xtc-converter` recommends, so the XTC conversion step has ideal input.)
- Every `<img>` gets `alt` preserved and a `<figcaption>` if source had one; failed downloads degrade to a "[image: alt text]" placeholder paragraph.

**CSS:** one small stylesheet per edition tuned for e-ink: serif body, no colors other than grayscale, `page-break-before` on chapters, blockquote-indent comment styling.

### 3.11 XTC conversion & publishing (`publish.rs`)

- Run the `epub-to-xtc-converter` CLI (Node 18+) on the X4 edition: invoke via `tokio::process::Command`, config keys `xtc.command` (default `epub-to-xtc`) and `xtc.args` (verify exact CLI name/flags from the repo README at implementation time; support both `.xtc` 1-bit and `.xtch` 4-level grayscale via config, default XTCH for image quality). Non-zero exit → log error, continue (XTC is a bonus artifact).
- **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.bookorbit_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) — KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered.
- **XTC delivery:** copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead.
- **Retention:** delete issue files older than `retention_days` (default 21) from both dirs (BookOrbit's scan removes the DB entries); SQLite issue/rating history is kept forever (it's the training data).

### 3.12 Server (`server.rs`)

axum on `127.0.0.1:3499`:
- `GET /r/{date}/{article_id}/{vote}?t=` — verify HMAC, upsert rating, tiny HTML response. No auth beyond the token (links live inside a private EPUB; tokens are per-article+vote and unguessable).
- `GET /opds/xtc.xml`, `GET /files/xtc/{name}` — optional Basic auth.
- `GET /healthz`, `GET /issues.json` (recent run reports; handy for debugging).
- `tower-http` request tracing; graceful shutdown on SIGTERM.

### 3.13 Database schema (sqlite, `migrations/`)

```sql
entries(id INTEGER PRIMARY KEY,            -- miniflux entry id
  feed_id INT, feed_title TEXT, category TEXT, title TEXT, url TEXT,
  canonical_url TEXT, author TEXT, published_at TEXT, comments_url TEXT,
  raw_content TEXT, fetched_at TEXT);
articles(id INTEGER PRIMARY KEY AUTOINCREMENT,   -- deduped cluster
  canonical_url TEXT UNIQUE, title TEXT, best_entry_id INT REFERENCES entries(id),
  content_html TEXT, word_count INT, excerpt_only BOOL, image_count INT,
  sources_json TEXT, first_seen TEXT);
social(article_id INT, source TEXT CHECK(source IN ('hn','lobsters','reddit','x')),
  item_id TEXT, score INT, num_comments INT, item_url TEXT, fetched_at TEXT,
  PRIMARY KEY (article_id, source));
scores(article_id INT, run_date TEXT, prefilter_score REAL, llm_score REAL,
  llm_category TEXT, rationale TEXT, PRIMARY KEY (article_id, run_date));
issues(date TEXT PRIMARY KEY, issue_number INT, generated_at TEXT,
  epub_path TEXT, x4_path TEXT, xtc_path TEXT, front_page_html TEXT, report_json TEXT);
issue_articles(issue_date TEXT, article_id INT, section TEXT, position INT,
  is_lead BOOL, summary TEXT, PRIMARY KEY (issue_date, article_id));
ratings(issue_date TEXT, article_id INT, vote INT CHECK(vote IN (-1,1)),
  rated_at TEXT, PRIMARY KEY (issue_date, article_id));
feed_priors(feed_id INT PRIMARY KEY, upvotes INT, downvotes INT, included INT);
runs(id INTEGER PRIMARY KEY AUTOINCREMENT, date TEXT, started_at TEXT, finished_at TEXT,
  entries_fetched INT, candidates INT, selected INT,
  input_tokens INT, cached_tokens INT, output_tokens INT, cost_usd REAL, status TEXT, error TEXT);
kv(key TEXT PRIMARY KEY, value TEXT);      -- watermark, taste_profile, profile_version
```

### 3.14 Configuration (`config.example.toml`)

```toml
timezone = "America/New_York"
lookback_hours = 26
target_article_count = 20
prefilter_keep = 120
retention_days = 21
max_daily_usd = 2.0
world_briefing = true

[miniflux]
base_url = "http://127.0.0.1:8082"
# api_key via DAILY_EPUB_MINIFLUX__API_KEY env

[deepseek]
base_url = "https://api.deepseek.com/v1"
model = "deepseek-v4-flash"        # verify exact id
# api_key via env

[curation]
always_include_feeds = []           # miniflux feed ids or site urls
blocked_domains = []
sections = ["Top Stories", "Tech & Engineering", "Science & Space",
  "AI & Machine Learning", "Culture & Essays", "Boston & Local",
  "Niche Corner", "From the Blogroll"]

[publish]
bookorbit_dir = "/srv/bookorbit/libraries/daily-epub"
xtc_dir = "/var/lib/daily-epub/xtc"

[xtc]
enabled = true
command = "epub-to-xtc"             # verify CLI name/flags from repo
format = "xtch"                     # xtc | xtch

[server]
bind = "127.0.0.1:3499"
public_url = "https://daily.hallada.net"
# hmac_secret via env; optional basic auth user/pass for OPDS
```

### 3.15 Deployment (systemd, `systemd/`)

- `daily-epub.service`: `ExecStart=/usr/local/bin/daily-epub serve`, `Restart=on-failure`, hardening (`DynamicUser` or dedicated user, `StateDirectory=daily-epub`, `ProtectSystem=strict` with write access to publish dirs).
- `daily-epub-generate.service` (oneshot) + `daily-epub-generate.timer`: `OnCalendar=*-*-* 05:30:00 America/New_York`, `Persistent=true` (catch up after downtime), `RandomizedDelaySec=300`.
- Install: `cargo build --release`, copy binary, `systemctl enable --now`. Reverse-proxy `daily.hallada.net` → `127.0.0.1:3499`.

---

## 4. Implementation milestones (each independently verifiable)

1. **M1 — Skeleton & ingest:** crate scaffold, config, migrations, `miniflux.rs`, `generate --dry-run` prints fetched entry stats. *Verify: run against live Miniflux, see ~daily volume.*
2. **M2 — Dedupe + extraction + social:** articles table populated with full text, word counts, HN/Reddit/Lobsters scores. *Verify: spot-check known HN stories carry correct points.*
3. **M3 — Pre-filter + LLM scoring/selection:** end-to-end lineup JSON printed in dry-run; token/cost report. *Verify: lineup is sane; cost < $0.50.*
4. **M4 — EPUB standard edition + publish:** full issue EPUB with cover, front page (temporary plain summaries), sections, articles, images; lands in BookOrbit, visible via OPDS on Kindle. *Verify: epubcheck clean; opens in KOReader with working TOC.*
5. **M5 — Editorial + comments:** DeepSeek summaries/intros/front page wired in; discussion chapters. *Verify: read an issue; comments legible on e-ink.*
6. **M6 — X4 edition + XTC + XTC OPDS:** second edition, converter invocation, static OPDS feed. *Verify: X4 fetches and renders both.*
7. **M7 — Feedback loop:** `serve` rating endpoints, links in chapters, feed priors in pre-filter, weekly profile rebuild. *Verify: tap 👍 in KOReader → row in `ratings` → prior changes next run.*
8. **M8 — Hardening & ops:** systemd units, retention, cost guardrail, run reports, `issues.json`, README.

## 5. Verification (end-to-end)

- `cargo test` — unit tests: URL canonicalization, dedupe clustering, HMAC round-trip, comment-tree truncation, prefilter scoring; integration tests over fixture JSON (recorded Miniflux/Algolia/Reddit responses) with the LLM stage mocked (`--skip-llm` uses prefilter order).
- `daily-epub generate --dry-run --out ./out --max-articles 6` with real keys → inspect `./out/*.epub` in Calibre + run `epubcheck` (if installed) → zero errors.
- Full live run: `daily-epub generate` → file appears in BookOrbit UI under the Daily EPUB library only → browse BookOrbit OPDS from KOReader (Kindle/Palma), download, read; X4: CrossPoint OPDS → both the X4 EPUB (via BookOrbit) and XTC (via `daily.hallada.net/opds/xtc.xml`).
- Tap a rating link on the Kindle → confirmation page loads → `sqlite3 … 'select * from ratings'` shows the vote.
- Watch `runs` for a week: cost per day, selection quality; tune `prefilter_keep`/prompts.

## 6. Future ideas (explicitly out of v1 scope; don't constrain the design)

Weekly "Sunday Edition" retrospective; LLM editorials/opinion columns on the day's themes; discussion summarization for 500+ comment threads; embedding-based personal ranker (fastembed + linfa) once ratings accumulate; weather/on-this-day front-page ear boxes; a puzzle page; per-section reading-time budgets; Miniflux starred-entry import as implicit positive signal; TTS audio edition; X/Twitter comments if API access ever becomes viable.

## 7. Known limitations & notes

- X/Twitter comments are omitted (no free API).
- Lobsters linkage only works when the entry originated from a lobste.rs feed (no URL-search API).
- Paywalled articles degrade to excerpt + link; they're penalized but not banned (social proof can still surface them).
- The X4 can't follow rating links (no browser) — accepted; rating happens from KOReader devices.
- DeepSeek exact model id and `epub-to-xtc-converter` CLI flags must be confirmed against current docs during implementation (both noted inline).
- Pricing basis (Aug 2026): DeepSeek V4 Flash ≈ $0.14/M input (cache-miss), $0.0028/M cached input, $0.28/M output — steady-state ≈ $0.05–0.30/day, hard-capped by `max_daily_usd`.