# The Daily EPUB β€” Project Plan ## 1. Overview **The Daily EPUB** is a Rust service that generates a personalized daily newspaper as an EPUB. Every morning it: 1. Pulls the last ~24h of entries from a self-hosted **Miniflux** instance (300–500 articles/day). 2. Deduplicates, extracts full text, and enriches entries with social-proof signals (HackerNews, Lobsters, Reddit). 3. Applies cheap heuristic pre-filters, then uses **DeepSeek V4 Flash** to score, select, and organize ~15–25 articles into newspaper sections. 4. Generates editorial framing: a front-page "day in brief," section intros, and per-article summaries. 5. Builds two EPUB editions (standard + Xteink X4-optimized), converts the X4 edition to XTC/XTCH. 6. Publishes into a dedicated **BookOrbit** library via watched folder (β†’ OPDS for KOReader devices) and serves XTC via a minimal built-in OPDS feed. 7. Collects πŸ‘/πŸ‘Ž feedback via rating links inside the EPUB to continuously improve curation. **Reader profile (bake into curation prompts):** prefers long-form, high-effort, well-written articles on *any* topic; uses social proof (HN/Reddit upvotes+comments) as a quality proxy; wants tech news, light general/US world news (prefers Wikipedia Current Events for world news), Boston-area news, and ultra-niche community news. The full interest list lives in `data/scour-interests.opml` (~220 Scour interests: Rust, systems programming, e-ink, self-hosting, PKM, sci-fi, creative coding, space, running, board games, Boston Tech, etc.) β€” compile it into the taste profile at build time. ### Existing infrastructure (all on this server) | Service | Local | External | Notes | |---|---|---|---| | Miniflux | `127.0.0.1:8082` | `miniflux.hallada.net` | Installed via PPA. API auth via `X-Auth-Token` header. | | BookOrbit | `127.0.0.1:3498` | `bookorbit.hallada.net` | NestJS/Vue/Postgres. Supports multiple isolated libraries, per-library watched folders, OPDS at `/api/v1/opds` (Basic auth, `opds_access` permission). | | The Daily EPUB (new) | `127.0.0.1:` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + XTC OPDS + static files. | ### Secrets/config the operator must provide - `MINIFLUX_API_KEY` (create in Miniflux: Settings β†’ API Keys) - `DEEPSEEK_API_KEY` - `DAILY_EPUB_SECRET` (random 32+ bytes; HMAC for rating links) - BookOrbit: create a "The Daily EPUB" library with its own folder, enable **Watch folders** for it; note the folder path for config. - Reverse proxy entry for `daily.hallada.net` β†’ `127.0.0.1:3499` (rating links must be reachable from devices on the internet; TLS via existing setup). - Node.js 18+ and `epub-to-xtc-converter` CLI installed (`npm i -g` per its README) for XTC output. ### Prior art studied - **feedpaper** (heyjonny.dev / jonashonecker/feedpaper): Feedbin β†’ filter unsuitable feeds β†’ EPUB β†’ manual copy to X4. Lesson: filter out YouTube/link-only/JS-heavy sources early; simplicity works. - **inkfeed** (adhamsalama/inkfeed): Go backend, Mozilla Readability extraction, MOBI/EPUB export, special handling for Reddit JSON and Google News redirects. Lesson: robust content extraction is the hard part. - **Calibre news system** (manual.calibre-ebook.com/news.html): recipe model β€” masthead, per-section feeds, article cleanup hooks, index pages. We mirror its structure: cover β†’ front page β†’ sections β†’ articles. --- ## 2. Architecture Single Rust binary crate `daily-epub` (workspace not needed yet) with clap subcommands: ``` daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] daily-epub serve # long-running: rating endpoints + XTC OPDS + static daily-epub profile rebuild # regenerate taste profile from ratings (also runs weekly inside generate) daily-epub backfill-social # re-poll social scores for recent entries (optional helper) daily-epub db migrate # run sqlx migrations (also auto-run on start) ``` `generate` is invoked by a systemd timer each morning; `serve` runs as a persistent systemd service. Both share one SQLite database. ### Pipeline (inside `generate`) ``` Miniflux ingest β†’ normalize/dedupe β†’ content extraction β†’ social enrichment β†’ heuristic pre-filter (500 β†’ ~120) β†’ LLM scoring (batched) β†’ LLM selection (~120 β†’ 15–25) β†’ comment fetching for selected β†’ LLM editorial (summaries, section intros, front page) β†’ EPUB build (standard + X4 editions) β†’ XTC conversion β†’ publish (BookOrbit folder, XTC dir, OPDS xml) β†’ retention pruning β†’ run report logged + stored ``` Every stage writes to SQLite so the run is resumable/idempotent per date: re-running `generate --date X` replaces that issue. ### Repository layout ``` the-daily-epub/ β”œβ”€β”€ Cargo.toml β”œβ”€β”€ config.example.toml β”œβ”€β”€ data/scour-interests.opml # (already present) β”œβ”€β”€ docs/plans/ β”œβ”€β”€ migrations/ # sqlx sqlite migrations β”œβ”€β”€ systemd/ β”‚ β”œβ”€β”€ daily-epub.service # serve β”‚ β”œβ”€β”€ daily-epub-generate.service # oneshot β”‚ └── daily-epub-generate.timer β”œβ”€β”€ src/ β”‚ β”œβ”€β”€ main.rs # clap dispatch β”‚ β”œβ”€β”€ config.rs # figment: TOML + env overrides β”‚ β”œβ”€β”€ db.rs # sqlx pool, queries β”‚ β”œβ”€β”€ miniflux.rs # API client β”‚ β”œβ”€β”€ extract.rs # readability + sanitization + word counts β”‚ β”œβ”€β”€ social/ β”‚ β”‚ β”œβ”€β”€ mod.rs # SocialRef model, orchestrator β”‚ β”‚ β”œβ”€β”€ hn.rs # Algolia search + item tree β”‚ β”‚ β”œβ”€β”€ lobsters.rs # /s/{id}.json β”‚ β”‚ └── reddit.rs # api/info.json + comments .json β”‚ β”œβ”€β”€ curate/ β”‚ β”‚ β”œβ”€β”€ prefilter.rs # heuristics + feed priors β”‚ β”‚ β”œβ”€β”€ llm.rs # DeepSeek client (async-openai, custom base) β”‚ β”‚ β”œβ”€β”€ score.rs # batched scoring stage β”‚ β”‚ β”œβ”€β”€ select.rs # lineup selection stage β”‚ β”‚ β”œβ”€β”€ editorial.rs # summaries, intros, front page β”‚ β”‚ └── profile.rs # taste profile build/rebuild β”‚ β”œβ”€β”€ comments.rs # comment tree β†’ rendered XHTML β”‚ β”œβ”€β”€ epub/ β”‚ β”‚ β”œβ”€β”€ build.rs # epub-builder assembly β”‚ β”‚ β”œβ”€β”€ templates/ # askama XHTML templates + CSS β”‚ β”‚ β”œβ”€β”€ images.rs # download, resize, grayscale, re-encode β”‚ β”‚ └── x4.rs # X4 edition transforms + XTC CLI invocation β”‚ β”œβ”€β”€ publish.rs # copy to BookOrbit folder, OPDS xml gen, retention β”‚ β”œβ”€β”€ server.rs # axum: /r/… ratings, /opds/xtc.xml, /files/… β”‚ └── report.rs # run summary (counts, cost, timings) └── tests/ # integration tests with fixture JSON ``` ### Crate choices (all popular, well-maintained) | Concern | Crate | |---|---| | async runtime | `tokio` | | HTTP client | `reqwest` (rustls-tls, gzip, cookies off) | | HTTP server | `axum` + `tower-http` (trace, fs) | | serialization | `serde`, `serde_json` | | DB | `sqlx` (sqlite, runtime-tokio, migrations) | | CLI | `clap` (derive) | | config | `figment` (TOML file + `DAILY_EPUB_*` env) | | errors | `thiserror` (lib-ish modules) + `anyhow` (top level) | | logging | `tracing` + `tracing-subscriber` (env-filter) | | time | `jiff` (tz-aware; day boundaries in `America/New_York`) | | feeds/URLs | `url`; (feed parsing not needed β€” Miniflux does it) | | readability | `dom_smoothie` (Rust port of Mozilla Readability; fallback `readability` crate if issues) | | HTML manipulation | `scraper` (select/rewrite), `ammonia` (sanitize to safe XHTML subset) | | templating | `askama` (typed XHTML templates) | | EPUB | `epub-builder` (EPUB3, nav TOC, resources, cover) | | images | `image` (decode, resize, grayscale, JPEG encode) | | LLM | `async-openai` with `OpenAIConfig::new().with_api_base("https://api.deepseek.com/v1")` | | auth tokens | `hmac` + `sha2`, `hex` | | retry | `backoff` or hand-rolled with `tokio::time` (jittered exponential, max 3) | --- ## 3. Stage details ### 3.1 Miniflux ingestion (`miniflux.rs`) - Client for `http://127.0.0.1:8082/v1`, header `X-Auth-Token`. - `GET /v1/entries?order=published_at&direction=desc&published_after=&limit=250&offset=…` β€” page through everything published in the window `[now - lookback_hours (default 26h), now]`, **regardless of read/unread status** (never mutate read state; this must not disturb normal reader usage). - Also `GET /v1/feeds` once per run to map `feed_id β†’ {title, site_url, category.title}`. - Persist raw entries. Fields used: `id`, `feed_id`, `title`, `url`, `comments_url` (hnrss/lobsters populate this β€” free social linkage!), `author`, `published_at`, `content` (Miniflux's stored content β€” full text if the feed's "fetch original content" is on, else the feed summary). - Watermark per run stored in `kv` table; the window overlap + upsert-by-entry-id makes re-runs safe. ### 3.2 Normalize & dedupe - **Canonical URL:** lowercase host, strip fragments, strip tracking params (`utm_*`, `ref`, `fbclid`, `gclid`, `s`, `si`), trim trailing `/`, resolve known redirectors (Google News links β†’ target param). - **Cluster duplicates** (same story via HN frontpage feed + Scour feed + the blog's own feed): primary key = canonical URL; secondary fuzzy pass = normalized title (lowercased, alphanumeric-only) exact match within the window. Merge into one `article` row keeping: the richest content, the union of social refs, and a `sources` list (used as a curation signal β€” appearing in multiple feeds is itself social proof; specifically flag "came via Scour" and "came via HN frontpage"). - Drop obvious non-articles early: audio/video enclosure-only entries, entries whose URL host is youtube/vimeo/spotify, empty-title entries. ### 3.3 Content extraction (`extract.rs`) Priority order per article: 1. Miniflux `content` if it looks like full text (word count β‰₯ 250 or β‰₯ 80% of a fetched version). 2. Fetch `url` (10s timeout, desktop UA, max 3 MB) β†’ `dom_smoothie` readability β†’ main content HTML. 3. Fallback: feed summary/excerpt with a "(excerpt only β€” read online)" note; such articles are penalized in pre-filter unless social score is high. Then: sanitize with `ammonia` (allow: p, h1–h4, ul/ol/li, blockquote, pre, code, em, strong, a, img, figure, figcaption, table basics, hr, br), compute `word_count`, collect image URLs (cap 12/article), detect paywall heuristically (very short text + known paywall domains list) β†’ mark `excerpt_only`. ### 3.4 Social enrichment (`social/`) For every deduped article (cheap, parallel with a semaphore of ~8, aggressive caching in `social` table): - **HackerNews** (Algolia, free, generous limits): - If `comments_url` is `news.ycombinator.com/item?id=N` β†’ that's the story id. - Else `GET https://hn.algolia.com/api/v1/search?query=&restrictSearchableAttributes=url` β†’ take best hit. Store `points`, `num_comments`, `objectID`. - **Lobsters:** only when the article arrived via a lobste.rs feed or `comments_url` points at `lobste.rs/s/` (no public URL-search API) β†’ later fetch `https://lobste.rs/s/.json` for score + comments. - **Reddit:** `GET https://www.reddit.com/api/info.json?url=` with a descriptive User-Agent (`the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`) β†’ best post by score; store `score`, `num_comments`, `permalink`. Respect ~1 req/sec pacing; on 429 back off and continue (social data is best-effort). - **X/Twitter:** **not supported** β€” no free API. Documented limitation; the `source` enum leaves room to add it later. Composite `social_score = log10(1 + hn_points) + 0.7*log10(1 + reddit_score) + log10(1 + lobsters_score) + 0.5*log10(1 + total_comments)`. ### 3.5 Heuristic pre-filter (`curate/prefilter.rs`) β€” 300–500 β†’ ~120 Score each article 0–100; keep top `prefilter_keep` (default 120) plus all auto-includes: - **Auto-include:** articles from feeds in the configured `always_include_feeds` list (the infrequent personal blogs Tyler always reads) skip filtering *and* LLM scoring is still run for section/summary purposes but they can't be dropped. - `+` word count (long-form preference: 0 pts <300 words, scaling to max at ~2500+) - `+` social_score (scaled) - `+` came via Scour (it already matched his interests), `+` came via HN frontpage - `+` feed prior (see Β§3.9: per-feed Bayesian upvote rate from ratings history) - `βˆ’` excerpt_only, `βˆ’` title looks like link-roundup/release-notes/sponsor post (regex list), `βˆ’` domain on a configurable blocklist - **Dedup vs. history:** exclude anything already included in a previous issue (`issue_articles`), and anything the LLM scored < 3 within the last 7 days (don't re-score churn). This stage is pure Rust, free, and keeps LLM cost flat as feed volume grows. ### 3.6 LLM curation (`curate/llm.rs`, `score.rs`, `select.rs`) β€” DeepSeek V4 Flash Client: `async-openai` against `https://api.deepseek.com/v1`, model id from config (default `deepseek-v4-flash` β€” **verify exact model id against DeepSeek docs at implementation time**), `response_format: json_object`, temperature 0.3 for scoring / 0.8 for editorial. DeepSeek automatically prefix-caches, so put the (identical, long) system prompt first in every request: cached input is $0.0028/M vs $0.14/M. **Taste profile (system prompt core, `curate/profile.rs`):** a ~600-word document assembled from: (a) the interest names parsed out of `data/scour-interests.opml`, grouped into themes; (b) hard-coded stated preferences (long-form, effort, any topic if excellent, social proof matters, Boston local, ultra-niche community news, Wikipedia-style neutral world news); (c) a "learned adjustments" section regenerated weekly by an LLM call that summarizes recent πŸ‘/πŸ‘Ž ratings ("consistently downvotes: crypto press releases; consistently upvotes: database internals deep-dives…"). Stored in the DB (`kv`) and versioned. **Stage A β€” scoring (batched):** batches of 12 articles per request. Per article send: title, source feed, author, word count, social stats, sources list, and a ~200-word excerpt. Output JSON per article: `{id, score: 0-10, category, rationale (≀20 words), is_paywalled_guess}`. ~120 articles = 10 requests β‰ˆ 90k input (mostly cache-miss article text) + ~4k output β‰ˆ **$0.02**. **Stage B β€” lineup selection (single call):** send the top ~40 by combined score (LLM score weighted with social + priors) with their rationales. Output: final 15–25 picks (`target_article_count` config, default 20), each assigned a **section**, an ordering, and one flagged `lead_story`. Sections chosen from a configured palette (LLM may only use these): *Top Stories; Tech & Engineering; Science & Space; AI & Machine Learning; Culture & Essays; Boston & Local; Niche Corner; From the Blogroll* (auto-includes land here by default); *World Briefing* is reserved (Β§3.8). Empty sections are omitted. **Stage C β€” editorial:** - Per selected article, one call with full text (truncated to ~5k tokens): 2–3 sentence summary written like a newspaper abstract (what it argues, why it's worth reading β€” not clickbait). 20 calls β‰ˆ 100k input / 3k output β‰ˆ **$0.015**. - One call for the front page: given the lineup + summaries, write "**From the Editor**" β€” 250–400 words identifying the day's themes and guiding the read β€” plus a 2–3 sentence intro per section. Voice: warm, literate, a little playful; never fabricates facts not present in the summaries. **Cost guardrail:** track token usage per run (returned in API responses) in `runs`; config `max_daily_usd` (default 2.00) β€” if exceeded mid-run, skip remaining editorial calls and fall back to feed excerpts as summaries, log loudly. Expected steady-state cost: **β‰ˆ $0.05–0.30/day**, far under the $5 ceiling, with headroom to feed more/fuller text later. ### 3.7 Comment chapters (`comments.rs`) For each **selected** article with social refs: - **HN:** `GET https://hn.algolia.com/api/v1/items/{objectID}` β†’ full tree. - **Lobsters:** `GET https://lobste.rs/s/{id}.json`. - **Reddit:** `GET https://www.reddit.com{permalink}.json?limit=100&depth=3&sort=top`. Rendering (heuristic, no LLM): pick top ~8 top-level threads by score, depth ≀ 3, ≀ 4 children per node, per-comment cap 1,200 chars (ellipsize), whole chapter cap ~4,000 words. Render as nested `
`-style indentation with author + points + relative depth styling that reads well on e-ink (no color, border-left indent). Sanitize with `ammonia`. Each discussion becomes its own chapter titled "πŸ’¬ Discussion: {article title} ({N} comments on {source})", placed immediately after its article and nested under it in the TOC. Multiple sources = one chapter with per-source subsections, ordered HN β†’ Lobsters β†’ Reddit. ### 3.8 World Briefing (Wikipedia Current Events) Since Tyler prefers Wikipedia's Current Events portal for world news, include it directly rather than curating wire-service articles: fetch the day's portal page (`https://en.wikipedia.org/wiki/Portal:Current_events/{YYYY}_{Month}_{D}` via the MediaWiki REST HTML API), extract the day's bulleted events, strip citations/edit links, keep internal links as plain text, and render as a compact "World Briefing" section chapter with CC BY-SA attribution + link. Config-toggleable (`world_briefing = true`). Failure is non-fatal (skip section). ### 3.9 Feedback loop (`server.rs` + `curate/profile.rs`) - Each article chapter ends with a footer: `Was this a good pick? [ πŸ‘ Yes ] Β· [ πŸ‘Ž No ]` + `Read online β†—` (original URL). - Link format: `https://daily.hallada.net/r/{issue_date}/{article_id}/{up|down}?t={token}` where `token = hex(hmac_sha256(secret, "{issue_date}/{article_id}/{vote}"))[..16]`. GET (KOReader opens links in its built-in browser/prompt; GET is the only thing that works from an e-reader). Idempotent upsert; response is a tiny static HTML page ("Recorded πŸ‘ β€” thanks!") sized for e-ink browsers. - Ratings drive: (a) **feed priors** β€” per-feed `(upvotes+1)/(upvotes+downvotes+2)` beta-smoothed score used in pre-filter; (b) the weekly **learned adjustments** rewrite of the taste profile (Β§3.6). - Future (out of v1 scope, schema-ready): embedding-based classifier β€” `fastembed` (bge-small ONNX) embeddings + `linfa` logistic regression over rated articles as an additional pre-filter signal once β‰₯ ~200 ratings exist. ### 3.10 EPUB assembly (`epub/`) Built with `epub-builder` (EPUB3 + nav + NCX fallback), content pages from `askama` templates, all assets embedded (fully offline). Structure: 1. **Cover** β€” generated PNG: masthead "The Daily EPUB", date ("Friday, August 15, 2026"), issue number (days since first issue), article count. Render simple typographic SVG β†’ rasterize (via `resvg`+`tiny-skia` β€” small, pure Rust) at 1200Γ—1600 (standard) / 480Γ—800 grayscale (X4). 2. **From the Editor** β€” front-page brief + issue stats line ("22 articles Β· ~1h 45m read Β· 6 sections"). 3. **In This Issue** β€” the introduction chapter: per-section, each article's title, source, reading time, and its 2–3 sentence summary, linked to the chapter. 4. **Sections** β€” section title page (name + LLM intro), then article chapters: header (title, author, source, date, word count/reading time, social stats line "β–² 342 on HN Β· 210 comments"), cleaned body with embedded images, footer (rating links + read-online link). Discussion chapter follows when present. 5. **World Briefing** section (when enabled). 6. **Colophon** β€” generation timestamp, models used, token cost, source feed counts. TOC: nav depth 2 (sections β†’ articles, discussions nested). Metadata: `dc:title` "The Daily EPUB β€” 2026-08-15", `dc:creator` "The Daily EPUB", `dc:date`, `dc:language en`, EPUB3 `belongs-to-collection` = "The Daily EPUB" with `group-position` = issue number (BookOrbit/KOReader sort correctly). Deterministic chapter ids (`art-{entry_id}`) so rating links and TOC stay stable across regenerations. **Images (`epub/images.rs`):** download (10s timeout, 5 MB cap, semaphore 8), re-encode with `image`: - *Standard edition:* max width 1200px, JPEG q80 (PNG kept for line art/transparency after white-flatten), strip metadata (re-encode does), skip decorative images < 24px, drop SVG/WebP-source images unless decodable, per-issue asset budget ~25 MB. - *X4 edition (`epub/x4.rs`):* grayscale (Luma8), fit within 480Γ—800, JPEG q70, flatten transparency to white; simplified CSS (no floats/flex/grid, no embedded fonts, larger base font, generous line-height, hyphenation on); cover at native 480Γ—800. (These mirror what `epub-to-xtc-converter` recommends, so the XTC conversion step has ideal input.) - Every `` gets `alt` preserved and a `
` if source had one; failed downloads degrade to a "[image: alt text]" placeholder paragraph. **CSS:** one small stylesheet per edition tuned for e-ink: serif body, no colors other than grayscale, `page-break-before` on chapters, blockquote-indent comment styling. ### 3.11 XTC conversion & publishing (`publish.rs`) - Run the `epub-to-xtc-converter` CLI (Node 18+) on the X4 edition: invoke via `tokio::process::Command`, config keys `xtc.command` (default `epub-to-xtc`) and `xtc.args` (verify exact CLI name/flags from the repo README at implementation time; support both `.xtc` 1-bit and `.xtch` 4-level grayscale via config, default XTCH for image quality). Non-zero exit β†’ log error, continue (XTC is a bonus artifact). - **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.bookorbit_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) β€” KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered. - **XTC delivery:** copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead. - **Retention:** delete issue files older than `retention_days` (default 21) from both dirs (BookOrbit's scan removes the DB entries); SQLite issue/rating history is kept forever (it's the training data). ### 3.12 Server (`server.rs`) axum on `127.0.0.1:3499`: - `GET /r/{date}/{article_id}/{vote}?t=` β€” verify HMAC, upsert rating, tiny HTML response. No auth beyond the token (links live inside a private EPUB; tokens are per-article+vote and unguessable). - `GET /opds/xtc.xml`, `GET /files/xtc/{name}` β€” optional Basic auth. - `GET /healthz`, `GET /issues.json` (recent run reports; handy for debugging). - `tower-http` request tracing; graceful shutdown on SIGTERM. ### 3.13 Database schema (sqlite, `migrations/`) ```sql entries(id INTEGER PRIMARY KEY, -- miniflux entry id feed_id INT, feed_title TEXT, category TEXT, title TEXT, url TEXT, canonical_url TEXT, author TEXT, published_at TEXT, comments_url TEXT, raw_content TEXT, fetched_at TEXT); articles(id INTEGER PRIMARY KEY AUTOINCREMENT, -- deduped cluster canonical_url TEXT UNIQUE, title TEXT, best_entry_id INT REFERENCES entries(id), content_html TEXT, word_count INT, excerpt_only BOOL, image_count INT, sources_json TEXT, first_seen TEXT); social(article_id INT, source TEXT CHECK(source IN ('hn','lobsters','reddit','x')), item_id TEXT, score INT, num_comments INT, item_url TEXT, fetched_at TEXT, PRIMARY KEY (article_id, source)); scores(article_id INT, run_date TEXT, prefilter_score REAL, llm_score REAL, llm_category TEXT, rationale TEXT, PRIMARY KEY (article_id, run_date)); issues(date TEXT PRIMARY KEY, issue_number INT, generated_at TEXT, epub_path TEXT, x4_path TEXT, xtc_path TEXT, front_page_html TEXT, report_json TEXT); issue_articles(issue_date TEXT, article_id INT, section TEXT, position INT, is_lead BOOL, summary TEXT, PRIMARY KEY (issue_date, article_id)); ratings(issue_date TEXT, article_id INT, vote INT CHECK(vote IN (-1,1)), rated_at TEXT, PRIMARY KEY (issue_date, article_id)); feed_priors(feed_id INT PRIMARY KEY, upvotes INT, downvotes INT, included INT); runs(id INTEGER PRIMARY KEY AUTOINCREMENT, date TEXT, started_at TEXT, finished_at TEXT, entries_fetched INT, candidates INT, selected INT, input_tokens INT, cached_tokens INT, output_tokens INT, cost_usd REAL, status TEXT, error TEXT); kv(key TEXT PRIMARY KEY, value TEXT); -- watermark, taste_profile, profile_version ``` ### 3.14 Configuration (`config.example.toml`) ```toml timezone = "America/New_York" lookback_hours = 26 target_article_count = 20 prefilter_keep = 120 retention_days = 21 max_daily_usd = 2.0 world_briefing = true [miniflux] base_url = "http://127.0.0.1:8082" # api_key via DAILY_EPUB_MINIFLUX__API_KEY env [deepseek] base_url = "https://api.deepseek.com/v1" model = "deepseek-v4-flash" # verify exact id # api_key via env [curation] always_include_feeds = [] # miniflux feed ids or site urls blocked_domains = [] sections = ["Top Stories", "Tech & Engineering", "Science & Space", "AI & Machine Learning", "Culture & Essays", "Boston & Local", "Niche Corner", "From the Blogroll"] [publish] bookorbit_dir = "/srv/bookorbit/libraries/daily-epub" xtc_dir = "/var/lib/daily-epub/xtc" [xtc] enabled = true command = "epub-to-xtc" # verify CLI name/flags from repo format = "xtch" # xtc | xtch [server] bind = "127.0.0.1:3499" public_url = "https://daily.hallada.net" # hmac_secret via env; optional basic auth user/pass for OPDS ``` ### 3.15 Deployment (systemd, `systemd/`) - `daily-epub.service`: `ExecStart=/usr/local/bin/daily-epub serve`, `Restart=on-failure`, hardening (`DynamicUser` or dedicated user, `StateDirectory=daily-epub`, `ProtectSystem=strict` with write access to publish dirs). - `daily-epub-generate.service` (oneshot) + `daily-epub-generate.timer`: `OnCalendar=*-*-* 05:30:00 America/New_York`, `Persistent=true` (catch up after downtime), `RandomizedDelaySec=300`. - Install: `cargo build --release`, copy binary, `systemctl enable --now`. Reverse-proxy `daily.hallada.net` β†’ `127.0.0.1:3499`. --- ## 4. Implementation milestones (each independently verifiable) 1. **M1 β€” Skeleton & ingest:** crate scaffold, config, migrations, `miniflux.rs`, `generate --dry-run` prints fetched entry stats. *Verify: run against live Miniflux, see ~daily volume.* 2. **M2 β€” Dedupe + extraction + social:** articles table populated with full text, word counts, HN/Reddit/Lobsters scores. *Verify: spot-check known HN stories carry correct points.* 3. **M3 β€” Pre-filter + LLM scoring/selection:** end-to-end lineup JSON printed in dry-run; token/cost report. *Verify: lineup is sane; cost < $0.50.* 4. **M4 β€” EPUB standard edition + publish:** full issue EPUB with cover, front page (temporary plain summaries), sections, articles, images; lands in BookOrbit, visible via OPDS on Kindle. *Verify: epubcheck clean; opens in KOReader with working TOC.* 5. **M5 β€” Editorial + comments:** DeepSeek summaries/intros/front page wired in; discussion chapters. *Verify: read an issue; comments legible on e-ink.* 6. **M6 β€” X4 edition + XTC + XTC OPDS:** second edition, converter invocation, static OPDS feed. *Verify: X4 fetches and renders both.* 7. **M7 β€” Feedback loop:** `serve` rating endpoints, links in chapters, feed priors in pre-filter, weekly profile rebuild. *Verify: tap πŸ‘ in KOReader β†’ row in `ratings` β†’ prior changes next run.* 8. **M8 β€” Hardening & ops:** systemd units, retention, cost guardrail, run reports, `issues.json`, README. ## 5. Verification (end-to-end) - `cargo test` β€” unit tests: URL canonicalization, dedupe clustering, HMAC round-trip, comment-tree truncation, prefilter scoring; integration tests over fixture JSON (recorded Miniflux/Algolia/Reddit responses) with the LLM stage mocked (`--skip-llm` uses prefilter order). - `daily-epub generate --dry-run --out ./out --max-articles 6` with real keys β†’ inspect `./out/*.epub` in Calibre + run `epubcheck` (if installed) β†’ zero errors. - Full live run: `daily-epub generate` β†’ file appears in BookOrbit UI under the Daily EPUB library only β†’ browse BookOrbit OPDS from KOReader (Kindle/Palma), download, read; X4: CrossPoint OPDS β†’ both the X4 EPUB (via BookOrbit) and XTC (via `daily.hallada.net/opds/xtc.xml`). - Tap a rating link on the Kindle β†’ confirmation page loads β†’ `sqlite3 … 'select * from ratings'` shows the vote. - Watch `runs` for a week: cost per day, selection quality; tune `prefilter_keep`/prompts. ## 6. Future ideas (explicitly out of v1 scope; don't constrain the design) Weekly "Sunday Edition" retrospective; LLM editorials/opinion columns on the day's themes; discussion summarization for 500+ comment threads; embedding-based personal ranker (fastembed + linfa) once ratings accumulate; weather/on-this-day front-page ear boxes; a puzzle page; per-section reading-time budgets; Miniflux starred-entry import as implicit positive signal; TTS audio edition; X/Twitter comments if API access ever becomes viable. ## 7. Known limitations & notes - X/Twitter comments are omitted (no free API). - Lobsters linkage only works when the entry originated from a lobste.rs feed (no URL-search API). - Paywalled articles degrade to excerpt + link; they're penalized but not banned (social proof can still surface them). - The X4 can't follow rating links (no browser) β€” accepted; rating happens from KOReader devices. - DeepSeek exact model id and `epub-to-xtc-converter` CLI flags must be confirmed against current docs during implementation (both noted inline). - Pricing basis (Aug 2026): DeepSeek V4 Flash β‰ˆ $0.14/M input (cache-miss), $0.0028/M cached input, $0.28/M output β€” steady-state β‰ˆ $0.05–0.30/day, hard-capped by `max_daily_usd`.