diff --git a/README.md b/README.md index 4cc03b0..1dd8dfb 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,9 @@ the articles, enriches them with HackerNews/Lobsters/Reddit social proof, filter 300–500 candidates down to ~120 with cheap heuristics, and asks DeepSeek to score, select and introduce 15–25 of them. It assembles two EPUB editions (a standard one and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and -drops everything into a [BookOrbit](https://github.com/thallada/bookorbit) watched -folder so KOReader can pick it up over OPDS. Each article chapter ends with πŸ‘/πŸ‘Ž -links that feed back into tomorrow's curation. +publishes the lot over its own OPDS catalog β€” which doubles as a +[BookOrbit](https://github.com/thallada/bookorbit) watched folder if you run one. +Each article chapter ends with πŸ‘/πŸ‘Ž links that feed back into tomorrow's curation. Steady-state cost is roughly **$0.05–0.30/day** in DeepSeek tokens, hard-capped by `max_daily_usd`. @@ -49,7 +49,7 @@ selects, feed excerpts stand in for summaries) instead of losing the day's issue | **Miniflux** with an API key | the only content source | Settings β†’ API Keys. The client is read-only and never mutates read state. | | **DeepSeek API key** | curation + editorial | . Optional: `--skip-llm` runs the whole pipeline without it. | | A 32+ byte random secret | signs the πŸ‘/πŸ‘Ž rating links | `openssl rand -hex 32` | -| **BookOrbit** library + watched folder | delivery to KOReader over OPDS | Create a dedicated "The Daily EPUB" library, enable *Watch folders*, note the folder path. | +| **BookOrbit** library + watched folder | *optional* β€” a richer library UI on top of the same folder | Delivery does not need it: `daily-epub serve` has its own OPDS catalog over `publish.epub_dir`. If you do run it, create a dedicated "The Daily EPUB" library, enable *Watch folders*, and point `publish.epub_dir` at it. | | **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF β€” see below. It has **no global npm bin** β€” it is invoked as `node /cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. | | A reverse proxy for `daily.hallada.net` β†’ `127.0.0.1:3499` | rating links must be reachable from e-readers on the internet | TLS via your existing setup. | @@ -71,7 +71,7 @@ sudo install -m0755 target/release/daily-epub /usr/local/bin/ ``` daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] -daily-epub serve # rating endpoints + XTC OPDS + static files +daily-epub serve # rating endpoints + OPDS catalog + downloads daily-epub profile rebuild # regenerate the taste profile from ratings (weekly inside generate) daily-epub backfill-social # re-poll social scores for recent articles daily-epub db migrate # run migrations (also automatic on every start) @@ -108,7 +108,8 @@ Secrets belong in the environment file, never in the TOML. | `lookback_hours` | `26` | Size of the ingest window ending at the issue day's end (clamped to now). | | `target_article_count` | `20` | Lineup size the selector aims for. `--max-articles` overrides it. | | `prefilter_keep` | `120` | Candidates surviving the heuristic pre-filter. Must be β‰₯ `target_article_count`. | -| `retention_days` | `21` | Published files older than this are deleted from both publish dirs. SQLite history is kept forever. | +| `retention_days` | `21` | EPUBs older than this are deleted from `publish.epub_dir`. SQLite history is kept forever. | +| `xtc_retention_count` | `5` | How many XTC issues to keep in `publish.xtc_dir`. Counted, not dated: each `.xtch` is ~80–100 MB, so the binding constraint is disk, not age. | | `max_daily_usd` | `2.0` | Hard ceiling on DeepSeek spend **per day**, not per run β€” a re-run inherits what earlier runs for that date already spent. Tripping it skips remaining LLM work and degrades to excerpts. | | `world_briefing` | `true` | Include the Wikipedia Current Events section. | | `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. | @@ -130,8 +131,8 @@ Secrets belong in the environment file, never in the TOML. | `curation.blocked_domains` | `[]` | Hosts excluded outright. | | `curation.paywall_domains` | `[]` | Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, economist, …). | | `curation.sections` | 8 sections | The **only** section names the model may use. `World Briefing` is reserved and never offered. | -| `publish.bookorbit_dir` | `/srv/bookorbit/libraries/daily-epub` | BookOrbit watched folder. Both EPUB editions land here by atomic copy, distinguished by a `(X4)` tag in **both** the filename and `dc:title` β€” libraries and OPDS clients list books by title, so the filename alone would make them look identical. | -| `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts + the generated `xtc.xml` OPDS feed. | +| `publish.epub_dir` | `/srv/bookorbit/libraries/daily-epub` | Both EPUB editions land here by atomic copy, and this is the directory the OPDS feed lists. The editions are distinguished by a `(X4)` tag in **both** the filename and `dc:title` β€” libraries and OPDS clients list books by title, so the filename alone would make them look identical. Point a BookOrbit watched folder at it if you want its UI too. **Renamed from `bookorbit_dir`**; the old key is a hard config error. | +| `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts. **Not** listed in the OPDS feed β€” CrossPoint cannot acquire them β€” but downloadable at `/files/xtc/` for sideloading. | | `xtc.enabled` | `true` | Set `false` to skip the converter entirely. | | `xtc.command` | `node` | Converter executable. | | `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends ` -o -f ` (plus `-c `). | @@ -140,7 +141,7 @@ Secrets belong in the environment file, never in the TOML. | `server.bind` | `127.0.0.1:3499` | Listen address. | | `server.public_url` | `https://daily.hallada.net` | Base URL the rating links inside the EPUB are built from. | | `server.hmac_secret` | β€” | **`DAILY_EPUB_SERVER__HMAC_SECRET`** (or `DAILY_EPUB_SECRET`). Without it, generated links are rejected with 403. | -| `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`. | +| `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/*` and `/files/*`. | --- @@ -185,7 +186,7 @@ sudo systemctl enable --now daily-epub.service daily-epub-generate.timer > ``` > ReadWritePaths=/home/thallada/bookorbit/books/daily-epub /var/lib/daily-epub/xtc > ``` -> If you change `publish.bookorbit_dir` or `publish.xtc_dir` in the config, change +> If you change `publish.epub_dir` or `publish.xtc_dir` in the config, change > these lines too and `systemctl daemon-reload`, or publishing fails with > `Read-only file system`. @@ -196,7 +197,7 @@ Node for the XTC converter, whose JIT needs W+X pages. The rating links baked into every article chapter point at `server.public_url`, so `daily.hallada.net` must resolve and serve TLS from the -internet (e-readers tap these links). The XTC OPDS feed rides on the same host. +internet (e-readers tap these links). The OPDS catalog rides on the same host. With an existing certificate, a minimal nginx site is: ```nginx @@ -266,23 +267,43 @@ sudo -u daily-epub node /opt/epub-to-xtc-converter/cli/index.js convert \ > Running as `daily-epub` both avoids the trap and proves the *service* can read > everything it needs. -### How the XTC edition reaches the X4 +### The OPDS catalog -The `.xtch` file in `publish.xtc_dir` is **not** meant for BookOrbit β€” BookOrbit -only watches the EPUB folder and wouldn't import the XTC binary format anyway. -`daily-epub serve` publishes its own OPDS 1.2 feed for it: point the X4's -CrossPoint OPDS browser at `https://daily.hallada.net/opds/xtc.xml` (the bare -`https://daily.hallada.net/opds` works too) and it will list the last 14 issues, -newest first, with the files served from `/files/xtc/`. (The X4 can also fall -back to the "(X4)" EPUB via BookOrbit's own OPDS catalog.) +`daily-epub serve` publishes its own OPDS 1.2 acquisition feed at +**`https://daily.hallada.net/opds/daily.xml`** (the bare +`https://daily.hallada.net/opds`, with or without a trailing slash, serves the +same feed β€” one less thing to type on a seven-button keyboard). Point any OPDS +client β€” the X4's CrossPoint browser, KOReader, Calibre β€” at it and you get the +last 14 issues, newest first, **both editions of each**, with the standard +edition listed first and the files served from `/files/epub/`. -`xtc.xml` is not an issue β€” it is a generated index, rewritten from scratch at -the end of every run by scanning `publish.xtc_dir` for `.xtc`/`.xtch` files. The -issues themselves are the dated files beside it; `/files/xtc/` is a route, not a -directory on disk. So the feed always carries the back catalogue, capped at the -last 14 days by `XTC_FEED_ENTRIES` and bounded by `retention_days` on disk. An -empty `` with no `` elements means the converter never produced a -file β€” check the run's warnings, not the server. +The feed is not a file. It is rendered per request by scanning +`publish.epub_dir` for `The Daily EPUB - YYYY-MM-DD*.epub`, so it cannot go +stale behind a failed publish and there is no generated index for the retention +sweep to step around. Entries are titled exactly like each EPUB's `dc:title`, so +the two editions of one issue are distinguishable in the list. + +This is deliberately independent of BookOrbit: it needs only the directory, so +BookOrbit is optional, and it puts the day's issue one screen from the X4's home +instead of several clicks down a library tree. + +### Why XTC is not in the feed + +XTC files are still generated and still land in `publish.xtc_dir` β€” they are just +not advertised over OPDS, because **CrossPoint's OPDS browser can only acquire +EPUBs**. Two independent reasons, both in the firmware: + +- its OPDS parser marks an entry as a book only when an acquisition link's `type` + is exactly `application/epub+zip` (a `strcmp`), and drops the entry otherwise β€” + which surfaces as *"No entries found"*; +- `OpdsBookBrowserActivity` hardcodes `.epub` as the saved filename regardless of + the URL or `Content-Disposition`, and the reader dispatches on extension, so a + downloaded `.xtch` would land under a name it then refuses to open. + +XTC remains a first-class format *on the device* β€” the file browser lists +`.xtc`/`.xtch` and there is a dedicated XTC reader β€” so the artifacts stay +downloadable at `/files/xtc/` (same Basic auth) for sideloading by SD card, +WebDAV or the device's web upload UI. **Budget the disk.** XTCH is a pre-rendered 2-bit page bitmap β€” 480Γ—800 px is ~96 KB per page regardless of what is on it β€” so an issue is large and its size @@ -294,8 +315,9 @@ tracks the page count, which `font.size` drives: | 30 (`xtc-settings.example.json`) | 820 | 79 MB | ~39 MB | Measured on the 20-article issue of 2026-08-15; conversion took ~13 s either way. -At `retention_days = 21` that is 1.7–2.2 GB in `publish.xtc_dir`, and it is also -what the X4 downloads over WiFi per issue. +That is why `xtc_dir` is swept by **count** rather than age: at the default +`xtc_retention_count = 5` it holds ~0.4–0.5 GB, while the EPUBs beside it age out +on the much longer `retention_days`. --- @@ -329,9 +351,9 @@ ls -la /srv/bookorbit/libraries/daily-epub /var/lib/daily-epub/xtc # 6. Delivery # KOReader (Kindle/Palma): browse BookOrbit's OPDS, download, read. -# Xteink X4 / CrossPoint: OPDS β†’ https://daily.hallada.net/opds/xtc.xml +# Xteink X4 / CrossPoint: OPDS β†’ https://daily.hallada.net/opds/daily.xml curl -s https://daily.hallada.net/healthz -curl -s https://daily.hallada.net/opds/xtc.xml | head +curl -s https://daily.hallada.net/opds/daily.xml | head curl -s https://daily.hallada.net/issues.json | jq '.[0]' # 7. Feedback loop: tap πŸ‘ in KOReader, then @@ -356,7 +378,8 @@ from what you see in step 8. | Run status `degraded` | a best-effort stage failed; the warnings are in the report (`/issues.json`, `runs.error`, the journal). | | No XTC file | `xtc.enabled = false`, Node missing, wrong `xtc.args` path, or `xtc.settings` unset/pointing at a font that does not exist. Non-fatal β€” the X4 can read the X4 EPUB from BookOrbit instead. The report warning quotes the converter's own error. | | `Font path is required` for a settings file that *does* set `font.path` | The process cannot read the file, and the converter cannot tell that apart from the file not existing. Almost always running the converter as yourself instead of `daily-epub` (see above), or a font path that has moved. `sudo -u daily-epub cat /etc/daily-epub/xtc-settings.json` and `sudo -u daily-epub test -r && echo ok` settle it. | -| The X4's OPDS browser says "Failed to fetch feed" | Usually an *empty* feed: `curl -s https://daily.hallada.net/opds/xtc.xml` and count the `` elements. Zero means no `.xtch` has ever been published β€” fix the converter, not the server. | +| The X4's OPDS browser says "No entries found" | It fetched and parsed the feed but accepted no entry. Every acquisition link must be typed exactly `application/epub+zip`; anything else is dropped silently. `curl -s -u user:pass https://daily.hallada.net/opds/daily.xml \| grep -c ""` β€” zero means nothing has been published yet. | +| The X4's OPDS browser says "Failed to fetch feed" | The request never completed: wrong URL, TLS, or credentials. "Failed to parse feed" means malformed XML. The three messages are distinct β€” read which one you got. | | No World Briefing | The portal page for the issue's own date is an empty stub until midday UTC, so the run falls back up to `world::MAX_LOOKBACK_DAYS` days. A warning means even those were empty or Wikipedia was unreachable. | --- diff --git a/config.example.toml b/config.example.toml index eadd34c..3dc7a07 100644 --- a/config.example.toml +++ b/config.example.toml @@ -11,7 +11,8 @@ timezone = "America/New_York" lookback_hours = 26 target_article_count = 20 prefilter_keep = 120 -retention_days = 21 +retention_days = 21 # EPUBs, by age +xtc_retention_count = 5 # XTC issues, by count (~80-100 MB each) max_daily_usd = 2.0 world_briefing = true @@ -59,7 +60,12 @@ sections = [ ] [publish] -bookorbit_dir = "/srv/bookorbit/libraries/daily-epub" +# Where both EPUB editions land, and what the OPDS feed lists. BookOrbit is +# optional β€” it just watches this folder if you run it. +# (Renamed from `bookorbit_dir`; the old key is now a hard config error.) +epub_dir = "/srv/bookorbit/libraries/daily-epub" +# XTC artifacts. Not listed in OPDS (CrossPoint cannot acquire them); reachable +# at /files/xtc/ for sideloading. xtc_dir = "/var/lib/daily-epub/xtc" [xtc] diff --git a/docs/plans/2026-08-15-implementation-notes.md b/docs/plans/2026-08-15-implementation-notes.md index efc3c72..6ae4e08 100644 --- a/docs/plans/2026-08-15-implementation-notes.md +++ b/docs/plans/2026-08-15-implementation-notes.md @@ -31,6 +31,25 @@ This file records implementation-time decisions and verified external facts. Fol - **Output size.** XTCH is a pre-rendered page bitmap: 480Γ—800 at 2bpp = ~96 KB/page. A 20-article issue rendered at `font.size = 34` came to 1,088 pages β‰ˆ **104 MB**, in ~13 s. `retention_days = 21` therefore implies ~2 GB in `publish.xtc_dir`. +- **CrossPoint's OPDS browser can only acquire EPUBs.** Verified against the + `yokki-vans/InkPointX` sources (an open fork of CrossPoint/CrossInk). Two independent + blockers, either one fatal for serving XTC over OPDS: + 1. `lib/OpdsParser/OpdsParser.cpp` sets an entry's `href` only for an acquisition link + whose `type` is **exactly** `application/epub+zip` (`strcmp`), or for a navigation + link (`application/atom+xml`). `endElement` then drops any entry with an empty + `href`. An `application/octet-stream` acquisition link therefore yields an empty + list and the UI reports `STR_NO_ENTRIES` β€” **"No entries found"**. + 2. `OpdsBookBrowserActivity::downloadBook` builds the destination filename as + `sanitizeFilename(author + " - " + title) + ".epub"` β€” hardcoded, ignoring the URL + and `Content-Disposition` β€” and `ReaderActivity` dispatches on extension only + (`FsHelpers::hasXtcExtension` β†’ `.xtc`/`.xtch`, no magic-byte sniffing). So even a + mistyped XTC link downloads into a file the device will not open. + The three OPDS failure strings are distinct and worth reading precisely: + `STR_FETCH_FEED_FAILED` ("Failed to fetch feed"), `STR_PARSE_FEED_FAILED` + ("Failed to parse feed") and `STR_NO_ENTRIES` ("No entries found") β€” only the last is + reachable after a *successful* fetch **and** parse. + Consequence: the built-in feed serves EPUBs (both editions), XTC is published but not + advertised, and `publish.xtc_dir` is swept by count. See the amendment in spec Β§3.11. - **X4 firmware rendering limits** (from the `epub-to-xtc-converter` optimizer's header, which cites papyrix-reader): 464Γ—788 usable viewport, max image decode 2048Γ—3072, **baseline JPEG only**, no GIF/SVG/WebP, **max 1500 CSS rules and simple selectors only** (`tag`, `.class`, diff --git a/docs/plans/2026-08-15-the-daily-epub.md b/docs/plans/2026-08-15-the-daily-epub.md index ee4c7ff..1aed13d 100644 --- a/docs/plans/2026-08-15-the-daily-epub.md +++ b/docs/plans/2026-08-15-the-daily-epub.md @@ -10,7 +10,7 @@ 3. Applies cheap heuristic pre-filters, then uses **DeepSeek V4 Flash** to score, select, and organize ~15–25 articles into newspaper sections. 4. Generates editorial framing: a front-page "day in brief," section intros, and per-article summaries. 5. Builds two EPUB editions (standard + Xteink X4-optimized), converts the X4 edition to XTC/XTCH. -6. Publishes into a dedicated **BookOrbit** library via watched folder (β†’ OPDS for KOReader devices) and serves XTC via a minimal built-in OPDS feed. +6. Publishes both EPUB editions into a folder that its own built-in OPDS feed serves (and that **BookOrbit** can optionally watch for a richer library UI). *(Amended: the built-in feed serves EPUBs, not XTC β€” see Β§3.11.)* 7. Collects πŸ‘/πŸ‘Ž feedback via rating links inside the EPUB to continuously improve curation. **Reader profile (bake into curation prompts):** prefers long-form, high-effort, well-written articles on *any* topic; uses social proof (HN/Reddit upvotes+comments) as a quality proxy; wants tech news, light general/US world news (prefers Wikipedia Current Events for world news), Boston-area news, and ultra-niche community news. The full interest list lives in `data/scour-interests.opml` (~220 Scour interests: Rust, systems programming, e-ink, self-hosting, PKM, sci-fi, creative coding, space, running, board games, Boston Tech, etc.) β€” compile it into the taste profile at build time. @@ -21,7 +21,7 @@ |---|---|---|---| | Miniflux | `127.0.0.1:8082` | `miniflux.hallada.net` | Installed via PPA. API auth via `X-Auth-Token` header. | | BookOrbit | `127.0.0.1:3498` | `bookorbit.hallada.net` | NestJS/Vue/Postgres. Supports multiple isolated libraries, per-library watched folders, OPDS at `/api/v1/opds` (Basic auth, `opds_access` permission). | -| The Daily EPUB (new) | `127.0.0.1:` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + XTC OPDS + static files. | +| The Daily EPUB (new) | `127.0.0.1:` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + OPDS catalog + downloads. | ### Secrets/config the operator must provide @@ -46,7 +46,7 @@ Single Rust binary crate `daily-epub` (workspace not needed yet) with clap subco ``` daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] -daily-epub serve # long-running: rating endpoints + XTC OPDS + static +daily-epub serve # long-running: rating endpoints + OPDS catalog + downloads daily-epub profile rebuild # regenerate taste profile from ratings (also runs weekly inside generate) daily-epub backfill-social # re-poll social scores for recent entries (optional helper) daily-epub db migrate # run sqlx migrations (also auto-run on start) @@ -60,7 +60,7 @@ daily-epub db migrate # run sqlx migrations (also auto-run on start) Miniflux ingest β†’ normalize/dedupe β†’ content extraction β†’ social enrichment β†’ heuristic pre-filter (500 β†’ ~120) β†’ LLM scoring (batched) β†’ LLM selection (~120 β†’ 15–25) β†’ comment fetching for selected β†’ LLM editorial (summaries, section intros, front page) - β†’ EPUB build (standard + X4 editions) β†’ XTC conversion β†’ publish (BookOrbit folder, XTC dir, OPDS xml) + β†’ EPUB build (standard + X4 editions) β†’ XTC conversion β†’ publish (EPUB dir, XTC dir) β†’ OPDS feed served from the EPUB dir β†’ retention pruning β†’ run report logged + stored ``` @@ -103,8 +103,8 @@ the-daily-epub/ β”‚ β”‚ β”œβ”€β”€ templates/ # askama XHTML templates + CSS β”‚ β”‚ β”œβ”€β”€ images.rs # download, resize, grayscale, re-encode β”‚ β”‚ └── x4.rs # X4 edition transforms + XTC CLI invocation -β”‚ β”œβ”€β”€ publish.rs # copy to BookOrbit folder, OPDS xml gen, retention -β”‚ β”œβ”€β”€ server.rs # axum: /r/… ratings, /opds/xtc.xml, /files/… +β”‚ β”œβ”€β”€ publish.rs # copy to the publish dirs, OPDS feed rendering, retention +β”‚ β”œβ”€β”€ server.rs # axum: /r/… ratings, /opds/daily.xml, /files/… β”‚ └── report.rs # run summary (counts, cost, timings) └── tests/ # integration tests with fixture JSON ``` @@ -248,15 +248,40 @@ TOC: nav depth 2 (sections β†’ articles, discussions nested). Metadata: `dc:titl ### 3.11 XTC conversion & publishing (`publish.rs`) - Run the `epub-to-xtc-converter` CLI (Node 18+) on the X4 edition: invoke via `tokio::process::Command`, config keys `xtc.command` (default `epub-to-xtc`) and `xtc.args` (verify exact CLI name/flags from the repo README at implementation time; support both `.xtc` 1-bit and `.xtch` 4-level grayscale via config, default XTCH for image quality). Non-zero exit β†’ log error, continue (XTC is a bonus artifact). -- **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.bookorbit_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) β€” KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered. -- **XTC delivery:** copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead. +- **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.epub_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) β€” KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered. +- **XTC delivery** *(superseded β€” see the amendment below; kept for the record)*: copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead. - **Retention:** delete issue files older than `retention_days` (default 21) from both dirs (BookOrbit's scan removes the DB entries); SQLite issue/rating history is kept forever (it's the training data). +> **Amended 2026-08-15 (post-M8), after testing against a real X4.** The XTC +> delivery bullet above does not work and has been replaced. CrossPoint's OPDS +> browser cannot acquire XTC at all β€” see the implementation notes' "Verified +> external facts" for the two firmware reasons β€” so: +> +> - The built-in feed lists **EPUBs, both editions**, from `publish.epub_dir`, +> with every acquisition link typed exactly `application/epub+zip` and files +> served from `/files/epub/{name}`. Canonical path `/opds/daily.xml`, with +> `/opds` and `/opds/` as aliases. +> - The feed is **rendered per request** from the directory rather than written to +> disk, so it cannot go stale behind a failed publish and the retention sweep has +> no generated index to step around. +> - This makes BookOrbit **optional**: the feed needs only the folder, and it puts +> the day's issue one screen from the X4's home rather than several clicks down a +> library tree. +> - XTC is still generated and still published to `publish.xtc_dir`, just not +> advertised. It stays fetchable at `/files/xtc/{name}` for sideloading. +> - `publish.xtc_dir` is swept by **count** (`xtc_retention_count`, default 5) +> rather than by age: an XTCH issue measured 79–104 MB, so disk is the binding +> constraint. EPUBs keep the dated `retention_days` sweep. +> - `publish.bookorbit_dir` was renamed **`publish.epub_dir`** to match: the feed +> needs the directory, not BookOrbit. The old key is rejected outright +> (`deny_unknown_fields`) rather than silently falling back to the default, +> which would publish into a directory the feed does not read. + ### 3.12 Server (`server.rs`) axum on `127.0.0.1:3499`: - `GET /r/{date}/{article_id}/{vote}?t=` β€” verify HMAC, upsert rating, tiny HTML response. No auth beyond the token (links live inside a private EPUB; tokens are per-article+vote and unguessable). -- `GET /opds/xtc.xml`, `GET /files/xtc/{name}` β€” optional Basic auth. +- `GET /opds/daily.xml` (aliases `/opds`, `/opds/`), `GET /files/epub/{name}`, `GET /files/xtc/{name}` β€” optional Basic auth. *(Amended: was `/opds/xtc.xml` + `/files/xtc/` only; see Β§3.11.)* - `GET /healthz`, `GET /issues.json` (recent run reports; handy for debugging). - `tower-http` request tracing; graceful shutdown on SIGTERM. @@ -317,7 +342,7 @@ sections = ["Top Stories", "Tech & Engineering", "Science & Space", "Niche Corner", "From the Blogroll"] [publish] -bookorbit_dir = "/srv/bookorbit/libraries/daily-epub" +epub_dir = "/srv/bookorbit/libraries/daily-epub" xtc_dir = "/var/lib/daily-epub/xtc" [xtc] @@ -346,7 +371,7 @@ public_url = "https://daily.hallada.net" 3. **M3 β€” Pre-filter + LLM scoring/selection:** end-to-end lineup JSON printed in dry-run; token/cost report. *Verify: lineup is sane; cost < $0.50.* 4. **M4 β€” EPUB standard edition + publish:** full issue EPUB with cover, front page (temporary plain summaries), sections, articles, images; lands in BookOrbit, visible via OPDS on Kindle. *Verify: epubcheck clean; opens in KOReader with working TOC.* 5. **M5 β€” Editorial + comments:** DeepSeek summaries/intros/front page wired in; discussion chapters. *Verify: read an issue; comments legible on e-ink.* -6. **M6 β€” X4 edition + XTC + XTC OPDS:** second edition, converter invocation, static OPDS feed. *Verify: X4 fetches and renders both.* +6. **M6 β€” X4 edition + XTC + OPDS:** second edition, converter invocation, OPDS feed. *Verify: X4 fetches and renders both.* 7. **M7 β€” Feedback loop:** `serve` rating endpoints, links in chapters, feed priors in pre-filter, weekly profile rebuild. *Verify: tap πŸ‘ in KOReader β†’ row in `ratings` β†’ prior changes next run.* 8. **M8 β€” Hardening & ops:** systemd units, retention, cost guardrail, run reports, `issues.json`, README. @@ -354,7 +379,7 @@ public_url = "https://daily.hallada.net" - `cargo test` β€” unit tests: URL canonicalization, dedupe clustering, HMAC round-trip, comment-tree truncation, prefilter scoring; integration tests over fixture JSON (recorded Miniflux/Algolia/Reddit responses) with the LLM stage mocked (`--skip-llm` uses prefilter order). - `daily-epub generate --dry-run --out ./out --max-articles 6` with real keys β†’ inspect `./out/*.epub` in Calibre + run `epubcheck` (if installed) β†’ zero errors. -- Full live run: `daily-epub generate` β†’ file appears in BookOrbit UI under the Daily EPUB library only β†’ browse BookOrbit OPDS from KOReader (Kindle/Palma), download, read; X4: CrossPoint OPDS β†’ both the X4 EPUB (via BookOrbit) and XTC (via `daily.hallada.net/opds/xtc.xml`). +- Full live run: `daily-epub generate` β†’ both editions appear in the publish dir (and in BookOrbit's UI if it is running) β†’ browse `daily.hallada.net/opds/daily.xml` from KOReader (Kindle/Palma) or the X4's CrossPoint, download, read. - Tap a rating link on the Kindle β†’ confirmation page loads β†’ `sqlite3 … 'select * from ratings'` shows the vote. - Watch `runs` for a week: cost per day, selection quality; tune `prefilter_keep`/prompts. diff --git a/src/config.rs b/src/config.rs index a750275..4a45614 100644 --- a/src/config.rs +++ b/src/config.rs @@ -51,8 +51,13 @@ pub struct Config { pub target_article_count: usize, /// How many articles survive the heuristic pre-filter (Β§3.5). pub prefilter_keep: usize, - /// Days of published files kept in the publish dirs (Β§3.11). + /// Days of published EPUBs kept in `publish.epub_dir` (Β§3.11). pub retention_days: u32, + /// How many XTC issues to keep in `publish.xtc_dir` (Β§3.11). + /// + /// Counted, not dated, because an XTCH issue is ~80–100 MB of pre-rendered + /// page bitmaps: the constraint is disk, not age. + pub xtc_retention_count: u32, /// Hard cost ceiling per run (Β§3.6 guardrail). pub max_daily_usd: f64, /// Include the Wikipedia Current Events section (Β§3.8). @@ -81,6 +86,7 @@ impl Default for Config { target_article_count: 20, prefilter_keep: 120, retention_days: 21, + xtc_retention_count: 5, max_daily_usd: 2.0, world_briefing: true, database_path: PathBuf::from("/var/lib/daily-epub/daily-epub.db"), @@ -195,16 +201,20 @@ impl Default for CurationConfig { #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields, default)] pub struct PublishConfig { - /// BookOrbit "The Daily EPUB" library watched folder. - pub bookorbit_dir: PathBuf, - /// Directory served at `/files/xtc/`. + /// Where both EPUB editions land: the source of the OPDS feed, served at + /// `/files/epub/`, and a BookOrbit watched folder if one is configured. + /// + /// Renamed from `bookorbit_dir` once the built-in feed started serving this + /// directory directly β€” BookOrbit is optional, the directory is not. + pub epub_dir: PathBuf, + /// Directory served at `/files/xtc/`. Not listed in the OPDS feed (Β§3.11). pub xtc_dir: PathBuf, } impl Default for PublishConfig { fn default() -> Self { Self { - bookorbit_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"), + epub_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"), xtc_dir: PathBuf::from("/var/lib/daily-epub/xtc"), } } @@ -274,7 +284,7 @@ pub struct ServerConfig { pub public_url: String, /// HMAC key for rating tokens; supply via `DAILY_EPUB_SERVER__HMAC_SECRET`. pub hmac_secret: Option, - /// Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`. + /// Optional Basic auth for `/opds/*` and `/files/*`. pub basic_auth_user: Option, pub basic_auth_pass: Option, } @@ -431,6 +441,26 @@ mod tests { )); } + /// `publish.bookorbit_dir` was renamed to `publish.epub_dir`. A config still + /// using the old key must fail loudly and name both β€” silently falling back + /// to the default would publish the issue into the wrong directory, where + /// the OPDS feed would then find nothing. + #[test] + fn the_renamed_publish_key_fails_loudly() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("config.toml"); + std::fs::write( + &path, + "[publish]\nbookorbit_dir = \"/srv/books\"\nxtc_dir = \"/srv/xtc\"\n", + ) + .unwrap(); + + let err = Config::load(Some(&path)).expect_err("the stale key must be rejected"); + let message = err.to_string(); + assert!(message.contains("bookorbit_dir"), "{message}"); + assert!(message.contains("epub_dir"), "{message}"); + } + #[test] fn shipped_example_config_parses() { let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml"); diff --git a/src/main.rs b/src/main.rs index 2d8d39b..b806347 100644 --- a/src/main.rs +++ b/src/main.rs @@ -31,7 +31,7 @@ struct Cli { enum Command { /// Build (and publish) one issue. Generate(GenerateArgs), - /// Run the rating endpoints, XTC OPDS feed and static files. + /// Run the rating endpoints, the OPDS catalog and downloads. Serve, /// Taste-profile maintenance. #[command(subcommand)] @@ -167,9 +167,6 @@ fn print_outcome(outcome: &GenerateOutcome) { if let Some(xtc) = &published.xtc { println!("published: {}", xtc.display()); } - if let Some(opds) = &published.opds { - println!("opds: {}", opds.display()); - } if published.pruned > 0 { println!("pruned: {} expired files", published.pruned); } diff --git a/src/pipeline.rs b/src/pipeline.rs index 9c53fd9..664a008 100644 --- a/src/pipeline.rs +++ b/src/pipeline.rs @@ -510,7 +510,7 @@ async fn run_stages( ); None } else { - let published = publish::publish_issue(db, config, &issue, &artifacts, xtc.as_deref()) + let published = publish::publish_issue(config, &issue, &artifacts, xtc.as_deref()) .await .context("publishing the issue")?; record_issue(db, &issue, &published) diff --git a/src/publish.rs b/src/publish.rs index e9d3052..0e26d5b 100644 --- a/src/publish.rs +++ b/src/publish.rs @@ -1,6 +1,9 @@ -//! Publishing: BookOrbit watched folder, XTC delivery, OPDS feed, retention +//! Publishing: the EPUB library folder, XTC delivery, OPDS feed, retention //! (spec Β§3.11). //! +//! `publish.epub_dir` holds both EPUB editions and is what the OPDS feed lists; +//! BookOrbit may watch the same folder but nothing here depends on it. +//! //! Everything here is deliberately dumb about *how* artifacts were produced: the //! EPUB/XTC stages hand over finished files, this module only copies, indexes and //! prunes them. Copies are atomic (temp file in the destination directory, then @@ -8,11 +11,15 @@ //! half-written book. //! //! [`crate::pipeline`] ends a non-dry run with one call β€” -//! `publish_issue(db, config, &issue, &artifacts, xtc_path.as_deref())`, where +//! `publish_issue(config, &issue, &artifacts, xtc_path.as_deref())`, where //! `artifacts` are the `epub::build_all` outputs and `xtc_path` is //! `epub::x4::convert`'s output (`None` when the converter is disabled or //! failed) β€” and feeds the returned [`Published`] paths into //! `db.upsert_issue(..., epub_path, x4_path, xtc_path, ...)`. +//! +//! The OPDS feed ([`build_opds`]) is rendered per request by +//! [`crate::server`] rather than written here, and lists **EPUBs only** β€” see +//! its docs for why XTC cannot be delivered over OPDS. use std::collections::BTreeMap; use std::path::{Path, PathBuf}; @@ -26,14 +33,19 @@ use crate::config::Config; use crate::db::Db; use crate::types::{Artifact, Edition, Issue}; -/// Filename of the generated static OPDS feed (Β§3.11). -pub const XTC_OPDS_FILENAME: &str = "xtc.xml"; -/// Number of issues listed in the XTC OPDS feed (Β§3.11). -pub const XTC_FEED_ENTRIES: usize = 14; +/// Number of issue *days* listed in the OPDS feed; each contributes one entry +/// per edition (Β§3.11). +pub const OPDS_FEED_ISSUES: usize = 14; /// Every published file starts with this (the retention sweep keys off it). pub const FILE_PREFIX: &str = "The Daily EPUB - "; /// Extensions the retention sweep is allowed to delete (Β§3.11). pub const PRUNABLE_EXTENSIONS: [&str; 3] = ["epub", "xtc", "xtch"]; +/// Extensions of the XTC artifacts, for the counted sweep of `xtc_dir` (Β§3.11). +pub const XTC_EXTENSIONS: [&str; 2] = ["xtc", "xtch"]; +/// Canonical path of the OPDS feed, relative to `server.public_url` (Β§3.11). +pub const OPDS_PATH: &str = "/opds/daily.xml"; +/// The one acquisition type CrossPoint's OPDS parser accepts (Β§3.11). +pub const EPUB_CONTENT_TYPE: &str = "application/epub+zip"; #[derive(Debug, thiserror::Error)] pub enum PublishError { @@ -61,8 +73,6 @@ pub struct Published { pub epubs: Vec, /// The XTC artifact's published location, when the converter produced one. pub xtc: Option, - /// The regenerated OPDS feed. - pub opds: Option, /// How many expired files the retention sweep removed. pub pruned: usize, } @@ -136,7 +146,7 @@ async fn ensure_dir(dir: &Path) -> Result<(), PublishError> { .map_err(PublishError::at(dir)) } -/// Copy both EPUB editions into the BookOrbit watched folder (Β§3.11). +/// Copy both EPUB editions into `publish.epub_dir` (Β§3.11). /// /// Returns the published paths in the same order as `artifacts`. pub async fn publish_epubs( @@ -144,7 +154,7 @@ pub async fn publish_epubs( issue: &Issue, cfg: &Config, ) -> Result, PublishError> { - let dir = &cfg.publish.bookorbit_dir; + let dir = &cfg.publish.epub_dir; ensure_dir(dir).await?; let mut published = Vec::with_capacity(artifacts.len()); for artifact in artifacts { @@ -159,7 +169,7 @@ pub async fn publish_epubs( edition = ?artifact.edition, dest = %dest.display(), bytes = artifact.bytes, - "published edition to the BookOrbit library" + "published edition to the EPUB library" ); published.push(dest); } @@ -180,12 +190,11 @@ pub async fn publish_xtc(xtc: &Path, cfg: &Config) -> Result Some(publish_xtc(src, cfg).await?), None => None, }; - let opds = Some(write_xtc_opds(db, cfg).await?); + // The OPDS feed is rendered per request from the publish directory, so + // there is nothing to write here (Β§3.11). let pruned = prune(cfg, issue.meta.date).await?; - Ok(Published { - epubs, - xtc, - opds, - pruned, - }) + Ok(Published { epubs, xtc, pruned }) } // --------------------------------------------------------------------------- // OPDS 1.2 acquisition feed (Β§3.11) // --------------------------------------------------------------------------- -/// One published XTC file, as listed in the feed. +/// One published EPUB, as listed in the feed. #[derive(Debug, Clone, PartialEq, Eq)] -struct XtcFile { +struct EpubFile { name: String, date: Option, + edition: Edition, modified: Timestamp, bytes: u64, } -/// Regenerate the static OPDS 1.2 acquisition feed for the XTC directory: -/// newest first, last [`XTC_FEED_ENTRIES`], entries typed -/// `application/octet-stream` (Β§3.11). -pub async fn write_xtc_opds(db: &Db, cfg: &Config) -> Result { - let dir = &cfg.publish.xtc_dir; +/// Render the OPDS 1.2 acquisition feed for the EPUB publish directory (Β§3.11). +/// +/// Rendered per request rather than written to disk: the directory is the only +/// source of truth, so the feed cannot go stale behind a failed publish, and +/// there is no generated file for the retention sweep to step around. +/// +/// XTC artifacts are deliberately **not** listed. CrossPoint's OPDS browser only +/// acquires links typed `application/epub+zip` and always saves the result with +/// a `.epub` extension, which its reader dispatches on β€” so an XTC offered here +/// would either be invisible or download into a file that cannot be opened. +pub async fn build_opds(db: &Db, cfg: &Config) -> Result { + let dir = &cfg.publish.epub_dir; ensure_dir(dir).await?; - let files = scan_xtc_dir(dir).await?; + let files = scan_epub_dir(dir).await?; let numbers = issue_numbers(db, &files).await; - let feed = render_opds(&files, &numbers, &cfg.server.public_url, Timestamp::now()); - - let dest = dir.join(XTC_OPDS_FILENAME); - write_atomic(&dest, feed.as_bytes()).await?; - tracing::info!(entries = files.len(), dest = %dest.display(), "wrote the XTC OPDS feed"); - Ok(dest) + Ok(render_opds( + &files, + &numbers, + &cfg.server.public_url, + Timestamp::now(), + )) } -/// XTC artifacts in `dir`, newest first, capped at [`XTC_FEED_ENTRIES`]. -async fn scan_xtc_dir(dir: &Path) -> Result, PublishError> { +/// Published EPUBs in `dir`, newest first, limited to the last +/// [`OPDS_FEED_ISSUES`] issue days (both editions of each). +async fn scan_epub_dir(dir: &Path) -> Result, PublishError> { let mut entries = tokio::fs::read_dir(dir) .await .map_err(PublishError::at(dir))?; let mut files = Vec::new(); while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? { let name = entry.file_name().to_string_lossy().into_owned(); - let extension = Path::new(&name) + let path = Path::new(&name); + let extension = path .extension() .and_then(|e| e.to_str()) .unwrap_or_default() .to_ascii_lowercase(); - if !matches!(extension.as_str(), "xtc" | "xtch") { + // Only our own issues: a shared library may hold other people's books. + if extension != "epub" || date_from_filename(&name).is_none() { continue; } let meta = match entry.metadata().await { Ok(meta) if meta.is_file() => meta, Ok(_) => continue, Err(e) => { - tracing::warn!(error = %e, name, "skipping unreadable XTC file"); + tracing::warn!(error = %e, name, "skipping unreadable EPUB"); continue; } }; @@ -277,29 +293,47 @@ async fn scan_xtc_dir(dir: &Path) -> Result, PublishError> { .ok() .and_then(|m| Timestamp::try_from(m).ok()) .unwrap_or_else(Timestamp::now); - files.push(XtcFile { + let stem = path + .file_stem() + .and_then(|s| s.to_str()) + .unwrap_or_default() + .to_string(); + files.push(EpubFile { date: date_from_filename(&name), + edition: Edition::from_file_stem(&stem), name, modified, bytes: meta.len(), }); } - // Newest first: by issue date when the filename carries one, else by mtime. + // Newest issue first, and within an issue the standard edition leads. files.sort_by(|a, b| { b.date .cmp(&a.date) + .then(a.edition.cmp(&b.edition)) .then(b.modified.cmp(&a.modified)) .then(a.name.cmp(&b.name)) }); - files.truncate(XTC_FEED_ENTRIES); + // Cap by issue day, not by file: an issue is two entries and truncating + // mid-issue would list one edition without the other. + let mut kept_dates: Vec> = Vec::new(); + files.retain(|file| { + if !kept_dates.contains(&file.date) { + kept_dates.push(file.date); + } + kept_dates.iter().position(|d| *d == file.date) < Some(OPDS_FEED_ISSUES) + }); Ok(files) } /// Issue numbers for the dated files, best-effort (the feed is still valid /// without them). Uses the `db` escape hatch β€” no bespoke helper in `db.rs`. -async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap { +async fn issue_numbers(db: &Db, files: &[EpubFile]) -> BTreeMap { let mut numbers = BTreeMap::new(); for date in files.iter().filter_map(|f| f.date) { + if numbers.contains_key(&date) { + continue; + } let row = sqlx::query("SELECT issue_number FROM issues WHERE date = ?") .bind(date.to_string()) .fetch_optional(db.pool()) @@ -317,13 +351,13 @@ async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap { /// Render the Atom/OPDS document (Β§3.11). fn render_opds( - files: &[XtcFile], + files: &[EpubFile], numbers: &BTreeMap, public_url: &str, now: Timestamp, ) -> String { let base = public_url.trim_end_matches('/'); - let self_href = format!("{base}/opds/xtc.xml"); + let self_href = format!("{base}{OPDS_PATH}"); let updated = files.first().map(|f| f.modified).unwrap_or(now); let mut out = String::with_capacity(1024 + files.len() * 512); @@ -333,8 +367,8 @@ fn render_opds( xmlns:dc=\"http://purl.org/dc/terms/\" \ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n", ); - out.push_str(" urn:daily-epub:xtc\n"); - out.push_str(" The Daily EPUB β€” XTC editions\n"); + out.push_str(" urn:daily-epub:issues\n"); + out.push_str(" The Daily EPUB\n"); out.push_str(&format!(" {}\n", rfc3339(updated))); out.push_str(" The Daily EPUB\n"); out.push_str(&format!( @@ -349,19 +383,25 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n", )); for file in files { + // Matches the EPUB's own `dc:title`, so the two editions of one issue + // are told apart in the list rather than showing as the same book. let title = match file.date { - Some(date) => format!("The Daily EPUB β€” {date}"), + Some(date) => format!("The Daily EPUB β€” {date}{}", file.edition.file_suffix()), None => file.name.clone(), }; - let summary = match file.date.and_then(|d| numbers.get(&d)) { - Some(n) => format!("Issue #{n} Β· {}", human_bytes(file.bytes)), - None => human_bytes(file.bytes), + let edition_note = match file.edition { + Edition::Standard => "Standard", + Edition::X4 => "Xteink X4", }; - let href = format!("{base}/files/xtc/{}", percent_encode(&file.name)); + let summary = match file.date.and_then(|d| numbers.get(&d)) { + Some(n) => format!("Issue #{n} Β· {edition_note} Β· {}", human_bytes(file.bytes)), + None => format!("{edition_note} Β· {}", human_bytes(file.bytes)), + }; + let href = format!("{base}/files/epub/{}", percent_encode(&file.name)); out.push_str(" \n"); out.push_str(&format!(" {}\n", xml_escape(&title))); out.push_str(&format!( - " urn:daily-epub:xtc:{}\n", + " urn:daily-epub:issue:{}\n", xml_escape(&percent_encode(&file.name)) )); out.push_str(&format!( @@ -376,9 +416,12 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n", " {}\n", xml_escape(&summary) )); + // The type must be exactly `application/epub+zip`: CrossPoint's OPDS + // parser compares it with `strcmp` and silently drops entries whose + // acquisition link is anything else, reporting "No entries found". out.push_str(&format!( " \n", +type=\"{EPUB_CONTENT_TYPE}\" length=\"{}\"/>\n", xml_escape(&href), file.bytes )); @@ -430,43 +473,82 @@ fn percent_encode(s: &str) -> String { out } -async fn write_atomic(dest: &Path, bytes: &[u8]) -> Result<(), PublishError> { - let dir = dest.parent().unwrap_or_else(|| Path::new(".")); - let tmp = dir.join(format!( - ".{}.{}.tmp", - dest.file_name() - .and_then(|n| n.to_str()) - .unwrap_or("daily-epub"), - std::process::id() - )); - tokio::fs::write(&tmp, bytes) - .await - .map_err(PublishError::at(&tmp))?; - tokio::fs::rename(&tmp, dest) - .await - .map_err(PublishError::at(dest)) -} - // --------------------------------------------------------------------------- // Retention (Β§3.11) // --------------------------------------------------------------------------- -/// Delete issue files older than `retention_days` from both publish dirs. -/// SQLite history is kept forever β€” it's the training data (Β§3.11). +/// Retention sweep over both publish dirs. SQLite history is kept forever β€” +/// it's the training data (Β§3.11). +/// +/// The two directories are swept on different rules: EPUBs age out after +/// `retention_days`, while XTC is capped at `xtc_retention_count` issues because +/// each one is ~80–100 MB of pre-rendered page bitmaps and the binding +/// constraint is disk rather than age. /// /// Only files named `The Daily EPUB - YYYY-MM-DD*.{epub,xtc,xtch}` are ever -/// considered; anything else in those directories (including `xtc.xml` and other -/// people's books) is left strictly alone. +/// considered; anything else in those directories (other people's books) is left +/// strictly alone. pub async fn prune(cfg: &Config, today: Date) -> Result { let cutoff = today .checked_sub(jiff::Span::new().days(i64::from(cfg.retention_days))) .unwrap_or(today); - let mut removed = 0; - for dir in [&cfg.publish.bookorbit_dir, &cfg.publish.xtc_dir] { - removed += prune_dir(dir, cutoff).await?; - } + let mut removed = prune_dir(&cfg.publish.epub_dir, cutoff).await?; if removed > 0 { - tracing::info!(removed, %cutoff, "retention sweep removed expired issues"); + tracing::info!(removed, %cutoff, "retention sweep removed expired EPUBs"); + } + let xtc_removed = prune_xtc_dir(&cfg.publish.xtc_dir, cfg.xtc_retention_count as usize).await?; + if xtc_removed > 0 { + tracing::info!( + removed = xtc_removed, + keep = cfg.xtc_retention_count, + "retention sweep trimmed the XTC directory" + ); + } + removed += xtc_removed; + Ok(removed) +} + +/// Keep only the newest `keep` XTC issues, deleting the rest (Β§3.11). +/// +/// Counted rather than dated so the directory has a hard size ceiling no matter +/// how often `generate` runs. +async fn prune_xtc_dir(dir: &Path, keep: usize) -> Result { + if !dir.exists() { + return Ok(0); + } + let mut entries = tokio::fs::read_dir(dir) + .await + .map_err(PublishError::at(dir))?; + let mut ours: Vec<(Date, PathBuf)> = Vec::new(); + while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? { + let name = entry.file_name().to_string_lossy().into_owned(); + let extension = Path::new(&name) + .extension() + .and_then(|e| e.to_str()) + .unwrap_or_default() + .to_ascii_lowercase(); + if !XTC_EXTENSIONS.contains(&extension.as_str()) { + continue; + } + let Some(date) = date_from_filename(&name) else { + continue; + }; + if !entry.metadata().await.map(|m| m.is_file()).unwrap_or(false) { + continue; + } + ours.push((date, entry.path())); + } + // Newest first, then drop everything past the cap. + ours.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(&b.1))); + let mut removed = 0; + for (date, path) in ours.into_iter().skip(keep) { + match tokio::fs::remove_file(&path).await { + Ok(()) => { + tracing::info!(path = %path.display(), %date, "pruned an XTC issue past the cap"); + removed += 1; + } + Err(e) => tracing::warn!(error = %e, path = %path.display(), "could not prune file"), + } } Ok(removed) } @@ -520,7 +602,7 @@ mod tests { fn cfg(dir: &Path) -> Config { let mut cfg = Config::default(); - cfg.publish.bookorbit_dir = dir.join("bookorbit"); + cfg.publish.epub_dir = dir.join("bookorbit"); cfg.publish.xtc_dir = dir.join("xtc"); cfg.server.public_url = "https://daily.hallada.net".into(); cfg @@ -561,7 +643,7 @@ mod tests { Some(date("2026-08-15")) ); for foreign in [ - "xtc.xml", + "The Daily EPUB - 2026-08-15.xml", "Moby Dick.epub", "The Daily EPUB - notadate.epub", "The Daily EPUB - 2026-08-15.txt", @@ -634,10 +716,10 @@ mod tests { paths, vec![ cfg.publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-08-15.epub"), cfg.publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-08-15 (X4).epub"), ] ); @@ -673,21 +755,37 @@ mod tests { } } + fn epub_file(name: &str, edition: Edition, modified: &str, bytes: u64) -> EpubFile { + EpubFile { + date: date_from_filename(name), + edition, + name: name.into(), + modified: ts(modified), + bytes, + } + } + #[test] fn opds_feed_is_newest_first_with_acquisition_links() { let files = vec![ - XtcFile { - name: "The Daily EPUB - 2026-08-15 (X4).xtch".into(), - date: Some(date("2026-08-15")), - modified: ts("2026-08-15T05:40:00Z"), - bytes: 2_500_000, - }, - XtcFile { - name: "The Daily EPUB - 2026-08-14 (X4).xtch".into(), - date: Some(date("2026-08-14")), - modified: ts("2026-08-14T05:40:00Z"), - bytes: 4096, - }, + epub_file( + "The Daily EPUB - 2026-08-15.epub", + Edition::Standard, + "2026-08-15T05:40:00Z", + 6_500_000, + ), + epub_file( + "The Daily EPUB - 2026-08-15 (X4).epub", + Edition::X4, + "2026-08-15T05:40:00Z", + 1_700_000, + ), + epub_file( + "The Daily EPUB - 2026-08-14.epub", + Edition::Standard, + "2026-08-14T05:40:00Z", + 4096, + ), ]; let mut numbers = BTreeMap::new(); numbers.insert(date("2026-08-15"), 12); @@ -700,37 +798,68 @@ mod tests { assert!(feed.starts_with("")); assert!(feed.contains("urn:daily-epub:xtc")); + assert!(feed.contains("urn:daily-epub:issues")); assert!(feed.contains("2026-08-15T05:40:00Z")); - assert_eq!(feed.matches("").count(), 2); - assert_eq!(feed.matches("").count(), 2); - // Newest first. - let i15 = feed.find("The Daily EPUB β€” 2026-08-15").unwrap(); - let i14 = feed.find("The Daily EPUB β€” 2026-08-14").unwrap(); - assert!(i15 < i14); - // Acquisition link, encoded filename, absolute public URL, size. + assert_eq!(feed.matches("").count(), 3); + assert_eq!(feed.matches("").count(), 3); + + // Newest issue first; the two editions of one issue are distinguishable + // by title, matching each EPUB's own `dc:title`. + let i15 = feed + .find("The Daily EPUB β€” 2026-08-15") + .unwrap(); + let i15_x4 = feed + .find("The Daily EPUB β€” 2026-08-15 (X4)") + .unwrap(); + let i14 = feed + .find("The Daily EPUB β€” 2026-08-14") + .unwrap(); + assert!(i15 < i15_x4 && i15_x4 < i14); + + // CrossPoint compares the acquisition type with `strcmp` against + // `application/epub+zip` and drops the entry on any mismatch (Β§3.11). + assert_eq!( + feed.matches("type=\"application/epub+zip\"").count(), + 3, + "{feed}" + ); + assert!(!feed.contains("application/octet-stream")); assert!(feed.contains( "" +href=\"https://daily.hallada.net/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub\" \ +type=\"application/epub+zip\" length=\"1700000\"/>" )); - assert!(feed.contains("Issue #12 Β· 2.4 MB")); + assert!(feed.contains("Issue #12 Β· Standard Β· 6.2 MB")); + assert!(feed.contains("Issue #12 Β· Xteink X4 Β· 1.6 MB")); + assert!( + feed.contains("")); assert!(!feed.contains("&<"), "unescaped markup leaked in"); } #[tokio::test] - async fn write_xtc_opds_lists_only_xtc_files_capped_at_fourteen() { + async fn the_feed_lists_both_editions_of_the_last_fourteen_issues() { let dir = tempfile::tempdir().unwrap(); let cfg = cfg(dir.path()); + std::fs::create_dir_all(&cfg.publish.epub_dir).unwrap(); std::fs::create_dir_all(&cfg.publish.xtc_dir).unwrap(); for day in 1..=20 { - let name = format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch"); - std::fs::write(cfg.publish.xtc_dir.join(name), b"x").unwrap(); + for edition in Edition::ALL { + let name = issue_filename(date(&format!("2026-08-{day:02}")), edition, "epub"); + std::fs::write(cfg.publish.epub_dir.join(name), b"x").unwrap(); + } } - // Non-XTC neighbours must be ignored. - std::fs::write(cfg.publish.xtc_dir.join("README.txt"), b"x").unwrap(); - std::fs::write(cfg.publish.xtc_dir.join("cover.epub"), b"x").unwrap(); + // Other people's books and our own XTC artifacts must be ignored. + std::fs::write(cfg.publish.epub_dir.join("Moby Dick.epub"), b"x").unwrap(); + std::fs::write(cfg.publish.epub_dir.join("metadata.db"), b"x").unwrap(); + std::fs::write( + cfg.publish + .xtc_dir + .join("The Daily EPUB - 2026-08-20 (X4).xtch"), + b"x", + ) + .unwrap(); let db = Db::open_and_migrate(&dir.path().join("db.sqlite")) .await @@ -748,64 +877,75 @@ type=\"application/octet-stream\" length=\"2500000\"/>" .await .unwrap(); - let path = write_xtc_opds(&db, &cfg).await.unwrap(); - assert_eq!(path, cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)); - let feed = std::fs::read_to_string(&path).unwrap(); - assert_eq!(feed.matches("").count(), XTC_FEED_ENTRIES); + let feed = build_opds(&db, &cfg).await.unwrap(); + // Capped by issue day, so both editions of each of the last 14 survive. + assert_eq!(feed.matches("").count(), OPDS_FEED_ISSUES * 2); assert!(feed.contains("2026-08-20")); assert!(!feed.contains("2026-08-06"), "older than the last 14"); - assert!(!feed.contains("README")); - assert!(!feed.contains("cover.epub")); + assert!(!feed.contains("Moby Dick")); + assert!(!feed.contains("metadata.db")); assert!(feed.contains("Issue #20")); - - // Regenerating replaces the file in place. - write_xtc_opds(&db, &cfg).await.unwrap(); - let leftovers: Vec = std::fs::read_dir(&cfg.publish.xtc_dir) - .unwrap() - .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) - .filter(|n| n.ends_with(".tmp")) - .collect(); - assert!(leftovers.is_empty(), "{leftovers:?}"); + // XTC is never offered: CrossPoint cannot acquire it (Β§3.11). + assert!(!feed.contains(".xtch"), "{feed}"); + assert!(!feed.contains("/files/xtc/")); } #[tokio::test] - async fn prune_only_deletes_old_matching_files() { + async fn prune_ages_out_epubs_but_counts_xtc() { let dir = tempfile::tempdir().unwrap(); let mut conf = cfg(dir.path()); conf.retention_days = 21; - std::fs::create_dir_all(&conf.publish.bookorbit_dir).unwrap(); + conf.xtc_retention_count = 5; + std::fs::create_dir_all(&conf.publish.epub_dir).unwrap(); std::fs::create_dir_all(&conf.publish.xtc_dir).unwrap(); let keep_epub = conf .publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-08-14.epub"); let old_epub = conf .publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-07-01.epub"); let old_x4 = conf .publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-07-01 (X4).epub"); - let foreign = conf.publish.bookorbit_dir.join("Moby Dick.epub"); - let old_xtc = conf - .publish - .xtc_dir - .join("The Daily EPUB - 2026-07-01 (X4).xtch"); - let feed = conf.publish.xtc_dir.join(XTC_OPDS_FILENAME); - for path in [&keep_epub, &old_epub, &old_x4, &foreign, &old_xtc, &feed] { + let foreign = conf.publish.epub_dir.join("Moby Dick.epub"); + for path in [&keep_epub, &old_epub, &old_x4, &foreign] { std::fs::write(path, b"x").unwrap(); } + // Eight consecutive XTC issues, all recent: age would keep every one, + // the count cap keeps the newest five. + let xtc: Vec = (8..=15) + .map(|day| { + let path = conf + .publish + .xtc_dir + .join(format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch")); + std::fs::write(&path, b"x").unwrap(); + path + }) + .collect(); + let foreign_xtc = conf.publish.xtc_dir.join("Someone Else.xtch"); + std::fs::write(&foreign_xtc, b"x").unwrap(); + + // 2 expired EPUBs + 3 XTC issues past the cap of 5. let removed = prune(&conf, date("2026-08-15")).await.unwrap(); - assert_eq!(removed, 3); + assert_eq!(removed, 5); assert!(keep_epub.exists()); assert!(foreign.exists(), "never touch other people's books"); - assert!(feed.exists(), "the OPDS feed is not an issue file"); + assert!(foreign_xtc.exists(), "nor their XTC files"); assert!(!old_epub.exists()); assert!(!old_x4.exists()); - assert!(!old_xtc.exists()); + // The five newest XTC issues (11th–15th) survive; 8th–10th are gone. + for path in &xtc[..3] { + assert!(!path.exists(), "{} should be pruned", path.display()); + } + for path in &xtc[3..] { + assert!(path.exists(), "{} should be kept", path.display()); + } // Idempotent, and tolerant of missing directories. assert_eq!(prune(&conf, date("2026-08-15")).await.unwrap(), 0); @@ -844,25 +984,31 @@ type=\"application/octet-stream\" length=\"2500000\"/>" ]; let issue = fake_issue(date("2026-08-15")); - let published = publish_issue(&db, &cfg, &issue, &artifacts, Some(&xtc_src)) + let published = publish_issue(&cfg, &issue, &artifacts, Some(&xtc_src)) .await .unwrap(); assert_eq!(published.epubs.len(), 2); assert!(published.epubs.iter().all(|a| a.path.exists())); assert_eq!(published.epubs[1].edition, Edition::X4); assert!(published.xtc.as_ref().is_some_and(|p| p.exists())); - assert!(published.opds.as_ref().is_some_and(|p| p.exists())); assert_eq!(published.pruned, 0); - let feed = std::fs::read_to_string(cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)).unwrap(); - assert!(feed.contains("out.xtch")); + // The feed is derived from the publish directory, so both editions show + // up without publish having written anything. + let feed = build_opds(&db, &cfg).await.unwrap(); + assert!( + feed.contains("The Daily EPUB β€” 2026-08-15"), + "{feed}" + ); + assert!( + feed.contains("The Daily EPUB β€” 2026-08-15 (X4)"), + "{feed}" + ); + assert!(!feed.contains("out.xtch")); - // No XTC artifact is fine β€” the feed is still regenerated. - let published = publish_issue(&db, &cfg, &issue, &artifacts, None) - .await - .unwrap(); + // No XTC artifact is fine β€” the EPUBs are what the feed lists anyway. + let published = publish_issue(&cfg, &issue, &artifacts, None).await.unwrap(); assert!(published.xtc.is_none()); - assert!(published.opds.is_some()); } #[test] diff --git a/src/server.rs b/src/server.rs index 335b140..a2707f0 100644 --- a/src/server.rs +++ b/src/server.rs @@ -1,4 +1,4 @@ -//! axum server: rating endpoints, XTC OPDS, static files (spec Β§3.9, Β§3.12). +//! axum server: rating endpoints, the OPDS catalog, downloads (spec Β§3.9, Β§3.12). //! //! Rating links must work from an e-reader's built-in browser, so every rating //! endpoint is a `GET` and the response is a tiny e-ink-sized HTML page. @@ -7,8 +7,9 @@ //! | route | behaviour | //! |---|---| //! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC, upsert rating, rebuild feed priors | -//! | `GET /opds/xtc.xml` | static OPDS 1.2 acquisition feed from `publish.xtc_dir` | -//! | `GET /files/xtc/{name}` | XTC artifact download (no path traversal) | +//! | `GET /opds/daily.xml` (also `/opds`, `/opds/`) | OPDS 1.2 acquisition feed over `publish.epub_dir` | +//! | `GET /files/epub/{name}` | EPUB download β€” what the feed's acquisition links point at | +//! | `GET /files/xtc/{name}` | XTC artifact download, unlisted (no path traversal) | //! | `GET /healthz` | liveness | //! | `GET /issues.json` | the last 30 run reports, newest first | //! @@ -81,16 +82,18 @@ pub use crate::auth::{constant_time_eq, rating_token, rating_url, verify_token}; // Router (Β§3.12) // --------------------------------------------------------------------------- -/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/xtc.xml`, -/// `/files/xtc/{name}`, `/healthz`, `/issues.json`, with `tower-http` tracing (Β§3.12). +/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/daily.xml`, +/// `/files/epub/{name}`, `/files/xtc/{name}`, `/healthz`, `/issues.json`, with +/// `tower-http` tracing (Β§3.12). pub fn router(state: AppState) -> Router { Router::new() .route("/r/{date}/{article_id}/{vote}", get(handle_rating)) - .route("/opds/xtc.xml", get(handle_opds)) + .route(crate::publish::OPDS_PATH, get(handle_opds)) // OPDS browsers are typed into by hand on a 6" e-ink keyboard: serve the // same feed from the catalog root so a URL without the filename works. .route("/opds", get(handle_opds)) .route("/opds/", get(handle_opds)) + .route("/files/epub/{name}", get(handle_epub_file)) .route("/files/xtc/{name}", get(handle_xtc_file)) .route("/healthz", get(handle_healthz)) .route("/issues.json", get(handle_issues_json)) @@ -284,44 +287,64 @@ async fn handle_rating( ) } -/// `GET /opds/xtc.xml` β€” the static feed written by [`crate::publish`] (Β§3.11). +/// `GET /opds/daily.xml` β€” both EPUB editions of the last issues, newest first, +/// rendered from the publish directory on each request (Β§3.11). async fn handle_opds(State(state): State, headers: HeaderMap) -> Response { if let Some(challenge) = check_basic_auth(&state.config, &headers) { return challenge; } - let path = state - .config - .publish - .xtc_dir - .join(crate::publish::XTC_OPDS_FILENAME); - match tokio::fs::read(&path).await { - Ok(bytes) => ( + match crate::publish::build_opds(&state.db, &state.config).await { + Ok(feed) => ( StatusCode::OK, [ (header::CONTENT_TYPE, OPDS_CONTENT_TYPE), (header::CACHE_CONTROL, "no-cache"), ], - bytes, + feed, ) .into_response(), Err(e) => { - tracing::warn!(error = %e, path = %path.display(), "no XTC OPDS feed yet"); - (StatusCode::NOT_FOUND, "no feed yet").into_response() + tracing::error!(error = %e, "could not build the OPDS feed"); + ( + StatusCode::INTERNAL_SERVER_ERROR, + "could not build the feed", + ) + .into_response() } } } +/// `GET /files/epub/{name}` β€” download one published EPUB; this is what the +/// OPDS acquisition links point at (Β§3.11). +async fn handle_epub_file( + State(state): State, + Path(name): Path, + headers: HeaderMap, +) -> Response { + let dir = state.config.publish.epub_dir.clone(); + serve_file(&state, &dir, &name, &headers).await +} + /// `GET /files/xtc/{name}` β€” download one XTC artifact (Β§3.11). +/// +/// Not listed in the OPDS feed β€” CrossPoint's browser cannot acquire XTC β€” but +/// kept so the artifacts can still be fetched by URL for sideloading. async fn handle_xtc_file( State(state): State, Path(name): Path, headers: HeaderMap, ) -> Response { - if let Some(challenge) = check_basic_auth(&state.config, &headers) { + let dir = state.config.publish.xtc_dir.clone(); + serve_file(&state, &dir, &name, &headers).await +} + +/// Stream one file out of `dir`, behind the OPDS Basic auth (Β§3.11). +async fn serve_file(state: &AppState, dir: &FsPath, name: &str, headers: &HeaderMap) -> Response { + if let Some(challenge) = check_basic_auth(&state.config, headers) { return challenge; } - let Some(path) = safe_join(&state.config.publish.xtc_dir, &name) else { - tracing::warn!(name, "rejected an unsafe XTC file name"); + let Some(path) = safe_join(dir, name) else { + tracing::warn!(name, "rejected an unsafe file name"); return (StatusCode::BAD_REQUEST, "bad file name").into_response(); }; // An XTCH issue is a pre-rendered page bitmap per page β€” ~100 MB for a full @@ -332,12 +355,14 @@ async fn handle_xtc_file( (file, len) } Err(e) => { - tracing::warn!(error = %e, path = %path.display(), "XTC file not found"); + tracing::warn!(error = %e, path = %path.display(), "file not found"); return (StatusCode::NOT_FOUND, "not found").into_response(); } }; - let content_type = if name.ends_with(".xml") { - OPDS_CONTENT_TYPE + // CrossPoint dispatches on the saved file's extension, not on this header, + // but Calibre and KOReader both use it. + let content_type = if name.ends_with(".epub") { + crate::publish::EPUB_CONTENT_TYPE } else { "application/octet-stream" }; @@ -629,7 +654,9 @@ mod tests { async fn start(with_auth: bool) -> TestServer { let dir = tempfile::tempdir().unwrap(); let xtc_dir = dir.path().join("xtc"); + let epub_dir = dir.path().join("epub"); std::fs::create_dir_all(&xtc_dir).unwrap(); + std::fs::create_dir_all(&epub_dir).unwrap(); let db = Db::open_and_migrate(&dir.path().join("db.sqlite")) .await .unwrap(); @@ -637,6 +664,8 @@ mod tests { let mut config = Config::default(); config.server.hmac_secret = Some(VECTOR_SECRET.into()); config.publish.xtc_dir = xtc_dir; + config.publish.epub_dir = epub_dir; + config.server.public_url = "https://daily.hallada.net".into(); if with_auth { config.server.basic_auth_user = Some("opds".into()); config.server.basic_auth_pass = Some("hunter2".into()); @@ -663,6 +692,10 @@ mod tests { self._dir.path().join("xtc") } + fn epub_dir(&self) -> PathBuf { + self._dir.path().join("epub") + } + async fn seed_article(&self) -> ArticleId { let entry = crate::types::Entry { id: 1, @@ -847,11 +880,10 @@ mod tests { #[tokio::test] async fn opds_and_files_are_served_behind_basic_auth() { let server = TestServer::start(true).await; - std::fs::write( - server.xtc_dir().join(crate::publish::XTC_OPDS_FILENAME), - "", - ) - .unwrap(); + for edition in crate::types::Edition::ALL { + let name = crate::publish::issue_filename(date(), edition, "epub"); + std::fs::write(server.epub_dir().join(name), b"EPUB").unwrap(); + } std::fs::write( server .xtc_dir() @@ -861,7 +893,7 @@ mod tests { .unwrap(); let res = client() - .get(format!("{}/opds/xtc.xml", server.base)) + .get(format!("{}/opds/daily.xml", server.base)) .send() .await .unwrap(); @@ -875,7 +907,7 @@ mod tests { ); let res = client() - .get(format!("{}/opds/xtc.xml", server.base)) + .get(format!("{}/opds/daily.xml", server.base)) .basic_auth("opds", Some("wrong")) .send() .await @@ -883,7 +915,7 @@ mod tests { assert_eq!(res.status(), 401); let res = client() - .get(format!("{}/opds/xtc.xml", server.base)) + .get(format!("{}/opds/daily.xml", server.base)) .basic_auth("opds", Some("hunter2")) .send() .await @@ -895,7 +927,11 @@ mod tests { .unwrap() .starts_with("application/atom+xml") ); - assert_eq!(res.text().await.unwrap(), ""); + let feed = res.text().await.unwrap(); + assert_eq!(feed.matches("").count(), 2, "{feed}"); + assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}"); + // XTC exists on disk but is never advertised (Β§3.11). + assert!(!feed.contains(".xtch"), "{feed}"); // The catalog root serves the same feed, behind the same auth. for alias in ["/opds", "/opds/"] { @@ -912,9 +948,29 @@ mod tests { .await .unwrap(); assert_eq!(res.status(), 200, "{alias}"); - assert_eq!(res.text().await.unwrap(), "", "{alias}"); + assert_eq!(res.text().await.unwrap(), feed, "{alias}"); } + // The acquisition link resolves, typed as an EPUB. + let res = client() + .get(format!( + "{}/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20(X4).epub", + server.base + )) + .basic_auth("opds", Some("hunter2")) + .send() + .await + .unwrap(); + assert_eq!(res.status(), 200); + // CrossPoint needs the size up front to show download progress. + assert_eq!(res.content_length(), Some(4)); + assert_eq!( + res.headers()[header::CONTENT_TYPE].to_str().unwrap(), + "application/epub+zip" + ); + assert_eq!(res.bytes().await.unwrap().as_ref(), b"EPUB"); + + // XTC is still fetchable by URL for sideloading, just not listed. let res = client() .get(format!( "{}/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20(X4).xtch", @@ -925,8 +981,6 @@ mod tests { .await .unwrap(); assert_eq!(res.status(), 200); - // CrossPoint needs the size up front to show download progress. - assert_eq!(res.content_length(), Some(4)); assert_eq!(res.bytes().await.unwrap().as_ref(), b"XTCH"); // Ratings are not behind auth (the token is the credential). diff --git a/src/types.rs b/src/types.rs index 145bbef..dfdcaac 100644 --- a/src/types.rs +++ b/src/types.rs @@ -411,7 +411,10 @@ pub const WORLD_BRIEFING_SECTION: &str = "World Briefing"; // --------------------------------------------------------------------------- /// Which of the two editions is being built (Β§3.10). -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +/// +/// The declaration order is the listing order: standard first, then X4. The +/// OPDS feed sorts on it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum Edition { /// 1200px images, full CSS. @@ -421,6 +424,9 @@ pub enum Edition { } impl Edition { + /// Every edition, in the order they are listed and built (Β§3.10). + pub const ALL: [Edition; 2] = [Edition::Standard, Edition::X4]; + /// Filename suffix: `""` / `" (X4)"` (Β§3.11). pub fn file_suffix(self) -> &'static str { match self { @@ -428,6 +434,18 @@ impl Edition { Edition::X4 => " (X4)", } } + + /// Recover the edition from a published filename's stem (Β§3.11). + /// + /// The OPDS feed is rebuilt by scanning the publish directory, so the + /// filename is the only record of which edition a file is. + pub fn from_file_stem(stem: &str) -> Edition { + if stem.ends_with(Edition::X4.file_suffix()) { + Edition::X4 + } else { + Edition::Standard + } + } } /// Issue-level metadata rendered on the cover, front page and OPF (Β§3.10). diff --git a/tests/e2e_pipeline.rs b/tests/e2e_pipeline.rs index 2a2ed34..de17e67 100644 --- a/tests/e2e_pipeline.rs +++ b/tests/e2e_pipeline.rs @@ -59,7 +59,7 @@ fn test_config(root: &Path) -> Config { prefilter_keep: 20, world_briefing: false, publish: PublishConfig { - bookorbit_dir: root.join("bookorbit"), + epub_dir: root.join("bookorbit"), xtc_dir: root.join("xtc"), }, xtc: XtcConfig { @@ -305,35 +305,44 @@ async fn assemble_build_publish( ); // --- Publish (Β§3.11) --- - let published = publish::publish_issue(db, cfg, &issue, &artifacts, None) + let published = publish::publish_issue(cfg, &issue, &artifacts, None) .await .expect("publish"); assert_eq!(published.epubs.len(), 2); for artifact in &published.epubs { - assert!(artifact.path.starts_with(&cfg.publish.bookorbit_dir)); + assert!(artifact.path.starts_with(&cfg.publish.epub_dir)); assert!(artifact.path.exists(), "{}", artifact.path.display()); } assert!( cfg.publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-08-15.epub") .exists() ); assert!( cfg.publish - .bookorbit_dir + .epub_dir .join("The Daily EPUB - 2026-08-15 (X4).epub") .exists() ); assert!(published.xtc.is_none(), "the converter is disabled here"); - // The OPDS feed is regenerated on every publish, even with no XTC files yet. - let opds = published.opds.clone().expect("an OPDS feed was written"); - assert_eq!(opds, cfg.publish.xtc_dir.join("xtc.xml")); - let feed = std::fs::read_to_string(&opds).expect("read the OPDS feed"); + // The OPDS feed is derived from what was just published β€” both editions, + // typed so CrossPoint will accept them (Β§3.11). + let feed = publish::build_opds(db, cfg).await.expect("build the feed"); assert!(feed.starts_with("").count(), 2, "{feed}"); + assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}"); + assert!( + feed.contains("The Daily EPUB β€” 2026-08-15"), + "{feed}" + ); + assert!( + feed.contains("The Daily EPUB β€” 2026-08-15 (X4)"), + "{feed}" + ); // --- Record (Β§3.13) --- let epub_path = published @@ -438,7 +447,6 @@ async fn skip_llm_pipeline_produces_a_published_issue() { // Re-running the same date replaces rather than duplicates (notes Β§12). let republished = publish::publish_issue( - &db, &cfg, &issue, &[ @@ -458,7 +466,7 @@ async fn skip_llm_pipeline_produces_a_published_issue() { .await .expect("republish"); assert_eq!(republished.epubs.len(), 2); - let files: Vec = std::fs::read_dir(&cfg.publish.bookorbit_dir) + let files: Vec = std::fs::read_dir(&cfg.publish.epub_dir) .expect("read bookorbit dir") .filter_map(|e| e.ok().map(|e| e.file_name().to_string_lossy().into_owned())) .collect(); diff --git a/tests/m7_server.rs b/tests/m7_server.rs index 155b3b0..f62eb25 100644 --- a/tests/m7_server.rs +++ b/tests/m7_server.rs @@ -41,6 +41,7 @@ impl Server { let dir = tempfile::tempdir().expect("tempdir"); let xtc_dir = dir.path().join("xtc"); std::fs::create_dir_all(&xtc_dir).expect("xtc dir"); + std::fs::create_dir_all(dir.path().join("bookorbit")).expect("epub dir"); let log = std::fs::File::create(dir.path().join("server.log")).expect("log file"); let port = free_port(); @@ -59,10 +60,7 @@ impl Server { format!("http://127.0.0.1:{port}"), ) .env("DAILY_EPUB_PUBLISH__XTC_DIR", &xtc_dir) - .env( - "DAILY_EPUB_PUBLISH__BOOKORBIT_DIR", - dir.path().join("bookorbit"), - ) + .env("DAILY_EPUB_PUBLISH__EPUB_DIR", dir.path().join("bookorbit")) .stdout(Stdio::null()) .stderr(Stdio::from(log)); if basic_auth { @@ -80,6 +78,10 @@ impl Server { self.dir.path().join("xtc") } + fn epub_dir(&self) -> std::path::PathBuf { + self.dir.path().join("bookorbit") + } + fn wait_until_ready(&self) { let deadline = Instant::now() + Duration::from_secs(30); while Instant::now() < deadline { @@ -217,22 +219,38 @@ fn binary_serves_health_issues_and_rating_endpoints() { #[test] fn binary_serves_opds_and_files_behind_basic_auth() { let server = Server::start(true); - let name = "The Daily EPUB - 2026-08-15 (X4).xtch"; - write(&server.xtc_dir().join("xtc.xml"), FEED); - write(&server.xtc_dir().join(name), "XTCH"); + write( + &server.epub_dir().join("The Daily EPUB - 2026-08-15.epub"), + "STANDARD", + ); + write( + &server + .epub_dir() + .join("The Daily EPUB - 2026-08-15 (X4).epub"), + "X4EPUB", + ); + write( + &server + .xtc_dir() + .join("The Daily EPUB - 2026-08-15 (X4).xtch"), + "XTCH", + ); write(&server.dir.path().join("secret"), "top secret"); // No credentials β†’ challenge. - let res = server.get("/opds/xtc.xml"); + let res = server.get("/opds/daily.xml"); assert_eq!(res.status, 401); assert_eq!( res.header("www-authenticate"), Some("Basic realm=\"The Daily EPUB\", charset=\"UTF-8\"") ); - assert_eq!(server.get_auth("/opds/xtc.xml", "bm9wZTpub3Bl").status, 401); + assert_eq!( + server.get_auth("/opds/daily.xml", "bm9wZTpub3Bl").status, + 401 + ); - // Correct credentials β†’ the feed, typed as OPDS. - let res = server.get_auth("/opds/xtc.xml", BASIC_AUTH); + // Correct credentials β†’ a feed built from the publish dir, typed as OPDS. + let res = server.get_auth("/opds/daily.xml", BASIC_AUTH); assert_eq!(res.status, 200); assert!( res.header("content-type") @@ -242,8 +260,27 @@ fn binary_serves_opds_and_files_behind_basic_auth() { res.headers ); assert!(res.body.contains("opds-spec.org/acquisition")); + // Both editions, and only as `application/epub+zip` β€” the one acquisition + // type CrossPoint's parser accepts. + assert_eq!(res.body.matches("").count(), 2, "{}", res.body); + assert_eq!( + res.body.matches("type=\"application/epub+zip\"").count(), + 2, + "{}", + res.body + ); + assert!(!res.body.contains(".xtch"), "{}", res.body); // The acquisition link in the feed resolves to the file itself. + let res = server.get_auth( + "/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub", + BASIC_AUTH, + ); + assert_eq!(res.status, 200); + assert_eq!(res.body, "X4EPUB"); + assert_eq!(res.header("content-type"), Some("application/epub+zip")); + + // XTC stays reachable by URL for sideloading, just unlisted. let res = server.get_auth( "/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch", BASIC_AUTH, @@ -272,18 +309,3 @@ fn binary_serves_opds_and_files_behind_basic_auth() { fn write(path: &Path, body: &str) { std::fs::write(path, body).unwrap_or_else(|e| panic!("writing {}: {e}", path.display())); } - -/// A feed shaped like the one `publish::write_xtc_opds` generates. -const FEED: &str = r#" - - urn:daily-epub:xtc - The Daily EPUB β€” XTC editions - 2026-08-15T05:40:00Z - - The Daily EPUB β€” 2026-08-15 - urn:daily-epub:xtc:x - 2026-08-15T05:40:00Z - - - -"#;