Serve EPUBs instead XTC in daily OPDS

It turns out it is currently impossible for the Xteink X4 to download
XTC files from an OPDS server. So I switched the built-in OPDS server in
this binary (daily.hallada.net/opds) to serve the EPUB files instead of
the XTC files. The X4 is pretty capable of reading the X4 edition of the
EPUB that was optimized for it anyways, and I prefer the flexibility of
EPUB, so I might end up eventually deleting the XTC conversion. For now,
I kept the conversion step (in case I ever want to ever try manually
copying them to the device) and I keep a limited number of XTC issues on
the server since they are quite big in filesize. The BookOrbit
integration is now optional since the built-in OPDS server is able to
serve the EPUBs. In my installation, I serve the EPUBs through both. The
daily.hallada.net/opds server is just a little quicker to navigate and
download the EPUBs on the X4 since the BookOrbit OPDS requires diving
into a couple layers of folders before you get to the files.
This commit is contained in:
2026-08-15 20:24:20 +00:00
parent 9e30c1dcdf
commit 9e27a32fb6
12 changed files with 630 additions and 282 deletions
+54 -31
View File
@@ -8,9 +8,9 @@ the articles, enriches them with HackerNews/Lobsters/Reddit social proof, filter
300500 candidates down to ~120 with cheap heuristics, and asks DeepSeek to score, 300500 candidates down to ~120 with cheap heuristics, and asks DeepSeek to score,
select and introduce 1525 of them. It assembles two EPUB editions (a standard one select and introduce 1525 of them. It assembles two EPUB editions (a standard one
and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and
drops everything into a [BookOrbit](https://github.com/thallada/bookorbit) watched publishes the lot over its own OPDS catalog — which doubles as a
folder so KOReader can pick it up over OPDS. Each article chapter ends with 👍/👎 [BookOrbit](https://github.com/thallada/bookorbit) watched folder if you run one.
links that feed back into tomorrow's curation. Each article chapter ends with 👍/👎 links that feed back into tomorrow's curation.
Steady-state cost is roughly **$0.050.30/day** in DeepSeek tokens, hard-capped by Steady-state cost is roughly **$0.050.30/day** in DeepSeek tokens, hard-capped by
`max_daily_usd`. `max_daily_usd`.
@@ -49,7 +49,7 @@ selects, feed excerpts stand in for summaries) instead of losing the day's issue
| **Miniflux** with an API key | the only content source | Settings → API Keys. The client is read-only and never mutates read state. | | **Miniflux** with an API key | the only content source | Settings → API Keys. The client is read-only and never mutates read state. |
| **DeepSeek API key** | curation + editorial | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. | | **DeepSeek API key** | curation + editorial | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. |
| A 32+ byte random secret | signs the 👍/👎 rating links | `openssl rand -hex 32` | | A 32+ byte random secret | signs the 👍/👎 rating links | `openssl rand -hex 32` |
| **BookOrbit** library + watched folder | delivery to KOReader over OPDS | Create a dedicated "The Daily EPUB" library, enable *Watch folders*, note the folder path. | | **BookOrbit** library + watched folder | *optional* — a richer library UI on top of the same folder | Delivery does not need it: `daily-epub serve` has its own OPDS catalog over `publish.epub_dir`. If you do run it, create a dedicated "The Daily EPUB" library, enable *Watch folders*, and point `publish.epub_dir` at it. |
| **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF — see below. It has **no global npm bin** — it is invoked as `node <repo>/cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. | | **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF — see below. It has **no global npm bin** — it is invoked as `node <repo>/cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. |
| A reverse proxy for `daily.hallada.net``127.0.0.1:3499` | rating links must be reachable from e-readers on the internet | TLS via your existing setup. | | A reverse proxy for `daily.hallada.net``127.0.0.1:3499` | rating links must be reachable from e-readers on the internet | TLS via your existing setup. |
@@ -71,7 +71,7 @@ sudo install -m0755 target/release/daily-epub /usr/local/bin/
``` ```
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
daily-epub serve # rating endpoints + XTC OPDS + static files daily-epub serve # rating endpoints + OPDS catalog + downloads
daily-epub profile rebuild # regenerate the taste profile from ratings (weekly inside generate) daily-epub profile rebuild # regenerate the taste profile from ratings (weekly inside generate)
daily-epub backfill-social # re-poll social scores for recent articles daily-epub backfill-social # re-poll social scores for recent articles
daily-epub db migrate # run migrations (also automatic on every start) daily-epub db migrate # run migrations (also automatic on every start)
@@ -108,7 +108,8 @@ Secrets belong in the environment file, never in the TOML.
| `lookback_hours` | `26` | Size of the ingest window ending at the issue day's end (clamped to now). | | `lookback_hours` | `26` | Size of the ingest window ending at the issue day's end (clamped to now). |
| `target_article_count` | `20` | Lineup size the selector aims for. `--max-articles` overrides it. | | `target_article_count` | `20` | Lineup size the selector aims for. `--max-articles` overrides it. |
| `prefilter_keep` | `120` | Candidates surviving the heuristic pre-filter. Must be ≥ `target_article_count`. | | `prefilter_keep` | `120` | Candidates surviving the heuristic pre-filter. Must be ≥ `target_article_count`. |
| `retention_days` | `21` | Published files older than this are deleted from both publish dirs. SQLite history is kept forever. | | `retention_days` | `21` | EPUBs older than this are deleted from `publish.epub_dir`. SQLite history is kept forever. |
| `xtc_retention_count` | `5` | How many XTC issues to keep in `publish.xtc_dir`. Counted, not dated: each `.xtch` is ~80100 MB, so the binding constraint is disk, not age. |
| `max_daily_usd` | `2.0` | Hard ceiling on DeepSeek spend **per day**, not per run — a re-run inherits what earlier runs for that date already spent. Tripping it skips remaining LLM work and degrades to excerpts. | | `max_daily_usd` | `2.0` | Hard ceiling on DeepSeek spend **per day**, not per run — a re-run inherits what earlier runs for that date already spent. Tripping it skips remaining LLM work and degrades to excerpts. |
| `world_briefing` | `true` | Include the Wikipedia Current Events section. | | `world_briefing` | `true` | Include the Wikipedia Current Events section. |
| `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. | | `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. |
@@ -130,8 +131,8 @@ Secrets belong in the environment file, never in the TOML.
| `curation.blocked_domains` | `[]` | Hosts excluded outright. | | `curation.blocked_domains` | `[]` | Hosts excluded outright. |
| `curation.paywall_domains` | `[]` | Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, economist, …). | | `curation.paywall_domains` | `[]` | Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, economist, …). |
| `curation.sections` | 8 sections | The **only** section names the model may use. `World Briefing` is reserved and never offered. | | `curation.sections` | 8 sections | The **only** section names the model may use. `World Briefing` is reserved and never offered. |
| `publish.bookorbit_dir` | `/srv/bookorbit/libraries/daily-epub` | BookOrbit watched folder. Both EPUB editions land here by atomic copy, distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. | | `publish.epub_dir` | `/srv/bookorbit/libraries/daily-epub` | Both EPUB editions land here by atomic copy, and this is the directory the OPDS feed lists. The editions are distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. Point a BookOrbit watched folder at it if you want its UI too. **Renamed from `bookorbit_dir`**; the old key is a hard config error. |
| `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts + the generated `xtc.xml` OPDS feed. | | `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts. **Not** listed in the OPDS feed — CrossPoint cannot acquire them — but downloadable at `/files/xtc/<name>` for sideloading. |
| `xtc.enabled` | `true` | Set `false` to skip the converter entirely. | | `xtc.enabled` | `true` | Set `false` to skip the converter entirely. |
| `xtc.command` | `node` | Converter executable. | | `xtc.command` | `node` | Converter executable. |
| `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends `<input.epub> -o <output> -f <format>` (plus `-c <settings>`). | | `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends `<input.epub> -o <output> -f <format>` (plus `-c <settings>`). |
@@ -140,7 +141,7 @@ Secrets belong in the environment file, never in the TOML.
| `server.bind` | `127.0.0.1:3499` | Listen address. | | `server.bind` | `127.0.0.1:3499` | Listen address. |
| `server.public_url` | `https://daily.hallada.net` | Base URL the rating links inside the EPUB are built from. | | `server.public_url` | `https://daily.hallada.net` | Base URL the rating links inside the EPUB are built from. |
| `server.hmac_secret` | — | **`DAILY_EPUB_SERVER__HMAC_SECRET`** (or `DAILY_EPUB_SECRET`). Without it, generated links are rejected with 403. | | `server.hmac_secret` | — | **`DAILY_EPUB_SERVER__HMAC_SECRET`** (or `DAILY_EPUB_SECRET`). Without it, generated links are rejected with 403. |
| `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`. | | `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/*` and `/files/*`. |
--- ---
@@ -185,7 +186,7 @@ sudo systemctl enable --now daily-epub.service daily-epub-generate.timer
> ``` > ```
> ReadWritePaths=/home/thallada/bookorbit/books/daily-epub /var/lib/daily-epub/xtc > ReadWritePaths=/home/thallada/bookorbit/books/daily-epub /var/lib/daily-epub/xtc
> ``` > ```
> If you change `publish.bookorbit_dir` or `publish.xtc_dir` in the config, change > If you change `publish.epub_dir` or `publish.xtc_dir` in the config, change
> these lines too and `systemctl daemon-reload`, or publishing fails with > these lines too and `systemctl daemon-reload`, or publishing fails with
> `Read-only file system`. > `Read-only file system`.
@@ -196,7 +197,7 @@ Node for the XTC converter, whose JIT needs W+X pages.
The rating links baked into every article chapter point at The rating links baked into every article chapter point at
`server.public_url`, so `daily.hallada.net` must resolve and serve TLS from the `server.public_url`, so `daily.hallada.net` must resolve and serve TLS from the
internet (e-readers tap these links). The XTC OPDS feed rides on the same host. internet (e-readers tap these links). The OPDS catalog rides on the same host.
With an existing certificate, a minimal nginx site is: With an existing certificate, a minimal nginx site is:
```nginx ```nginx
@@ -266,23 +267,43 @@ sudo -u daily-epub node /opt/epub-to-xtc-converter/cli/index.js convert \
> Running as `daily-epub` both avoids the trap and proves the *service* can read > Running as `daily-epub` both avoids the trap and proves the *service* can read
> everything it needs. > everything it needs.
### How the XTC edition reaches the X4 ### The OPDS catalog
The `.xtch` file in `publish.xtc_dir` is **not** meant for BookOrbit — BookOrbit `daily-epub serve` publishes its own OPDS 1.2 acquisition feed at
only watches the EPUB folder and wouldn't import the XTC binary format anyway. **`https://daily.hallada.net/opds/daily.xml`** (the bare
`daily-epub serve` publishes its own OPDS 1.2 feed for it: point the X4's `https://daily.hallada.net/opds`, with or without a trailing slash, serves the
CrossPoint OPDS browser at `https://daily.hallada.net/opds/xtc.xml` (the bare same feed — one less thing to type on a seven-button keyboard). Point any OPDS
`https://daily.hallada.net/opds` works too) and it will list the last 14 issues, client — the X4's CrossPoint browser, KOReader, Calibre — at it and you get the
newest first, with the files served from `/files/xtc/`. (The X4 can also fall last 14 issues, newest first, **both editions of each**, with the standard
back to the "(X4)" EPUB via BookOrbit's own OPDS catalog.) edition listed first and the files served from `/files/epub/`.
`xtc.xml` is not an issue — it is a generated index, rewritten from scratch at The feed is not a file. It is rendered per request by scanning
the end of every run by scanning `publish.xtc_dir` for `.xtc`/`.xtch` files. The `publish.epub_dir` for `The Daily EPUB - YYYY-MM-DD*.epub`, so it cannot go
issues themselves are the dated files beside it; `/files/xtc/` is a route, not a stale behind a failed publish and there is no generated index for the retention
directory on disk. So the feed always carries the back catalogue, capped at the sweep to step around. Entries are titled exactly like each EPUB's `dc:title`, so
last 14 days by `XTC_FEED_ENTRIES` and bounded by `retention_days` on disk. An the two editions of one issue are distinguishable in the list.
empty `<feed>` with no `<entry>` elements means the converter never produced a
file — check the run's warnings, not the server. This is deliberately independent of BookOrbit: it needs only the directory, so
BookOrbit is optional, and it puts the day's issue one screen from the X4's home
instead of several clicks down a library tree.
### Why XTC is not in the feed
XTC files are still generated and still land in `publish.xtc_dir` — they are just
not advertised over OPDS, because **CrossPoint's OPDS browser can only acquire
EPUBs**. Two independent reasons, both in the firmware:
- its OPDS parser marks an entry as a book only when an acquisition link's `type`
is exactly `application/epub+zip` (a `strcmp`), and drops the entry otherwise —
which surfaces as *"No entries found"*;
- `OpdsBookBrowserActivity` hardcodes `.epub` as the saved filename regardless of
the URL or `Content-Disposition`, and the reader dispatches on extension, so a
downloaded `.xtch` would land under a name it then refuses to open.
XTC remains a first-class format *on the device* — the file browser lists
`.xtc`/`.xtch` and there is a dedicated XTC reader — so the artifacts stay
downloadable at `/files/xtc/<name>` (same Basic auth) for sideloading by SD card,
WebDAV or the device's web upload UI.
**Budget the disk.** XTCH is a pre-rendered 2-bit page bitmap — 480×800 px is **Budget the disk.** XTCH is a pre-rendered 2-bit page bitmap — 480×800 px is
~96 KB per page regardless of what is on it — so an issue is large and its size ~96 KB per page regardless of what is on it — so an issue is large and its size
@@ -294,8 +315,9 @@ tracks the page count, which `font.size` drives:
| 30 (`xtc-settings.example.json`) | 820 | 79 MB | ~39 MB | | 30 (`xtc-settings.example.json`) | 820 | 79 MB | ~39 MB |
Measured on the 20-article issue of 2026-08-15; conversion took ~13 s either way. Measured on the 20-article issue of 2026-08-15; conversion took ~13 s either way.
At `retention_days = 21` that is 1.72.2 GB in `publish.xtc_dir`, and it is also That is why `xtc_dir` is swept by **count** rather than age: at the default
what the X4 downloads over WiFi per issue. `xtc_retention_count = 5` it holds ~0.40.5 GB, while the EPUBs beside it age out
on the much longer `retention_days`.
--- ---
@@ -329,9 +351,9 @@ ls -la /srv/bookorbit/libraries/daily-epub /var/lib/daily-epub/xtc
# 6. Delivery # 6. Delivery
# KOReader (Kindle/Palma): browse BookOrbit's OPDS, download, read. # KOReader (Kindle/Palma): browse BookOrbit's OPDS, download, read.
# Xteink X4 / CrossPoint: OPDS → https://daily.hallada.net/opds/xtc.xml # Xteink X4 / CrossPoint: OPDS → https://daily.hallada.net/opds/daily.xml
curl -s https://daily.hallada.net/healthz curl -s https://daily.hallada.net/healthz
curl -s https://daily.hallada.net/opds/xtc.xml | head curl -s https://daily.hallada.net/opds/daily.xml | head
curl -s https://daily.hallada.net/issues.json | jq '.[0]' curl -s https://daily.hallada.net/issues.json | jq '.[0]'
# 7. Feedback loop: tap 👍 in KOReader, then # 7. Feedback loop: tap 👍 in KOReader, then
@@ -356,7 +378,8 @@ from what you see in step 8.
| Run status `degraded` | a best-effort stage failed; the warnings are in the report (`/issues.json`, `runs.error`, the journal). | | Run status `degraded` | a best-effort stage failed; the warnings are in the report (`/issues.json`, `runs.error`, the journal). |
| No XTC file | `xtc.enabled = false`, Node missing, wrong `xtc.args` path, or `xtc.settings` unset/pointing at a font that does not exist. Non-fatal — the X4 can read the X4 EPUB from BookOrbit instead. The report warning quotes the converter's own error. | | No XTC file | `xtc.enabled = false`, Node missing, wrong `xtc.args` path, or `xtc.settings` unset/pointing at a font that does not exist. Non-fatal — the X4 can read the X4 EPUB from BookOrbit instead. The report warning quotes the converter's own error. |
| `Font path is required` for a settings file that *does* set `font.path` | The process cannot read the file, and the converter cannot tell that apart from the file not existing. Almost always running the converter as yourself instead of `daily-epub` (see above), or a font path that has moved. `sudo -u daily-epub cat /etc/daily-epub/xtc-settings.json` and `sudo -u daily-epub test -r <font> && echo ok` settle it. | | `Font path is required` for a settings file that *does* set `font.path` | The process cannot read the file, and the converter cannot tell that apart from the file not existing. Almost always running the converter as yourself instead of `daily-epub` (see above), or a font path that has moved. `sudo -u daily-epub cat /etc/daily-epub/xtc-settings.json` and `sudo -u daily-epub test -r <font> && echo ok` settle it. |
| The X4's OPDS browser says "Failed to fetch feed" | Usually an *empty* feed: `curl -s https://daily.hallada.net/opds/xtc.xml` and count the `<entry>` elements. Zero means no `.xtch` has ever been published — fix the converter, not the server. | | The X4's OPDS browser says "No entries found" | It fetched and parsed the feed but accepted no entry. Every acquisition link must be typed exactly `application/epub+zip`; anything else is dropped silently. `curl -s -u user:pass https://daily.hallada.net/opds/daily.xml \| grep -c "<entry>"` — zero means nothing has been published yet. |
| The X4's OPDS browser says "Failed to fetch feed" | The request never completed: wrong URL, TLS, or credentials. "Failed to parse feed" means malformed XML. The three messages are distinct — read which one you got. |
| No World Briefing | The portal page for the issue's own date is an empty stub until midday UTC, so the run falls back up to `world::MAX_LOOKBACK_DAYS` days. A warning means even those were empty or Wikipedia was unreachable. | | No World Briefing | The portal page for the issue's own date is an empty stub until midday UTC, so the run falls back up to `world::MAX_LOOKBACK_DAYS` days. A warning means even those were empty or Wikipedia was unreachable. |
--- ---
+8 -2
View File
@@ -11,7 +11,8 @@ timezone = "America/New_York"
lookback_hours = 26 lookback_hours = 26
target_article_count = 20 target_article_count = 20
prefilter_keep = 120 prefilter_keep = 120
retention_days = 21 retention_days = 21 # EPUBs, by age
xtc_retention_count = 5 # XTC issues, by count (~80-100 MB each)
max_daily_usd = 2.0 max_daily_usd = 2.0
world_briefing = true world_briefing = true
@@ -59,7 +60,12 @@ sections = [
] ]
[publish] [publish]
bookorbit_dir = "/srv/bookorbit/libraries/daily-epub" # Where both EPUB editions land, and what the OPDS feed lists. BookOrbit is
# optional — it just watches this folder if you run it.
# (Renamed from `bookorbit_dir`; the old key is now a hard config error.)
epub_dir = "/srv/bookorbit/libraries/daily-epub"
# XTC artifacts. Not listed in OPDS (CrossPoint cannot acquire them); reachable
# at /files/xtc/<name> for sideloading.
xtc_dir = "/var/lib/daily-epub/xtc" xtc_dir = "/var/lib/daily-epub/xtc"
[xtc] [xtc]
@@ -31,6 +31,25 @@ This file records implementation-time decisions and verified external facts. Fol
- **Output size.** XTCH is a pre-rendered page bitmap: 480×800 at 2bpp = ~96 KB/page. A - **Output size.** XTCH is a pre-rendered page bitmap: 480×800 at 2bpp = ~96 KB/page. A
20-article issue rendered at `font.size = 34` came to 1,088 pages ≈ **104 MB**, in ~13 s. 20-article issue rendered at `font.size = 34` came to 1,088 pages ≈ **104 MB**, in ~13 s.
`retention_days = 21` therefore implies ~2 GB in `publish.xtc_dir`. `retention_days = 21` therefore implies ~2 GB in `publish.xtc_dir`.
- **CrossPoint's OPDS browser can only acquire EPUBs.** Verified against the
`yokki-vans/InkPointX` sources (an open fork of CrossPoint/CrossInk). Two independent
blockers, either one fatal for serving XTC over OPDS:
1. `lib/OpdsParser/OpdsParser.cpp` sets an entry's `href` only for an acquisition link
whose `type` is **exactly** `application/epub+zip` (`strcmp`), or for a navigation
link (`application/atom+xml`). `endElement` then drops any entry with an empty
`href`. An `application/octet-stream` acquisition link therefore yields an empty
list and the UI reports `STR_NO_ENTRIES`**"No entries found"**.
2. `OpdsBookBrowserActivity::downloadBook` builds the destination filename as
`sanitizeFilename(author + " - " + title) + ".epub"` — hardcoded, ignoring the URL
and `Content-Disposition` — and `ReaderActivity` dispatches on extension only
(`FsHelpers::hasXtcExtension``.xtc`/`.xtch`, no magic-byte sniffing). So even a
mistyped XTC link downloads into a file the device will not open.
The three OPDS failure strings are distinct and worth reading precisely:
`STR_FETCH_FEED_FAILED` ("Failed to fetch feed"), `STR_PARSE_FEED_FAILED`
("Failed to parse feed") and `STR_NO_ENTRIES` ("No entries found") — only the last is
reachable after a *successful* fetch **and** parse.
Consequence: the built-in feed serves EPUBs (both editions), XTC is published but not
advertised, and `publish.xtc_dir` is swept by count. See the amendment in spec §3.11.
- **X4 firmware rendering limits** (from the `epub-to-xtc-converter` optimizer's header, which - **X4 firmware rendering limits** (from the `epub-to-xtc-converter` optimizer's header, which
cites papyrix-reader): 464×788 usable viewport, max image decode 2048×3072, **baseline JPEG cites papyrix-reader): 464×788 usable viewport, max image decode 2048×3072, **baseline JPEG
only**, no GIF/SVG/WebP, **max 1500 CSS rules and simple selectors only** (`tag`, `.class`, only**, no GIF/SVG/WebP, **max 1500 CSS rules and simple selectors only** (`tag`, `.class`,
+37 -12
View File
@@ -10,7 +10,7 @@
3. Applies cheap heuristic pre-filters, then uses **DeepSeek V4 Flash** to score, select, and organize ~1525 articles into newspaper sections. 3. Applies cheap heuristic pre-filters, then uses **DeepSeek V4 Flash** to score, select, and organize ~1525 articles into newspaper sections.
4. Generates editorial framing: a front-page "day in brief," section intros, and per-article summaries. 4. Generates editorial framing: a front-page "day in brief," section intros, and per-article summaries.
5. Builds two EPUB editions (standard + Xteink X4-optimized), converts the X4 edition to XTC/XTCH. 5. Builds two EPUB editions (standard + Xteink X4-optimized), converts the X4 edition to XTC/XTCH.
6. Publishes into a dedicated **BookOrbit** library via watched folder (→ OPDS for KOReader devices) and serves XTC via a minimal built-in OPDS feed. 6. Publishes both EPUB editions into a folder that its own built-in OPDS feed serves (and that **BookOrbit** can optionally watch for a richer library UI). *(Amended: the built-in feed serves EPUBs, not XTC — see §3.11.)*
7. Collects 👍/👎 feedback via rating links inside the EPUB to continuously improve curation. 7. Collects 👍/👎 feedback via rating links inside the EPUB to continuously improve curation.
**Reader profile (bake into curation prompts):** prefers long-form, high-effort, well-written articles on *any* topic; uses social proof (HN/Reddit upvotes+comments) as a quality proxy; wants tech news, light general/US world news (prefers Wikipedia Current Events for world news), Boston-area news, and ultra-niche community news. The full interest list lives in `data/scour-interests.opml` (~220 Scour interests: Rust, systems programming, e-ink, self-hosting, PKM, sci-fi, creative coding, space, running, board games, Boston Tech, etc.) — compile it into the taste profile at build time. **Reader profile (bake into curation prompts):** prefers long-form, high-effort, well-written articles on *any* topic; uses social proof (HN/Reddit upvotes+comments) as a quality proxy; wants tech news, light general/US world news (prefers Wikipedia Current Events for world news), Boston-area news, and ultra-niche community news. The full interest list lives in `data/scour-interests.opml` (~220 Scour interests: Rust, systems programming, e-ink, self-hosting, PKM, sci-fi, creative coding, space, running, board games, Boston Tech, etc.) — compile it into the taste profile at build time.
@@ -21,7 +21,7 @@
|---|---|---|---| |---|---|---|---|
| Miniflux | `127.0.0.1:8082` | `miniflux.hallada.net` | Installed via PPA. API auth via `X-Auth-Token` header. | | Miniflux | `127.0.0.1:8082` | `miniflux.hallada.net` | Installed via PPA. API auth via `X-Auth-Token` header. |
| BookOrbit | `127.0.0.1:3498` | `bookorbit.hallada.net` | NestJS/Vue/Postgres. Supports multiple isolated libraries, per-library watched folders, OPDS at `/api/v1/opds` (Basic auth, `opds_access` permission). | | BookOrbit | `127.0.0.1:3498` | `bookorbit.hallada.net` | NestJS/Vue/Postgres. Supports multiple isolated libraries, per-library watched folders, OPDS at `/api/v1/opds` (Basic auth, `opds_access` permission). |
| The Daily EPUB (new) | `127.0.0.1:<port, e.g. 3499>` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + XTC OPDS + static files. | | The Daily EPUB (new) | `127.0.0.1:<port, e.g. 3499>` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + OPDS catalog + downloads. |
### Secrets/config the operator must provide ### Secrets/config the operator must provide
@@ -46,7 +46,7 @@ Single Rust binary crate `daily-epub` (workspace not needed yet) with clap subco
``` ```
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
daily-epub serve # long-running: rating endpoints + XTC OPDS + static daily-epub serve # long-running: rating endpoints + OPDS catalog + downloads
daily-epub profile rebuild # regenerate taste profile from ratings (also runs weekly inside generate) daily-epub profile rebuild # regenerate taste profile from ratings (also runs weekly inside generate)
daily-epub backfill-social # re-poll social scores for recent entries (optional helper) daily-epub backfill-social # re-poll social scores for recent entries (optional helper)
daily-epub db migrate # run sqlx migrations (also auto-run on start) daily-epub db migrate # run sqlx migrations (also auto-run on start)
@@ -60,7 +60,7 @@ daily-epub db migrate # run sqlx migrations (also auto-run on start)
Miniflux ingest → normalize/dedupe → content extraction → social enrichment Miniflux ingest → normalize/dedupe → content extraction → social enrichment
→ heuristic pre-filter (500 → ~120) → LLM scoring (batched) → LLM selection (~120 → 1525) → heuristic pre-filter (500 → ~120) → LLM scoring (batched) → LLM selection (~120 → 1525)
→ comment fetching for selected → LLM editorial (summaries, section intros, front page) → comment fetching for selected → LLM editorial (summaries, section intros, front page)
→ EPUB build (standard + X4 editions) → XTC conversion → publish (BookOrbit folder, XTC dir, OPDS xml) → EPUB build (standard + X4 editions) → XTC conversion → publish (EPUB dir, XTC dir) → OPDS feed served from the EPUB dir
→ retention pruning → run report logged + stored → retention pruning → run report logged + stored
``` ```
@@ -103,8 +103,8 @@ the-daily-epub/
│ │ ├── templates/ # askama XHTML templates + CSS │ │ ├── templates/ # askama XHTML templates + CSS
│ │ ├── images.rs # download, resize, grayscale, re-encode │ │ ├── images.rs # download, resize, grayscale, re-encode
│ │ └── x4.rs # X4 edition transforms + XTC CLI invocation │ │ └── x4.rs # X4 edition transforms + XTC CLI invocation
│ ├── publish.rs # copy to BookOrbit folder, OPDS xml gen, retention │ ├── publish.rs # copy to the publish dirs, OPDS feed rendering, retention
│ ├── server.rs # axum: /r/… ratings, /opds/xtc.xml, /files/… │ ├── server.rs # axum: /r/… ratings, /opds/daily.xml, /files/…
│ └── report.rs # run summary (counts, cost, timings) │ └── report.rs # run summary (counts, cost, timings)
└── tests/ # integration tests with fixture JSON └── tests/ # integration tests with fixture JSON
``` ```
@@ -248,15 +248,40 @@ TOC: nav depth 2 (sections → articles, discussions nested). Metadata: `dc:titl
### 3.11 XTC conversion & publishing (`publish.rs`) ### 3.11 XTC conversion & publishing (`publish.rs`)
- Run the `epub-to-xtc-converter` CLI (Node 18+) on the X4 edition: invoke via `tokio::process::Command`, config keys `xtc.command` (default `epub-to-xtc`) and `xtc.args` (verify exact CLI name/flags from the repo README at implementation time; support both `.xtc` 1-bit and `.xtch` 4-level grayscale via config, default XTCH for image quality). Non-zero exit → log error, continue (XTC is a bonus artifact). - Run the `epub-to-xtc-converter` CLI (Node 18+) on the X4 edition: invoke via `tokio::process::Command`, config keys `xtc.command` (default `epub-to-xtc`) and `xtc.args` (verify exact CLI name/flags from the repo README at implementation time; support both `.xtc` 1-bit and `.xtch` 4-level grayscale via config, default XTCH for image quality). Non-zero exit → log error, continue (XTC is a bonus artifact).
- **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.bookorbit_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) — KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered. - **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.epub_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) — KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered.
- **XTC delivery:** copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead. - **XTC delivery** *(superseded — see the amendment below; kept for the record)*: copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead.
- **Retention:** delete issue files older than `retention_days` (default 21) from both dirs (BookOrbit's scan removes the DB entries); SQLite issue/rating history is kept forever (it's the training data). - **Retention:** delete issue files older than `retention_days` (default 21) from both dirs (BookOrbit's scan removes the DB entries); SQLite issue/rating history is kept forever (it's the training data).
> **Amended 2026-08-15 (post-M8), after testing against a real X4.** The XTC
> delivery bullet above does not work and has been replaced. CrossPoint's OPDS
> browser cannot acquire XTC at all — see the implementation notes' "Verified
> external facts" for the two firmware reasons — so:
>
> - The built-in feed lists **EPUBs, both editions**, from `publish.epub_dir`,
> with every acquisition link typed exactly `application/epub+zip` and files
> served from `/files/epub/{name}`. Canonical path `/opds/daily.xml`, with
> `/opds` and `/opds/` as aliases.
> - The feed is **rendered per request** from the directory rather than written to
> disk, so it cannot go stale behind a failed publish and the retention sweep has
> no generated index to step around.
> - This makes BookOrbit **optional**: the feed needs only the folder, and it puts
> the day's issue one screen from the X4's home rather than several clicks down a
> library tree.
> - XTC is still generated and still published to `publish.xtc_dir`, just not
> advertised. It stays fetchable at `/files/xtc/{name}` for sideloading.
> - `publish.xtc_dir` is swept by **count** (`xtc_retention_count`, default 5)
> rather than by age: an XTCH issue measured 79104 MB, so disk is the binding
> constraint. EPUBs keep the dated `retention_days` sweep.
> - `publish.bookorbit_dir` was renamed **`publish.epub_dir`** to match: the feed
> needs the directory, not BookOrbit. The old key is rejected outright
> (`deny_unknown_fields`) rather than silently falling back to the default,
> which would publish into a directory the feed does not read.
### 3.12 Server (`server.rs`) ### 3.12 Server (`server.rs`)
axum on `127.0.0.1:3499`: axum on `127.0.0.1:3499`:
- `GET /r/{date}/{article_id}/{vote}?t=` — verify HMAC, upsert rating, tiny HTML response. No auth beyond the token (links live inside a private EPUB; tokens are per-article+vote and unguessable). - `GET /r/{date}/{article_id}/{vote}?t=` — verify HMAC, upsert rating, tiny HTML response. No auth beyond the token (links live inside a private EPUB; tokens are per-article+vote and unguessable).
- `GET /opds/xtc.xml`, `GET /files/xtc/{name}` — optional Basic auth. - `GET /opds/daily.xml` (aliases `/opds`, `/opds/`), `GET /files/epub/{name}`, `GET /files/xtc/{name}` — optional Basic auth. *(Amended: was `/opds/xtc.xml` + `/files/xtc/` only; see §3.11.)*
- `GET /healthz`, `GET /issues.json` (recent run reports; handy for debugging). - `GET /healthz`, `GET /issues.json` (recent run reports; handy for debugging).
- `tower-http` request tracing; graceful shutdown on SIGTERM. - `tower-http` request tracing; graceful shutdown on SIGTERM.
@@ -317,7 +342,7 @@ sections = ["Top Stories", "Tech & Engineering", "Science & Space",
"Niche Corner", "From the Blogroll"] "Niche Corner", "From the Blogroll"]
[publish] [publish]
bookorbit_dir = "/srv/bookorbit/libraries/daily-epub" epub_dir = "/srv/bookorbit/libraries/daily-epub"
xtc_dir = "/var/lib/daily-epub/xtc" xtc_dir = "/var/lib/daily-epub/xtc"
[xtc] [xtc]
@@ -346,7 +371,7 @@ public_url = "https://daily.hallada.net"
3. **M3 — Pre-filter + LLM scoring/selection:** end-to-end lineup JSON printed in dry-run; token/cost report. *Verify: lineup is sane; cost < $0.50.* 3. **M3 — Pre-filter + LLM scoring/selection:** end-to-end lineup JSON printed in dry-run; token/cost report. *Verify: lineup is sane; cost < $0.50.*
4. **M4 — EPUB standard edition + publish:** full issue EPUB with cover, front page (temporary plain summaries), sections, articles, images; lands in BookOrbit, visible via OPDS on Kindle. *Verify: epubcheck clean; opens in KOReader with working TOC.* 4. **M4 — EPUB standard edition + publish:** full issue EPUB with cover, front page (temporary plain summaries), sections, articles, images; lands in BookOrbit, visible via OPDS on Kindle. *Verify: epubcheck clean; opens in KOReader with working TOC.*
5. **M5 — Editorial + comments:** DeepSeek summaries/intros/front page wired in; discussion chapters. *Verify: read an issue; comments legible on e-ink.* 5. **M5 — Editorial + comments:** DeepSeek summaries/intros/front page wired in; discussion chapters. *Verify: read an issue; comments legible on e-ink.*
6. **M6 — X4 edition + XTC + XTC OPDS:** second edition, converter invocation, static OPDS feed. *Verify: X4 fetches and renders both.* 6. **M6 — X4 edition + XTC + OPDS:** second edition, converter invocation, OPDS feed. *Verify: X4 fetches and renders both.*
7. **M7 — Feedback loop:** `serve` rating endpoints, links in chapters, feed priors in pre-filter, weekly profile rebuild. *Verify: tap 👍 in KOReader → row in `ratings` → prior changes next run.* 7. **M7 — Feedback loop:** `serve` rating endpoints, links in chapters, feed priors in pre-filter, weekly profile rebuild. *Verify: tap 👍 in KOReader → row in `ratings` → prior changes next run.*
8. **M8 — Hardening & ops:** systemd units, retention, cost guardrail, run reports, `issues.json`, README. 8. **M8 — Hardening & ops:** systemd units, retention, cost guardrail, run reports, `issues.json`, README.
@@ -354,7 +379,7 @@ public_url = "https://daily.hallada.net"
- `cargo test` — unit tests: URL canonicalization, dedupe clustering, HMAC round-trip, comment-tree truncation, prefilter scoring; integration tests over fixture JSON (recorded Miniflux/Algolia/Reddit responses) with the LLM stage mocked (`--skip-llm` uses prefilter order). - `cargo test` — unit tests: URL canonicalization, dedupe clustering, HMAC round-trip, comment-tree truncation, prefilter scoring; integration tests over fixture JSON (recorded Miniflux/Algolia/Reddit responses) with the LLM stage mocked (`--skip-llm` uses prefilter order).
- `daily-epub generate --dry-run --out ./out --max-articles 6` with real keys → inspect `./out/*.epub` in Calibre + run `epubcheck` (if installed) → zero errors. - `daily-epub generate --dry-run --out ./out --max-articles 6` with real keys → inspect `./out/*.epub` in Calibre + run `epubcheck` (if installed) → zero errors.
- Full live run: `daily-epub generate`file appears in BookOrbit UI under the Daily EPUB library only → browse BookOrbit OPDS from KOReader (Kindle/Palma), download, read; X4: CrossPoint OPDS → both the X4 EPUB (via BookOrbit) and XTC (via `daily.hallada.net/opds/xtc.xml`). - Full live run: `daily-epub generate`both editions appear in the publish dir (and in BookOrbit's UI if it is running) → browse `daily.hallada.net/opds/daily.xml` from KOReader (Kindle/Palma) or the X4's CrossPoint, download, read.
- Tap a rating link on the Kindle → confirmation page loads → `sqlite3 … 'select * from ratings'` shows the vote. - Tap a rating link on the Kindle → confirmation page loads → `sqlite3 … 'select * from ratings'` shows the vote.
- Watch `runs` for a week: cost per day, selection quality; tune `prefilter_keep`/prompts. - Watch `runs` for a week: cost per day, selection quality; tune `prefilter_keep`/prompts.
+36 -6
View File
@@ -51,8 +51,13 @@ pub struct Config {
pub target_article_count: usize, pub target_article_count: usize,
/// How many articles survive the heuristic pre-filter (§3.5). /// How many articles survive the heuristic pre-filter (§3.5).
pub prefilter_keep: usize, pub prefilter_keep: usize,
/// Days of published files kept in the publish dirs (§3.11). /// Days of published EPUBs kept in `publish.epub_dir` (§3.11).
pub retention_days: u32, pub retention_days: u32,
/// How many XTC issues to keep in `publish.xtc_dir` (§3.11).
///
/// Counted, not dated, because an XTCH issue is ~80100 MB of pre-rendered
/// page bitmaps: the constraint is disk, not age.
pub xtc_retention_count: u32,
/// Hard cost ceiling per run (§3.6 guardrail). /// Hard cost ceiling per run (§3.6 guardrail).
pub max_daily_usd: f64, pub max_daily_usd: f64,
/// Include the Wikipedia Current Events section (§3.8). /// Include the Wikipedia Current Events section (§3.8).
@@ -81,6 +86,7 @@ impl Default for Config {
target_article_count: 20, target_article_count: 20,
prefilter_keep: 120, prefilter_keep: 120,
retention_days: 21, retention_days: 21,
xtc_retention_count: 5,
max_daily_usd: 2.0, max_daily_usd: 2.0,
world_briefing: true, world_briefing: true,
database_path: PathBuf::from("/var/lib/daily-epub/daily-epub.db"), database_path: PathBuf::from("/var/lib/daily-epub/daily-epub.db"),
@@ -195,16 +201,20 @@ impl Default for CurationConfig {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields, default)] #[serde(deny_unknown_fields, default)]
pub struct PublishConfig { pub struct PublishConfig {
/// BookOrbit "The Daily EPUB" library watched folder. /// Where both EPUB editions land: the source of the OPDS feed, served at
pub bookorbit_dir: PathBuf, /// `/files/epub/`, and a BookOrbit watched folder if one is configured.
/// Directory served at `/files/xtc/`. ///
/// Renamed from `bookorbit_dir` once the built-in feed started serving this
/// directory directly — BookOrbit is optional, the directory is not.
pub epub_dir: PathBuf,
/// Directory served at `/files/xtc/`. Not listed in the OPDS feed (§3.11).
pub xtc_dir: PathBuf, pub xtc_dir: PathBuf,
} }
impl Default for PublishConfig { impl Default for PublishConfig {
fn default() -> Self { fn default() -> Self {
Self { Self {
bookorbit_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"), epub_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"),
xtc_dir: PathBuf::from("/var/lib/daily-epub/xtc"), xtc_dir: PathBuf::from("/var/lib/daily-epub/xtc"),
} }
} }
@@ -274,7 +284,7 @@ pub struct ServerConfig {
pub public_url: String, pub public_url: String,
/// HMAC key for rating tokens; supply via `DAILY_EPUB_SERVER__HMAC_SECRET`. /// HMAC key for rating tokens; supply via `DAILY_EPUB_SERVER__HMAC_SECRET`.
pub hmac_secret: Option<String>, pub hmac_secret: Option<String>,
/// Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`. /// Optional Basic auth for `/opds/*` and `/files/*`.
pub basic_auth_user: Option<String>, pub basic_auth_user: Option<String>,
pub basic_auth_pass: Option<String>, pub basic_auth_pass: Option<String>,
} }
@@ -431,6 +441,26 @@ mod tests {
)); ));
} }
/// `publish.bookorbit_dir` was renamed to `publish.epub_dir`. A config still
/// using the old key must fail loudly and name both — silently falling back
/// to the default would publish the issue into the wrong directory, where
/// the OPDS feed would then find nothing.
#[test]
fn the_renamed_publish_key_fails_loudly() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(
&path,
"[publish]\nbookorbit_dir = \"/srv/books\"\nxtc_dir = \"/srv/xtc\"\n",
)
.unwrap();
let err = Config::load(Some(&path)).expect_err("the stale key must be rejected");
let message = err.to_string();
assert!(message.contains("bookorbit_dir"), "{message}");
assert!(message.contains("epub_dir"), "{message}");
}
#[test] #[test]
fn shipped_example_config_parses() { fn shipped_example_config_parses() {
let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml"); let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml");
+1 -4
View File
@@ -31,7 +31,7 @@ struct Cli {
enum Command { enum Command {
/// Build (and publish) one issue. /// Build (and publish) one issue.
Generate(GenerateArgs), Generate(GenerateArgs),
/// Run the rating endpoints, XTC OPDS feed and static files. /// Run the rating endpoints, the OPDS catalog and downloads.
Serve, Serve,
/// Taste-profile maintenance. /// Taste-profile maintenance.
#[command(subcommand)] #[command(subcommand)]
@@ -167,9 +167,6 @@ fn print_outcome(outcome: &GenerateOutcome) {
if let Some(xtc) = &published.xtc { if let Some(xtc) = &published.xtc {
println!("published: {}", xtc.display()); println!("published: {}", xtc.display());
} }
if let Some(opds) = &published.opds {
println!("opds: {}", opds.display());
}
if published.pruned > 0 { if published.pruned > 0 {
println!("pruned: {} expired files", published.pruned); println!("pruned: {} expired files", published.pruned);
} }
+1 -1
View File
@@ -510,7 +510,7 @@ async fn run_stages(
); );
None None
} else { } else {
let published = publish::publish_issue(db, config, &issue, &artifacts, xtc.as_deref()) let published = publish::publish_issue(config, &issue, &artifacts, xtc.as_deref())
.await .await
.context("publishing the issue")?; .context("publishing the issue")?;
record_issue(db, &issue, &published) record_issue(db, &issue, &published)
+299 -153
View File
@@ -1,6 +1,9 @@
//! Publishing: BookOrbit watched folder, XTC delivery, OPDS feed, retention //! Publishing: the EPUB library folder, XTC delivery, OPDS feed, retention
//! (spec §3.11). //! (spec §3.11).
//! //!
//! `publish.epub_dir` holds both EPUB editions and is what the OPDS feed lists;
//! BookOrbit may watch the same folder but nothing here depends on it.
//!
//! Everything here is deliberately dumb about *how* artifacts were produced: the //! Everything here is deliberately dumb about *how* artifacts were produced: the
//! EPUB/XTC stages hand over finished files, this module only copies, indexes and //! EPUB/XTC stages hand over finished files, this module only copies, indexes and
//! prunes them. Copies are atomic (temp file in the destination directory, then //! prunes them. Copies are atomic (temp file in the destination directory, then
@@ -8,11 +11,15 @@
//! half-written book. //! half-written book.
//! //!
//! [`crate::pipeline`] ends a non-dry run with one call — //! [`crate::pipeline`] ends a non-dry run with one call —
//! `publish_issue(db, config, &issue, &artifacts, xtc_path.as_deref())`, where //! `publish_issue(config, &issue, &artifacts, xtc_path.as_deref())`, where
//! `artifacts` are the `epub::build_all` outputs and `xtc_path` is //! `artifacts` are the `epub::build_all` outputs and `xtc_path` is
//! `epub::x4::convert`'s output (`None` when the converter is disabled or //! `epub::x4::convert`'s output (`None` when the converter is disabled or
//! failed) — and feeds the returned [`Published`] paths into //! failed) — and feeds the returned [`Published`] paths into
//! `db.upsert_issue(..., epub_path, x4_path, xtc_path, ...)`. //! `db.upsert_issue(..., epub_path, x4_path, xtc_path, ...)`.
//!
//! The OPDS feed ([`build_opds`]) is rendered per request by
//! [`crate::server`] rather than written here, and lists **EPUBs only** — see
//! its docs for why XTC cannot be delivered over OPDS.
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@@ -26,14 +33,19 @@ use crate::config::Config;
use crate::db::Db; use crate::db::Db;
use crate::types::{Artifact, Edition, Issue}; use crate::types::{Artifact, Edition, Issue};
/// Filename of the generated static OPDS feed (§3.11). /// Number of issue *days* listed in the OPDS feed; each contributes one entry
pub const XTC_OPDS_FILENAME: &str = "xtc.xml"; /// per edition (§3.11).
/// Number of issues listed in the XTC OPDS feed (§3.11). pub const OPDS_FEED_ISSUES: usize = 14;
pub const XTC_FEED_ENTRIES: usize = 14;
/// Every published file starts with this (the retention sweep keys off it). /// Every published file starts with this (the retention sweep keys off it).
pub const FILE_PREFIX: &str = "The Daily EPUB - "; pub const FILE_PREFIX: &str = "The Daily EPUB - ";
/// Extensions the retention sweep is allowed to delete (§3.11). /// Extensions the retention sweep is allowed to delete (§3.11).
pub const PRUNABLE_EXTENSIONS: [&str; 3] = ["epub", "xtc", "xtch"]; pub const PRUNABLE_EXTENSIONS: [&str; 3] = ["epub", "xtc", "xtch"];
/// Extensions of the XTC artifacts, for the counted sweep of `xtc_dir` (§3.11).
pub const XTC_EXTENSIONS: [&str; 2] = ["xtc", "xtch"];
/// Canonical path of the OPDS feed, relative to `server.public_url` (§3.11).
pub const OPDS_PATH: &str = "/opds/daily.xml";
/// The one acquisition type CrossPoint's OPDS parser accepts (§3.11).
pub const EPUB_CONTENT_TYPE: &str = "application/epub+zip";
#[derive(Debug, thiserror::Error)] #[derive(Debug, thiserror::Error)]
pub enum PublishError { pub enum PublishError {
@@ -61,8 +73,6 @@ pub struct Published {
pub epubs: Vec<Artifact>, pub epubs: Vec<Artifact>,
/// The XTC artifact's published location, when the converter produced one. /// The XTC artifact's published location, when the converter produced one.
pub xtc: Option<PathBuf>, pub xtc: Option<PathBuf>,
/// The regenerated OPDS feed.
pub opds: Option<PathBuf>,
/// How many expired files the retention sweep removed. /// How many expired files the retention sweep removed.
pub pruned: usize, pub pruned: usize,
} }
@@ -136,7 +146,7 @@ async fn ensure_dir(dir: &Path) -> Result<(), PublishError> {
.map_err(PublishError::at(dir)) .map_err(PublishError::at(dir))
} }
/// Copy both EPUB editions into the BookOrbit watched folder (§3.11). /// Copy both EPUB editions into `publish.epub_dir` (§3.11).
/// ///
/// Returns the published paths in the same order as `artifacts`. /// Returns the published paths in the same order as `artifacts`.
pub async fn publish_epubs( pub async fn publish_epubs(
@@ -144,7 +154,7 @@ pub async fn publish_epubs(
issue: &Issue, issue: &Issue,
cfg: &Config, cfg: &Config,
) -> Result<Vec<PathBuf>, PublishError> { ) -> Result<Vec<PathBuf>, PublishError> {
let dir = &cfg.publish.bookorbit_dir; let dir = &cfg.publish.epub_dir;
ensure_dir(dir).await?; ensure_dir(dir).await?;
let mut published = Vec::with_capacity(artifacts.len()); let mut published = Vec::with_capacity(artifacts.len());
for artifact in artifacts { for artifact in artifacts {
@@ -159,7 +169,7 @@ pub async fn publish_epubs(
edition = ?artifact.edition, edition = ?artifact.edition,
dest = %dest.display(), dest = %dest.display(),
bytes = artifact.bytes, bytes = artifact.bytes,
"published edition to the BookOrbit library" "published edition to the EPUB library"
); );
published.push(dest); published.push(dest);
} }
@@ -180,12 +190,11 @@ pub async fn publish_xtc(xtc: &Path, cfg: &Config) -> Result<PathBuf, PublishErr
Ok(dest) Ok(dest)
} }
/// Publish everything one run produced, refresh the OPDS feed and prune (§3.11). /// Publish everything one run produced and prune (§3.11).
/// ///
/// `xtc` is `None` when the converter is disabled or failed — that is not an /// `xtc` is `None` when the converter is disabled or failed — that is not an
/// error, the X4 falls back to the EPUB edition from BookOrbit. /// error, since the OPDS feed lists the EPUB editions either way.
pub async fn publish_issue( pub async fn publish_issue(
db: &Db,
cfg: &Config, cfg: &Config,
issue: &Issue, issue: &Issue,
artifacts: &[Artifact], artifacts: &[Artifact],
@@ -208,67 +217,74 @@ pub async fn publish_issue(
Some(src) => Some(publish_xtc(src, cfg).await?), Some(src) => Some(publish_xtc(src, cfg).await?),
None => None, None => None,
}; };
let opds = Some(write_xtc_opds(db, cfg).await?); // The OPDS feed is rendered per request from the publish directory, so
// there is nothing to write here (§3.11).
let pruned = prune(cfg, issue.meta.date).await?; let pruned = prune(cfg, issue.meta.date).await?;
Ok(Published { Ok(Published { epubs, xtc, pruned })
epubs,
xtc,
opds,
pruned,
})
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// OPDS 1.2 acquisition feed (§3.11) // OPDS 1.2 acquisition feed (§3.11)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// One published XTC file, as listed in the feed. /// One published EPUB, as listed in the feed.
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
struct XtcFile { struct EpubFile {
name: String, name: String,
date: Option<Date>, date: Option<Date>,
edition: Edition,
modified: Timestamp, modified: Timestamp,
bytes: u64, bytes: u64,
} }
/// Regenerate the static OPDS 1.2 acquisition feed for the XTC directory: /// Render the OPDS 1.2 acquisition feed for the EPUB publish directory (§3.11).
/// newest first, last [`XTC_FEED_ENTRIES`], entries typed ///
/// `application/octet-stream` (§3.11). /// Rendered per request rather than written to disk: the directory is the only
pub async fn write_xtc_opds(db: &Db, cfg: &Config) -> Result<PathBuf, PublishError> { /// source of truth, so the feed cannot go stale behind a failed publish, and
let dir = &cfg.publish.xtc_dir; /// there is no generated file for the retention sweep to step around.
///
/// XTC artifacts are deliberately **not** listed. CrossPoint's OPDS browser only
/// acquires links typed `application/epub+zip` and always saves the result with
/// a `.epub` extension, which its reader dispatches on — so an XTC offered here
/// would either be invisible or download into a file that cannot be opened.
pub async fn build_opds(db: &Db, cfg: &Config) -> Result<String, PublishError> {
let dir = &cfg.publish.epub_dir;
ensure_dir(dir).await?; ensure_dir(dir).await?;
let files = scan_xtc_dir(dir).await?; let files = scan_epub_dir(dir).await?;
let numbers = issue_numbers(db, &files).await; let numbers = issue_numbers(db, &files).await;
let feed = render_opds(&files, &numbers, &cfg.server.public_url, Timestamp::now()); Ok(render_opds(
&files,
let dest = dir.join(XTC_OPDS_FILENAME); &numbers,
write_atomic(&dest, feed.as_bytes()).await?; &cfg.server.public_url,
tracing::info!(entries = files.len(), dest = %dest.display(), "wrote the XTC OPDS feed"); Timestamp::now(),
Ok(dest) ))
} }
/// XTC artifacts in `dir`, newest first, capped at [`XTC_FEED_ENTRIES`]. /// Published EPUBs in `dir`, newest first, limited to the last
async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> { /// [`OPDS_FEED_ISSUES`] issue days (both editions of each).
async fn scan_epub_dir(dir: &Path) -> Result<Vec<EpubFile>, PublishError> {
let mut entries = tokio::fs::read_dir(dir) let mut entries = tokio::fs::read_dir(dir)
.await .await
.map_err(PublishError::at(dir))?; .map_err(PublishError::at(dir))?;
let mut files = Vec::new(); let mut files = Vec::new();
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? { while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
let name = entry.file_name().to_string_lossy().into_owned(); let name = entry.file_name().to_string_lossy().into_owned();
let extension = Path::new(&name) let path = Path::new(&name);
let extension = path
.extension() .extension()
.and_then(|e| e.to_str()) .and_then(|e| e.to_str())
.unwrap_or_default() .unwrap_or_default()
.to_ascii_lowercase(); .to_ascii_lowercase();
if !matches!(extension.as_str(), "xtc" | "xtch") { // Only our own issues: a shared library may hold other people's books.
if extension != "epub" || date_from_filename(&name).is_none() {
continue; continue;
} }
let meta = match entry.metadata().await { let meta = match entry.metadata().await {
Ok(meta) if meta.is_file() => meta, Ok(meta) if meta.is_file() => meta,
Ok(_) => continue, Ok(_) => continue,
Err(e) => { Err(e) => {
tracing::warn!(error = %e, name, "skipping unreadable XTC file"); tracing::warn!(error = %e, name, "skipping unreadable EPUB");
continue; continue;
} }
}; };
@@ -277,29 +293,47 @@ async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
.ok() .ok()
.and_then(|m| Timestamp::try_from(m).ok()) .and_then(|m| Timestamp::try_from(m).ok())
.unwrap_or_else(Timestamp::now); .unwrap_or_else(Timestamp::now);
files.push(XtcFile { let stem = path
.file_stem()
.and_then(|s| s.to_str())
.unwrap_or_default()
.to_string();
files.push(EpubFile {
date: date_from_filename(&name), date: date_from_filename(&name),
edition: Edition::from_file_stem(&stem),
name, name,
modified, modified,
bytes: meta.len(), bytes: meta.len(),
}); });
} }
// Newest first: by issue date when the filename carries one, else by mtime. // Newest issue first, and within an issue the standard edition leads.
files.sort_by(|a, b| { files.sort_by(|a, b| {
b.date b.date
.cmp(&a.date) .cmp(&a.date)
.then(a.edition.cmp(&b.edition))
.then(b.modified.cmp(&a.modified)) .then(b.modified.cmp(&a.modified))
.then(a.name.cmp(&b.name)) .then(a.name.cmp(&b.name))
}); });
files.truncate(XTC_FEED_ENTRIES); // Cap by issue day, not by file: an issue is two entries and truncating
// mid-issue would list one edition without the other.
let mut kept_dates: Vec<Option<Date>> = Vec::new();
files.retain(|file| {
if !kept_dates.contains(&file.date) {
kept_dates.push(file.date);
}
kept_dates.iter().position(|d| *d == file.date) < Some(OPDS_FEED_ISSUES)
});
Ok(files) Ok(files)
} }
/// Issue numbers for the dated files, best-effort (the feed is still valid /// Issue numbers for the dated files, best-effort (the feed is still valid
/// without them). Uses the `db` escape hatch — no bespoke helper in `db.rs`. /// without them). Uses the `db` escape hatch — no bespoke helper in `db.rs`.
async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> { async fn issue_numbers(db: &Db, files: &[EpubFile]) -> BTreeMap<Date, i64> {
let mut numbers = BTreeMap::new(); let mut numbers = BTreeMap::new();
for date in files.iter().filter_map(|f| f.date) { for date in files.iter().filter_map(|f| f.date) {
if numbers.contains_key(&date) {
continue;
}
let row = sqlx::query("SELECT issue_number FROM issues WHERE date = ?") let row = sqlx::query("SELECT issue_number FROM issues WHERE date = ?")
.bind(date.to_string()) .bind(date.to_string())
.fetch_optional(db.pool()) .fetch_optional(db.pool())
@@ -317,13 +351,13 @@ async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
/// Render the Atom/OPDS document (§3.11). /// Render the Atom/OPDS document (§3.11).
fn render_opds( fn render_opds(
files: &[XtcFile], files: &[EpubFile],
numbers: &BTreeMap<Date, i64>, numbers: &BTreeMap<Date, i64>,
public_url: &str, public_url: &str,
now: Timestamp, now: Timestamp,
) -> String { ) -> String {
let base = public_url.trim_end_matches('/'); let base = public_url.trim_end_matches('/');
let self_href = format!("{base}/opds/xtc.xml"); let self_href = format!("{base}{OPDS_PATH}");
let updated = files.first().map(|f| f.modified).unwrap_or(now); let updated = files.first().map(|f| f.modified).unwrap_or(now);
let mut out = String::with_capacity(1024 + files.len() * 512); let mut out = String::with_capacity(1024 + files.len() * 512);
@@ -333,8 +367,8 @@ fn render_opds(
xmlns:dc=\"http://purl.org/dc/terms/\" \ xmlns:dc=\"http://purl.org/dc/terms/\" \
xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n", xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
); );
out.push_str(" <id>urn:daily-epub:xtc</id>\n"); out.push_str(" <id>urn:daily-epub:issues</id>\n");
out.push_str(" <title>The Daily EPUB — XTC editions</title>\n"); out.push_str(" <title>The Daily EPUB</title>\n");
out.push_str(&format!(" <updated>{}</updated>\n", rfc3339(updated))); out.push_str(&format!(" <updated>{}</updated>\n", rfc3339(updated)));
out.push_str(" <author><name>The Daily EPUB</name></author>\n"); out.push_str(" <author><name>The Daily EPUB</name></author>\n");
out.push_str(&format!( out.push_str(&format!(
@@ -349,19 +383,25 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
)); ));
for file in files { for file in files {
// Matches the EPUB's own `dc:title`, so the two editions of one issue
// are told apart in the list rather than showing as the same book.
let title = match file.date { let title = match file.date {
Some(date) => format!("The Daily EPUB — {date}"), Some(date) => format!("The Daily EPUB — {date}{}", file.edition.file_suffix()),
None => file.name.clone(), None => file.name.clone(),
}; };
let summary = match file.date.and_then(|d| numbers.get(&d)) { let edition_note = match file.edition {
Some(n) => format!("Issue #{n} · {}", human_bytes(file.bytes)), Edition::Standard => "Standard",
None => human_bytes(file.bytes), Edition::X4 => "Xteink X4",
}; };
let href = format!("{base}/files/xtc/{}", percent_encode(&file.name)); let summary = match file.date.and_then(|d| numbers.get(&d)) {
Some(n) => format!("Issue #{n} · {edition_note} · {}", human_bytes(file.bytes)),
None => format!("{edition_note} · {}", human_bytes(file.bytes)),
};
let href = format!("{base}/files/epub/{}", percent_encode(&file.name));
out.push_str(" <entry>\n"); out.push_str(" <entry>\n");
out.push_str(&format!(" <title>{}</title>\n", xml_escape(&title))); out.push_str(&format!(" <title>{}</title>\n", xml_escape(&title)));
out.push_str(&format!( out.push_str(&format!(
" <id>urn:daily-epub:xtc:{}</id>\n", " <id>urn:daily-epub:issue:{}</id>\n",
xml_escape(&percent_encode(&file.name)) xml_escape(&percent_encode(&file.name))
)); ));
out.push_str(&format!( out.push_str(&format!(
@@ -376,9 +416,12 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
" <summary>{}</summary>\n", " <summary>{}</summary>\n",
xml_escape(&summary) xml_escape(&summary)
)); ));
// The type must be exactly `application/epub+zip`: CrossPoint's OPDS
// parser compares it with `strcmp` and silently drops entries whose
// acquisition link is anything else, reporting "No entries found".
out.push_str(&format!( out.push_str(&format!(
" <link rel=\"http://opds-spec.org/acquisition\" href=\"{}\" \ " <link rel=\"http://opds-spec.org/acquisition\" href=\"{}\" \
type=\"application/octet-stream\" length=\"{}\"/>\n", type=\"{EPUB_CONTENT_TYPE}\" length=\"{}\"/>\n",
xml_escape(&href), xml_escape(&href),
file.bytes file.bytes
)); ));
@@ -430,43 +473,82 @@ fn percent_encode(s: &str) -> String {
out out
} }
async fn write_atomic(dest: &Path, bytes: &[u8]) -> Result<(), PublishError> {
let dir = dest.parent().unwrap_or_else(|| Path::new("."));
let tmp = dir.join(format!(
".{}.{}.tmp",
dest.file_name()
.and_then(|n| n.to_str())
.unwrap_or("daily-epub"),
std::process::id()
));
tokio::fs::write(&tmp, bytes)
.await
.map_err(PublishError::at(&tmp))?;
tokio::fs::rename(&tmp, dest)
.await
.map_err(PublishError::at(dest))
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Retention (§3.11) // Retention (§3.11)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Delete issue files older than `retention_days` from both publish dirs. /// Retention sweep over both publish dirs. SQLite history is kept forever —
/// SQLite history is kept forever — it's the training data (§3.11). /// it's the training data (§3.11).
///
/// The two directories are swept on different rules: EPUBs age out after
/// `retention_days`, while XTC is capped at `xtc_retention_count` issues because
/// each one is ~80100 MB of pre-rendered page bitmaps and the binding
/// constraint is disk rather than age.
/// ///
/// Only files named `The Daily EPUB - YYYY-MM-DD*.{epub,xtc,xtch}` are ever /// Only files named `The Daily EPUB - YYYY-MM-DD*.{epub,xtc,xtch}` are ever
/// considered; anything else in those directories (including `xtc.xml` and other /// considered; anything else in those directories (other people's books) is left
/// people's books) is left strictly alone. /// strictly alone.
pub async fn prune(cfg: &Config, today: Date) -> Result<usize, PublishError> { pub async fn prune(cfg: &Config, today: Date) -> Result<usize, PublishError> {
let cutoff = today let cutoff = today
.checked_sub(jiff::Span::new().days(i64::from(cfg.retention_days))) .checked_sub(jiff::Span::new().days(i64::from(cfg.retention_days)))
.unwrap_or(today); .unwrap_or(today);
let mut removed = 0; let mut removed = prune_dir(&cfg.publish.epub_dir, cutoff).await?;
for dir in [&cfg.publish.bookorbit_dir, &cfg.publish.xtc_dir] {
removed += prune_dir(dir, cutoff).await?;
}
if removed > 0 { if removed > 0 {
tracing::info!(removed, %cutoff, "retention sweep removed expired issues"); tracing::info!(removed, %cutoff, "retention sweep removed expired EPUBs");
}
let xtc_removed = prune_xtc_dir(&cfg.publish.xtc_dir, cfg.xtc_retention_count as usize).await?;
if xtc_removed > 0 {
tracing::info!(
removed = xtc_removed,
keep = cfg.xtc_retention_count,
"retention sweep trimmed the XTC directory"
);
}
removed += xtc_removed;
Ok(removed)
}
/// Keep only the newest `keep` XTC issues, deleting the rest (§3.11).
///
/// Counted rather than dated so the directory has a hard size ceiling no matter
/// how often `generate` runs.
async fn prune_xtc_dir(dir: &Path, keep: usize) -> Result<usize, PublishError> {
if !dir.exists() {
return Ok(0);
}
let mut entries = tokio::fs::read_dir(dir)
.await
.map_err(PublishError::at(dir))?;
let mut ours: Vec<(Date, PathBuf)> = Vec::new();
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
let name = entry.file_name().to_string_lossy().into_owned();
let extension = Path::new(&name)
.extension()
.and_then(|e| e.to_str())
.unwrap_or_default()
.to_ascii_lowercase();
if !XTC_EXTENSIONS.contains(&extension.as_str()) {
continue;
}
let Some(date) = date_from_filename(&name) else {
continue;
};
if !entry.metadata().await.map(|m| m.is_file()).unwrap_or(false) {
continue;
}
ours.push((date, entry.path()));
}
// Newest first, then drop everything past the cap.
ours.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(&b.1)));
let mut removed = 0;
for (date, path) in ours.into_iter().skip(keep) {
match tokio::fs::remove_file(&path).await {
Ok(()) => {
tracing::info!(path = %path.display(), %date, "pruned an XTC issue past the cap");
removed += 1;
}
Err(e) => tracing::warn!(error = %e, path = %path.display(), "could not prune file"),
}
} }
Ok(removed) Ok(removed)
} }
@@ -520,7 +602,7 @@ mod tests {
fn cfg(dir: &Path) -> Config { fn cfg(dir: &Path) -> Config {
let mut cfg = Config::default(); let mut cfg = Config::default();
cfg.publish.bookorbit_dir = dir.join("bookorbit"); cfg.publish.epub_dir = dir.join("bookorbit");
cfg.publish.xtc_dir = dir.join("xtc"); cfg.publish.xtc_dir = dir.join("xtc");
cfg.server.public_url = "https://daily.hallada.net".into(); cfg.server.public_url = "https://daily.hallada.net".into();
cfg cfg
@@ -561,7 +643,7 @@ mod tests {
Some(date("2026-08-15")) Some(date("2026-08-15"))
); );
for foreign in [ for foreign in [
"xtc.xml", "The Daily EPUB - 2026-08-15.xml",
"Moby Dick.epub", "Moby Dick.epub",
"The Daily EPUB - notadate.epub", "The Daily EPUB - notadate.epub",
"The Daily EPUB - 2026-08-15.txt", "The Daily EPUB - 2026-08-15.txt",
@@ -634,10 +716,10 @@ mod tests {
paths, paths,
vec![ vec![
cfg.publish cfg.publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-08-15.epub"), .join("The Daily EPUB - 2026-08-15.epub"),
cfg.publish cfg.publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-08-15 (X4).epub"), .join("The Daily EPUB - 2026-08-15 (X4).epub"),
] ]
); );
@@ -673,21 +755,37 @@ mod tests {
} }
} }
fn epub_file(name: &str, edition: Edition, modified: &str, bytes: u64) -> EpubFile {
EpubFile {
date: date_from_filename(name),
edition,
name: name.into(),
modified: ts(modified),
bytes,
}
}
#[test] #[test]
fn opds_feed_is_newest_first_with_acquisition_links() { fn opds_feed_is_newest_first_with_acquisition_links() {
let files = vec![ let files = vec![
XtcFile { epub_file(
name: "The Daily EPUB - 2026-08-15 (X4).xtch".into(), "The Daily EPUB - 2026-08-15.epub",
date: Some(date("2026-08-15")), Edition::Standard,
modified: ts("2026-08-15T05:40:00Z"), "2026-08-15T05:40:00Z",
bytes: 2_500_000, 6_500_000,
}, ),
XtcFile { epub_file(
name: "The Daily EPUB - 2026-08-14 (X4).xtch".into(), "The Daily EPUB - 2026-08-15 (X4).epub",
date: Some(date("2026-08-14")), Edition::X4,
modified: ts("2026-08-14T05:40:00Z"), "2026-08-15T05:40:00Z",
bytes: 4096, 1_700_000,
}, ),
epub_file(
"The Daily EPUB - 2026-08-14.epub",
Edition::Standard,
"2026-08-14T05:40:00Z",
4096,
),
]; ];
let mut numbers = BTreeMap::new(); let mut numbers = BTreeMap::new();
numbers.insert(date("2026-08-15"), 12); numbers.insert(date("2026-08-15"), 12);
@@ -700,37 +798,68 @@ mod tests {
assert!(feed.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>")); assert!(feed.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>"));
assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\"")); assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\""));
assert!(feed.contains("<id>urn:daily-epub:xtc</id>")); assert!(feed.contains("<id>urn:daily-epub:issues</id>"));
assert!(feed.contains("<updated>2026-08-15T05:40:00Z</updated>")); assert!(feed.contains("<updated>2026-08-15T05:40:00Z</updated>"));
assert_eq!(feed.matches("<entry>").count(), 2); assert_eq!(feed.matches("<entry>").count(), 3);
assert_eq!(feed.matches("</entry>").count(), 2); assert_eq!(feed.matches("</entry>").count(), 3);
// Newest first.
let i15 = feed.find("The Daily EPUB — 2026-08-15").unwrap(); // Newest issue first; the two editions of one issue are distinguishable
let i14 = feed.find("The Daily EPUB — 2026-08-14").unwrap(); // by title, matching each EPUB's own `dc:title`.
assert!(i15 < i14); let i15 = feed
// Acquisition link, encoded filename, absolute public URL, size. .find("<title>The Daily EPUB — 2026-08-15</title>")
.unwrap();
let i15_x4 = feed
.find("<title>The Daily EPUB — 2026-08-15 (X4)</title>")
.unwrap();
let i14 = feed
.find("<title>The Daily EPUB — 2026-08-14</title>")
.unwrap();
assert!(i15 < i15_x4 && i15_x4 < i14);
// CrossPoint compares the acquisition type with `strcmp` against
// `application/epub+zip` and drops the entry on any mismatch (§3.11).
assert_eq!(
feed.matches("type=\"application/epub+zip\"").count(),
3,
"{feed}"
);
assert!(!feed.contains("application/octet-stream"));
assert!(feed.contains( assert!(feed.contains(
"<link rel=\"http://opds-spec.org/acquisition\" \ "<link rel=\"http://opds-spec.org/acquisition\" \
href=\"https://daily.hallada.net/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch\" \ href=\"https://daily.hallada.net/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub\" \
type=\"application/octet-stream\" length=\"2500000\"/>" type=\"application/epub+zip\" length=\"1700000\"/>"
)); ));
assert!(feed.contains("Issue #12 · 2.4 MB")); assert!(feed.contains("Issue #12 · Standard · 6.2 MB"));
assert!(feed.contains("Issue #12 · Xteink X4 · 1.6 MB"));
assert!(
feed.contains("<link rel=\"self\" href=\"https://daily.hallada.net/opds/daily.xml\"")
);
assert!(feed.trim_end().ends_with("</feed>")); assert!(feed.trim_end().ends_with("</feed>"));
assert!(!feed.contains("&<"), "unescaped markup leaked in"); assert!(!feed.contains("&<"), "unescaped markup leaked in");
} }
#[tokio::test] #[tokio::test]
async fn write_xtc_opds_lists_only_xtc_files_capped_at_fourteen() { async fn the_feed_lists_both_editions_of_the_last_fourteen_issues() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let cfg = cfg(dir.path()); let cfg = cfg(dir.path());
std::fs::create_dir_all(&cfg.publish.epub_dir).unwrap();
std::fs::create_dir_all(&cfg.publish.xtc_dir).unwrap(); std::fs::create_dir_all(&cfg.publish.xtc_dir).unwrap();
for day in 1..=20 { for day in 1..=20 {
let name = format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch"); for edition in Edition::ALL {
std::fs::write(cfg.publish.xtc_dir.join(name), b"x").unwrap(); let name = issue_filename(date(&format!("2026-08-{day:02}")), edition, "epub");
std::fs::write(cfg.publish.epub_dir.join(name), b"x").unwrap();
} }
// Non-XTC neighbours must be ignored. }
std::fs::write(cfg.publish.xtc_dir.join("README.txt"), b"x").unwrap(); // Other people's books and our own XTC artifacts must be ignored.
std::fs::write(cfg.publish.xtc_dir.join("cover.epub"), b"x").unwrap(); std::fs::write(cfg.publish.epub_dir.join("Moby Dick.epub"), b"x").unwrap();
std::fs::write(cfg.publish.epub_dir.join("metadata.db"), b"x").unwrap();
std::fs::write(
cfg.publish
.xtc_dir
.join("The Daily EPUB - 2026-08-20 (X4).xtch"),
b"x",
)
.unwrap();
let db = Db::open_and_migrate(&dir.path().join("db.sqlite")) let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
.await .await
@@ -748,64 +877,75 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
.await .await
.unwrap(); .unwrap();
let path = write_xtc_opds(&db, &cfg).await.unwrap(); let feed = build_opds(&db, &cfg).await.unwrap();
assert_eq!(path, cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)); // Capped by issue day, so both editions of each of the last 14 survive.
let feed = std::fs::read_to_string(&path).unwrap(); assert_eq!(feed.matches("<entry>").count(), OPDS_FEED_ISSUES * 2);
assert_eq!(feed.matches("<entry>").count(), XTC_FEED_ENTRIES);
assert!(feed.contains("2026-08-20")); assert!(feed.contains("2026-08-20"));
assert!(!feed.contains("2026-08-06"), "older than the last 14"); assert!(!feed.contains("2026-08-06"), "older than the last 14");
assert!(!feed.contains("README")); assert!(!feed.contains("Moby Dick"));
assert!(!feed.contains("cover.epub")); assert!(!feed.contains("metadata.db"));
assert!(feed.contains("Issue #20")); assert!(feed.contains("Issue #20"));
// XTC is never offered: CrossPoint cannot acquire it (§3.11).
// Regenerating replaces the file in place. assert!(!feed.contains(".xtch"), "{feed}");
write_xtc_opds(&db, &cfg).await.unwrap(); assert!(!feed.contains("/files/xtc/"));
let leftovers: Vec<String> = std::fs::read_dir(&cfg.publish.xtc_dir)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
.filter(|n| n.ends_with(".tmp"))
.collect();
assert!(leftovers.is_empty(), "{leftovers:?}");
} }
#[tokio::test] #[tokio::test]
async fn prune_only_deletes_old_matching_files() { async fn prune_ages_out_epubs_but_counts_xtc() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let mut conf = cfg(dir.path()); let mut conf = cfg(dir.path());
conf.retention_days = 21; conf.retention_days = 21;
std::fs::create_dir_all(&conf.publish.bookorbit_dir).unwrap(); conf.xtc_retention_count = 5;
std::fs::create_dir_all(&conf.publish.epub_dir).unwrap();
std::fs::create_dir_all(&conf.publish.xtc_dir).unwrap(); std::fs::create_dir_all(&conf.publish.xtc_dir).unwrap();
let keep_epub = conf let keep_epub = conf
.publish .publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-08-14.epub"); .join("The Daily EPUB - 2026-08-14.epub");
let old_epub = conf let old_epub = conf
.publish .publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-07-01.epub"); .join("The Daily EPUB - 2026-07-01.epub");
let old_x4 = conf let old_x4 = conf
.publish .publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-07-01 (X4).epub"); .join("The Daily EPUB - 2026-07-01 (X4).epub");
let foreign = conf.publish.bookorbit_dir.join("Moby Dick.epub"); let foreign = conf.publish.epub_dir.join("Moby Dick.epub");
let old_xtc = conf for path in [&keep_epub, &old_epub, &old_x4, &foreign] {
.publish
.xtc_dir
.join("The Daily EPUB - 2026-07-01 (X4).xtch");
let feed = conf.publish.xtc_dir.join(XTC_OPDS_FILENAME);
for path in [&keep_epub, &old_epub, &old_x4, &foreign, &old_xtc, &feed] {
std::fs::write(path, b"x").unwrap(); std::fs::write(path, b"x").unwrap();
} }
// Eight consecutive XTC issues, all recent: age would keep every one,
// the count cap keeps the newest five.
let xtc: Vec<PathBuf> = (8..=15)
.map(|day| {
let path = conf
.publish
.xtc_dir
.join(format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch"));
std::fs::write(&path, b"x").unwrap();
path
})
.collect();
let foreign_xtc = conf.publish.xtc_dir.join("Someone Else.xtch");
std::fs::write(&foreign_xtc, b"x").unwrap();
// 2 expired EPUBs + 3 XTC issues past the cap of 5.
let removed = prune(&conf, date("2026-08-15")).await.unwrap(); let removed = prune(&conf, date("2026-08-15")).await.unwrap();
assert_eq!(removed, 3); assert_eq!(removed, 5);
assert!(keep_epub.exists()); assert!(keep_epub.exists());
assert!(foreign.exists(), "never touch other people's books"); assert!(foreign.exists(), "never touch other people's books");
assert!(feed.exists(), "the OPDS feed is not an issue file"); assert!(foreign_xtc.exists(), "nor their XTC files");
assert!(!old_epub.exists()); assert!(!old_epub.exists());
assert!(!old_x4.exists()); assert!(!old_x4.exists());
assert!(!old_xtc.exists()); // The five newest XTC issues (11th15th) survive; 8th10th are gone.
for path in &xtc[..3] {
assert!(!path.exists(), "{} should be pruned", path.display());
}
for path in &xtc[3..] {
assert!(path.exists(), "{} should be kept", path.display());
}
// Idempotent, and tolerant of missing directories. // Idempotent, and tolerant of missing directories.
assert_eq!(prune(&conf, date("2026-08-15")).await.unwrap(), 0); assert_eq!(prune(&conf, date("2026-08-15")).await.unwrap(), 0);
@@ -844,25 +984,31 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
]; ];
let issue = fake_issue(date("2026-08-15")); let issue = fake_issue(date("2026-08-15"));
let published = publish_issue(&db, &cfg, &issue, &artifacts, Some(&xtc_src)) let published = publish_issue(&cfg, &issue, &artifacts, Some(&xtc_src))
.await .await
.unwrap(); .unwrap();
assert_eq!(published.epubs.len(), 2); assert_eq!(published.epubs.len(), 2);
assert!(published.epubs.iter().all(|a| a.path.exists())); assert!(published.epubs.iter().all(|a| a.path.exists()));
assert_eq!(published.epubs[1].edition, Edition::X4); assert_eq!(published.epubs[1].edition, Edition::X4);
assert!(published.xtc.as_ref().is_some_and(|p| p.exists())); assert!(published.xtc.as_ref().is_some_and(|p| p.exists()));
assert!(published.opds.as_ref().is_some_and(|p| p.exists()));
assert_eq!(published.pruned, 0); assert_eq!(published.pruned, 0);
let feed = std::fs::read_to_string(cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)).unwrap(); // The feed is derived from the publish directory, so both editions show
assert!(feed.contains("out.xtch")); // up without publish having written anything.
let feed = build_opds(&db, &cfg).await.unwrap();
assert!(
feed.contains("The Daily EPUB — 2026-08-15</title>"),
"{feed}"
);
assert!(
feed.contains("The Daily EPUB — 2026-08-15 (X4)</title>"),
"{feed}"
);
assert!(!feed.contains("out.xtch"));
// No XTC artifact is fine — the feed is still regenerated. // No XTC artifact is fine — the EPUBs are what the feed lists anyway.
let published = publish_issue(&db, &cfg, &issue, &artifacts, None) let published = publish_issue(&cfg, &issue, &artifacts, None).await.unwrap();
.await
.unwrap();
assert!(published.xtc.is_none()); assert!(published.xtc.is_none());
assert!(published.opds.is_some());
} }
#[test] #[test]
+89 -35
View File
@@ -1,4 +1,4 @@
//! axum server: rating endpoints, XTC OPDS, static files (spec §3.9, §3.12). //! axum server: rating endpoints, the OPDS catalog, downloads (spec §3.9, §3.12).
//! //!
//! Rating links must work from an e-reader's built-in browser, so every rating //! Rating links must work from an e-reader's built-in browser, so every rating
//! endpoint is a `GET` and the response is a tiny e-ink-sized HTML page. //! endpoint is a `GET` and the response is a tiny e-ink-sized HTML page.
@@ -7,8 +7,9 @@
//! | route | behaviour | //! | route | behaviour |
//! |---|---| //! |---|---|
//! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC, upsert rating, rebuild feed priors | //! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC, upsert rating, rebuild feed priors |
//! | `GET /opds/xtc.xml` | static OPDS 1.2 acquisition feed from `publish.xtc_dir` | //! | `GET /opds/daily.xml` (also `/opds`, `/opds/`) | OPDS 1.2 acquisition feed over `publish.epub_dir` |
//! | `GET /files/xtc/{name}` | XTC artifact download (no path traversal) | //! | `GET /files/epub/{name}` | EPUB download — what the feed's acquisition links point at |
//! | `GET /files/xtc/{name}` | XTC artifact download, unlisted (no path traversal) |
//! | `GET /healthz` | liveness | //! | `GET /healthz` | liveness |
//! | `GET /issues.json` | the last 30 run reports, newest first | //! | `GET /issues.json` | the last 30 run reports, newest first |
//! //!
@@ -81,16 +82,18 @@ pub use crate::auth::{constant_time_eq, rating_token, rating_url, verify_token};
// Router (§3.12) // Router (§3.12)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/xtc.xml`, /// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/daily.xml`,
/// `/files/xtc/{name}`, `/healthz`, `/issues.json`, with `tower-http` tracing (§3.12). /// `/files/epub/{name}`, `/files/xtc/{name}`, `/healthz`, `/issues.json`, with
/// `tower-http` tracing (§3.12).
pub fn router(state: AppState) -> Router { pub fn router(state: AppState) -> Router {
Router::new() Router::new()
.route("/r/{date}/{article_id}/{vote}", get(handle_rating)) .route("/r/{date}/{article_id}/{vote}", get(handle_rating))
.route("/opds/xtc.xml", get(handle_opds)) .route(crate::publish::OPDS_PATH, get(handle_opds))
// OPDS browsers are typed into by hand on a 6" e-ink keyboard: serve the // OPDS browsers are typed into by hand on a 6" e-ink keyboard: serve the
// same feed from the catalog root so a URL without the filename works. // same feed from the catalog root so a URL without the filename works.
.route("/opds", get(handle_opds)) .route("/opds", get(handle_opds))
.route("/opds/", get(handle_opds)) .route("/opds/", get(handle_opds))
.route("/files/epub/{name}", get(handle_epub_file))
.route("/files/xtc/{name}", get(handle_xtc_file)) .route("/files/xtc/{name}", get(handle_xtc_file))
.route("/healthz", get(handle_healthz)) .route("/healthz", get(handle_healthz))
.route("/issues.json", get(handle_issues_json)) .route("/issues.json", get(handle_issues_json))
@@ -284,44 +287,64 @@ async fn handle_rating(
) )
} }
/// `GET /opds/xtc.xml` — the static feed written by [`crate::publish`] (§3.11). /// `GET /opds/daily.xml` — both EPUB editions of the last issues, newest first,
/// rendered from the publish directory on each request (§3.11).
async fn handle_opds(State(state): State<AppState>, headers: HeaderMap) -> Response { async fn handle_opds(State(state): State<AppState>, headers: HeaderMap) -> Response {
if let Some(challenge) = check_basic_auth(&state.config, &headers) { if let Some(challenge) = check_basic_auth(&state.config, &headers) {
return challenge; return challenge;
} }
let path = state match crate::publish::build_opds(&state.db, &state.config).await {
.config Ok(feed) => (
.publish
.xtc_dir
.join(crate::publish::XTC_OPDS_FILENAME);
match tokio::fs::read(&path).await {
Ok(bytes) => (
StatusCode::OK, StatusCode::OK,
[ [
(header::CONTENT_TYPE, OPDS_CONTENT_TYPE), (header::CONTENT_TYPE, OPDS_CONTENT_TYPE),
(header::CACHE_CONTROL, "no-cache"), (header::CACHE_CONTROL, "no-cache"),
], ],
bytes, feed,
) )
.into_response(), .into_response(),
Err(e) => { Err(e) => {
tracing::warn!(error = %e, path = %path.display(), "no XTC OPDS feed yet"); tracing::error!(error = %e, "could not build the OPDS feed");
(StatusCode::NOT_FOUND, "no feed yet").into_response() (
StatusCode::INTERNAL_SERVER_ERROR,
"could not build the feed",
)
.into_response()
} }
} }
} }
/// `GET /files/epub/{name}` — download one published EPUB; this is what the
/// OPDS acquisition links point at (§3.11).
async fn handle_epub_file(
State(state): State<AppState>,
Path(name): Path<String>,
headers: HeaderMap,
) -> Response {
let dir = state.config.publish.epub_dir.clone();
serve_file(&state, &dir, &name, &headers).await
}
/// `GET /files/xtc/{name}` — download one XTC artifact (§3.11). /// `GET /files/xtc/{name}` — download one XTC artifact (§3.11).
///
/// Not listed in the OPDS feed — CrossPoint's browser cannot acquire XTC — but
/// kept so the artifacts can still be fetched by URL for sideloading.
async fn handle_xtc_file( async fn handle_xtc_file(
State(state): State<AppState>, State(state): State<AppState>,
Path(name): Path<String>, Path(name): Path<String>,
headers: HeaderMap, headers: HeaderMap,
) -> Response { ) -> Response {
if let Some(challenge) = check_basic_auth(&state.config, &headers) { let dir = state.config.publish.xtc_dir.clone();
serve_file(&state, &dir, &name, &headers).await
}
/// Stream one file out of `dir`, behind the OPDS Basic auth (§3.11).
async fn serve_file(state: &AppState, dir: &FsPath, name: &str, headers: &HeaderMap) -> Response {
if let Some(challenge) = check_basic_auth(&state.config, headers) {
return challenge; return challenge;
} }
let Some(path) = safe_join(&state.config.publish.xtc_dir, &name) else { let Some(path) = safe_join(dir, name) else {
tracing::warn!(name, "rejected an unsafe XTC file name"); tracing::warn!(name, "rejected an unsafe file name");
return (StatusCode::BAD_REQUEST, "bad file name").into_response(); return (StatusCode::BAD_REQUEST, "bad file name").into_response();
}; };
// An XTCH issue is a pre-rendered page bitmap per page — ~100 MB for a full // An XTCH issue is a pre-rendered page bitmap per page — ~100 MB for a full
@@ -332,12 +355,14 @@ async fn handle_xtc_file(
(file, len) (file, len)
} }
Err(e) => { Err(e) => {
tracing::warn!(error = %e, path = %path.display(), "XTC file not found"); tracing::warn!(error = %e, path = %path.display(), "file not found");
return (StatusCode::NOT_FOUND, "not found").into_response(); return (StatusCode::NOT_FOUND, "not found").into_response();
} }
}; };
let content_type = if name.ends_with(".xml") { // CrossPoint dispatches on the saved file's extension, not on this header,
OPDS_CONTENT_TYPE // but Calibre and KOReader both use it.
let content_type = if name.ends_with(".epub") {
crate::publish::EPUB_CONTENT_TYPE
} else { } else {
"application/octet-stream" "application/octet-stream"
}; };
@@ -629,7 +654,9 @@ mod tests {
async fn start(with_auth: bool) -> TestServer { async fn start(with_auth: bool) -> TestServer {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let xtc_dir = dir.path().join("xtc"); let xtc_dir = dir.path().join("xtc");
let epub_dir = dir.path().join("epub");
std::fs::create_dir_all(&xtc_dir).unwrap(); std::fs::create_dir_all(&xtc_dir).unwrap();
std::fs::create_dir_all(&epub_dir).unwrap();
let db = Db::open_and_migrate(&dir.path().join("db.sqlite")) let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
.await .await
.unwrap(); .unwrap();
@@ -637,6 +664,8 @@ mod tests {
let mut config = Config::default(); let mut config = Config::default();
config.server.hmac_secret = Some(VECTOR_SECRET.into()); config.server.hmac_secret = Some(VECTOR_SECRET.into());
config.publish.xtc_dir = xtc_dir; config.publish.xtc_dir = xtc_dir;
config.publish.epub_dir = epub_dir;
config.server.public_url = "https://daily.hallada.net".into();
if with_auth { if with_auth {
config.server.basic_auth_user = Some("opds".into()); config.server.basic_auth_user = Some("opds".into());
config.server.basic_auth_pass = Some("hunter2".into()); config.server.basic_auth_pass = Some("hunter2".into());
@@ -663,6 +692,10 @@ mod tests {
self._dir.path().join("xtc") self._dir.path().join("xtc")
} }
fn epub_dir(&self) -> PathBuf {
self._dir.path().join("epub")
}
async fn seed_article(&self) -> ArticleId { async fn seed_article(&self) -> ArticleId {
let entry = crate::types::Entry { let entry = crate::types::Entry {
id: 1, id: 1,
@@ -847,11 +880,10 @@ mod tests {
#[tokio::test] #[tokio::test]
async fn opds_and_files_are_served_behind_basic_auth() { async fn opds_and_files_are_served_behind_basic_auth() {
let server = TestServer::start(true).await; let server = TestServer::start(true).await;
std::fs::write( for edition in crate::types::Edition::ALL {
server.xtc_dir().join(crate::publish::XTC_OPDS_FILENAME), let name = crate::publish::issue_filename(date(), edition, "epub");
"<feed/>", std::fs::write(server.epub_dir().join(name), b"EPUB").unwrap();
) }
.unwrap();
std::fs::write( std::fs::write(
server server
.xtc_dir() .xtc_dir()
@@ -861,7 +893,7 @@ mod tests {
.unwrap(); .unwrap();
let res = client() let res = client()
.get(format!("{}/opds/xtc.xml", server.base)) .get(format!("{}/opds/daily.xml", server.base))
.send() .send()
.await .await
.unwrap(); .unwrap();
@@ -875,7 +907,7 @@ mod tests {
); );
let res = client() let res = client()
.get(format!("{}/opds/xtc.xml", server.base)) .get(format!("{}/opds/daily.xml", server.base))
.basic_auth("opds", Some("wrong")) .basic_auth("opds", Some("wrong"))
.send() .send()
.await .await
@@ -883,7 +915,7 @@ mod tests {
assert_eq!(res.status(), 401); assert_eq!(res.status(), 401);
let res = client() let res = client()
.get(format!("{}/opds/xtc.xml", server.base)) .get(format!("{}/opds/daily.xml", server.base))
.basic_auth("opds", Some("hunter2")) .basic_auth("opds", Some("hunter2"))
.send() .send()
.await .await
@@ -895,7 +927,11 @@ mod tests {
.unwrap() .unwrap()
.starts_with("application/atom+xml") .starts_with("application/atom+xml")
); );
assert_eq!(res.text().await.unwrap(), "<feed/>"); let feed = res.text().await.unwrap();
assert_eq!(feed.matches("<entry>").count(), 2, "{feed}");
assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}");
// XTC exists on disk but is never advertised (§3.11).
assert!(!feed.contains(".xtch"), "{feed}");
// The catalog root serves the same feed, behind the same auth. // The catalog root serves the same feed, behind the same auth.
for alias in ["/opds", "/opds/"] { for alias in ["/opds", "/opds/"] {
@@ -912,9 +948,29 @@ mod tests {
.await .await
.unwrap(); .unwrap();
assert_eq!(res.status(), 200, "{alias}"); assert_eq!(res.status(), 200, "{alias}");
assert_eq!(res.text().await.unwrap(), "<feed/>", "{alias}"); assert_eq!(res.text().await.unwrap(), feed, "{alias}");
} }
// The acquisition link resolves, typed as an EPUB.
let res = client()
.get(format!(
"{}/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20(X4).epub",
server.base
))
.basic_auth("opds", Some("hunter2"))
.send()
.await
.unwrap();
assert_eq!(res.status(), 200);
// CrossPoint needs the size up front to show download progress.
assert_eq!(res.content_length(), Some(4));
assert_eq!(
res.headers()[header::CONTENT_TYPE].to_str().unwrap(),
"application/epub+zip"
);
assert_eq!(res.bytes().await.unwrap().as_ref(), b"EPUB");
// XTC is still fetchable by URL for sideloading, just not listed.
let res = client() let res = client()
.get(format!( .get(format!(
"{}/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20(X4).xtch", "{}/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20(X4).xtch",
@@ -925,8 +981,6 @@ mod tests {
.await .await
.unwrap(); .unwrap();
assert_eq!(res.status(), 200); assert_eq!(res.status(), 200);
// CrossPoint needs the size up front to show download progress.
assert_eq!(res.content_length(), Some(4));
assert_eq!(res.bytes().await.unwrap().as_ref(), b"XTCH"); assert_eq!(res.bytes().await.unwrap().as_ref(), b"XTCH");
// Ratings are not behind auth (the token is the credential). // Ratings are not behind auth (the token is the credential).
+19 -1
View File
@@ -411,7 +411,10 @@ pub const WORLD_BRIEFING_SECTION: &str = "World Briefing";
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Which of the two editions is being built (§3.10). /// Which of the two editions is being built (§3.10).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] ///
/// The declaration order is the listing order: standard first, then X4. The
/// OPDS feed sorts on it.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")] #[serde(rename_all = "snake_case")]
pub enum Edition { pub enum Edition {
/// 1200px images, full CSS. /// 1200px images, full CSS.
@@ -421,6 +424,9 @@ pub enum Edition {
} }
impl Edition { impl Edition {
/// Every edition, in the order they are listed and built (§3.10).
pub const ALL: [Edition; 2] = [Edition::Standard, Edition::X4];
/// Filename suffix: `""` / `" (X4)"` (§3.11). /// Filename suffix: `""` / `" (X4)"` (§3.11).
pub fn file_suffix(self) -> &'static str { pub fn file_suffix(self) -> &'static str {
match self { match self {
@@ -428,6 +434,18 @@ impl Edition {
Edition::X4 => " (X4)", Edition::X4 => " (X4)",
} }
} }
/// Recover the edition from a published filename's stem (§3.11).
///
/// The OPDS feed is rebuilt by scanning the publish directory, so the
/// filename is the only record of which edition a file is.
pub fn from_file_stem(stem: &str) -> Edition {
if stem.ends_with(Edition::X4.file_suffix()) {
Edition::X4
} else {
Edition::Standard
}
}
} }
/// Issue-level metadata rendered on the cover, front page and OPF (§3.10). /// Issue-level metadata rendered on the cover, front page and OPF (§3.10).
+19 -11
View File
@@ -59,7 +59,7 @@ fn test_config(root: &Path) -> Config {
prefilter_keep: 20, prefilter_keep: 20,
world_briefing: false, world_briefing: false,
publish: PublishConfig { publish: PublishConfig {
bookorbit_dir: root.join("bookorbit"), epub_dir: root.join("bookorbit"),
xtc_dir: root.join("xtc"), xtc_dir: root.join("xtc"),
}, },
xtc: XtcConfig { xtc: XtcConfig {
@@ -305,35 +305,44 @@ async fn assemble_build_publish(
); );
// --- Publish (§3.11) --- // --- Publish (§3.11) ---
let published = publish::publish_issue(db, cfg, &issue, &artifacts, None) let published = publish::publish_issue(cfg, &issue, &artifacts, None)
.await .await
.expect("publish"); .expect("publish");
assert_eq!(published.epubs.len(), 2); assert_eq!(published.epubs.len(), 2);
for artifact in &published.epubs { for artifact in &published.epubs {
assert!(artifact.path.starts_with(&cfg.publish.bookorbit_dir)); assert!(artifact.path.starts_with(&cfg.publish.epub_dir));
assert!(artifact.path.exists(), "{}", artifact.path.display()); assert!(artifact.path.exists(), "{}", artifact.path.display());
} }
assert!( assert!(
cfg.publish cfg.publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-08-15.epub") .join("The Daily EPUB - 2026-08-15.epub")
.exists() .exists()
); );
assert!( assert!(
cfg.publish cfg.publish
.bookorbit_dir .epub_dir
.join("The Daily EPUB - 2026-08-15 (X4).epub") .join("The Daily EPUB - 2026-08-15 (X4).epub")
.exists() .exists()
); );
assert!(published.xtc.is_none(), "the converter is disabled here"); assert!(published.xtc.is_none(), "the converter is disabled here");
// The OPDS feed is regenerated on every publish, even with no XTC files yet. // The OPDS feed is derived from what was just published — both editions,
let opds = published.opds.clone().expect("an OPDS feed was written"); // typed so CrossPoint will accept them (§3.11).
assert_eq!(opds, cfg.publish.xtc_dir.join("xtc.xml")); let feed = publish::build_opds(db, cfg).await.expect("build the feed");
let feed = std::fs::read_to_string(&opds).expect("read the OPDS feed");
assert!(feed.starts_with("<?xml"), "{feed}"); assert!(feed.starts_with("<?xml"), "{feed}");
assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\"")); assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\""));
assert!(feed.contains(&cfg.server.public_url)); assert!(feed.contains(&cfg.server.public_url));
assert_eq!(feed.matches("<entry>").count(), 2, "{feed}");
assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}");
assert!(
feed.contains("The Daily EPUB — 2026-08-15</title>"),
"{feed}"
);
assert!(
feed.contains("The Daily EPUB — 2026-08-15 (X4)</title>"),
"{feed}"
);
// --- Record (§3.13) --- // --- Record (§3.13) ---
let epub_path = published let epub_path = published
@@ -438,7 +447,6 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
// Re-running the same date replaces rather than duplicates (notes §12). // Re-running the same date replaces rather than duplicates (notes §12).
let republished = publish::publish_issue( let republished = publish::publish_issue(
&db,
&cfg, &cfg,
&issue, &issue,
&[ &[
@@ -458,7 +466,7 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
.await .await
.expect("republish"); .expect("republish");
assert_eq!(republished.epubs.len(), 2); assert_eq!(republished.epubs.len(), 2);
let files: Vec<String> = std::fs::read_dir(&cfg.publish.bookorbit_dir) let files: Vec<String> = std::fs::read_dir(&cfg.publish.epub_dir)
.expect("read bookorbit dir") .expect("read bookorbit dir")
.filter_map(|e| e.ok().map(|e| e.file_name().to_string_lossy().into_owned())) .filter_map(|e| e.ok().map(|e| e.file_name().to_string_lossy().into_owned()))
.collect(); .collect();
+48 -26
View File
@@ -41,6 +41,7 @@ impl Server {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let xtc_dir = dir.path().join("xtc"); let xtc_dir = dir.path().join("xtc");
std::fs::create_dir_all(&xtc_dir).expect("xtc dir"); std::fs::create_dir_all(&xtc_dir).expect("xtc dir");
std::fs::create_dir_all(dir.path().join("bookorbit")).expect("epub dir");
let log = std::fs::File::create(dir.path().join("server.log")).expect("log file"); let log = std::fs::File::create(dir.path().join("server.log")).expect("log file");
let port = free_port(); let port = free_port();
@@ -59,10 +60,7 @@ impl Server {
format!("http://127.0.0.1:{port}"), format!("http://127.0.0.1:{port}"),
) )
.env("DAILY_EPUB_PUBLISH__XTC_DIR", &xtc_dir) .env("DAILY_EPUB_PUBLISH__XTC_DIR", &xtc_dir)
.env( .env("DAILY_EPUB_PUBLISH__EPUB_DIR", dir.path().join("bookorbit"))
"DAILY_EPUB_PUBLISH__BOOKORBIT_DIR",
dir.path().join("bookorbit"),
)
.stdout(Stdio::null()) .stdout(Stdio::null())
.stderr(Stdio::from(log)); .stderr(Stdio::from(log));
if basic_auth { if basic_auth {
@@ -80,6 +78,10 @@ impl Server {
self.dir.path().join("xtc") self.dir.path().join("xtc")
} }
fn epub_dir(&self) -> std::path::PathBuf {
self.dir.path().join("bookorbit")
}
fn wait_until_ready(&self) { fn wait_until_ready(&self) {
let deadline = Instant::now() + Duration::from_secs(30); let deadline = Instant::now() + Duration::from_secs(30);
while Instant::now() < deadline { while Instant::now() < deadline {
@@ -217,22 +219,38 @@ fn binary_serves_health_issues_and_rating_endpoints() {
#[test] #[test]
fn binary_serves_opds_and_files_behind_basic_auth() { fn binary_serves_opds_and_files_behind_basic_auth() {
let server = Server::start(true); let server = Server::start(true);
let name = "The Daily EPUB - 2026-08-15 (X4).xtch"; write(
write(&server.xtc_dir().join("xtc.xml"), FEED); &server.epub_dir().join("The Daily EPUB - 2026-08-15.epub"),
write(&server.xtc_dir().join(name), "XTCH"); "STANDARD",
);
write(
&server
.epub_dir()
.join("The Daily EPUB - 2026-08-15 (X4).epub"),
"X4EPUB",
);
write(
&server
.xtc_dir()
.join("The Daily EPUB - 2026-08-15 (X4).xtch"),
"XTCH",
);
write(&server.dir.path().join("secret"), "top secret"); write(&server.dir.path().join("secret"), "top secret");
// No credentials → challenge. // No credentials → challenge.
let res = server.get("/opds/xtc.xml"); let res = server.get("/opds/daily.xml");
assert_eq!(res.status, 401); assert_eq!(res.status, 401);
assert_eq!( assert_eq!(
res.header("www-authenticate"), res.header("www-authenticate"),
Some("Basic realm=\"The Daily EPUB\", charset=\"UTF-8\"") Some("Basic realm=\"The Daily EPUB\", charset=\"UTF-8\"")
); );
assert_eq!(server.get_auth("/opds/xtc.xml", "bm9wZTpub3Bl").status, 401); assert_eq!(
server.get_auth("/opds/daily.xml", "bm9wZTpub3Bl").status,
401
);
// Correct credentials → the feed, typed as OPDS. // Correct credentials → a feed built from the publish dir, typed as OPDS.
let res = server.get_auth("/opds/xtc.xml", BASIC_AUTH); let res = server.get_auth("/opds/daily.xml", BASIC_AUTH);
assert_eq!(res.status, 200); assert_eq!(res.status, 200);
assert!( assert!(
res.header("content-type") res.header("content-type")
@@ -242,8 +260,27 @@ fn binary_serves_opds_and_files_behind_basic_auth() {
res.headers res.headers
); );
assert!(res.body.contains("opds-spec.org/acquisition")); assert!(res.body.contains("opds-spec.org/acquisition"));
// Both editions, and only as `application/epub+zip` — the one acquisition
// type CrossPoint's parser accepts.
assert_eq!(res.body.matches("<entry>").count(), 2, "{}", res.body);
assert_eq!(
res.body.matches("type=\"application/epub+zip\"").count(),
2,
"{}",
res.body
);
assert!(!res.body.contains(".xtch"), "{}", res.body);
// The acquisition link in the feed resolves to the file itself. // The acquisition link in the feed resolves to the file itself.
let res = server.get_auth(
"/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub",
BASIC_AUTH,
);
assert_eq!(res.status, 200);
assert_eq!(res.body, "X4EPUB");
assert_eq!(res.header("content-type"), Some("application/epub+zip"));
// XTC stays reachable by URL for sideloading, just unlisted.
let res = server.get_auth( let res = server.get_auth(
"/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch", "/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch",
BASIC_AUTH, BASIC_AUTH,
@@ -272,18 +309,3 @@ fn binary_serves_opds_and_files_behind_basic_auth() {
fn write(path: &Path, body: &str) { fn write(path: &Path, body: &str) {
std::fs::write(path, body).unwrap_or_else(|e| panic!("writing {}: {e}", path.display())); std::fs::write(path, body).unwrap_or_else(|e| panic!("writing {}: {e}", path.display()));
} }
/// A feed shaped like the one `publish::write_xtc_opds` generates.
const FEED: &str = r#"<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<id>urn:daily-epub:xtc</id>
<title>The Daily EPUB — XTC editions</title>
<updated>2026-08-15T05:40:00Z</updated>
<entry>
<title>The Daily EPUB — 2026-08-15</title>
<id>urn:daily-epub:xtc:x</id>
<updated>2026-08-15T05:40:00Z</updated>
<link rel="http://opds-spec.org/acquisition" href="http://127.0.0.1/files/xtc/x.xtch" type="application/octet-stream" length="4"/>
</entry>
</feed>
"#;