Serve EPUBs instead XTC in daily OPDS
It turns out it is currently impossible for the Xteink X4 to download XTC files from an OPDS server. So I switched the built-in OPDS server in this binary (daily.hallada.net/opds) to serve the EPUB files instead of the XTC files. The X4 is pretty capable of reading the X4 edition of the EPUB that was optimized for it anyways, and I prefer the flexibility of EPUB, so I might end up eventually deleting the XTC conversion. For now, I kept the conversion step (in case I ever want to ever try manually copying them to the device) and I keep a limited number of XTC issues on the server since they are quite big in filesize. The BookOrbit integration is now optional since the built-in OPDS server is able to serve the EPUBs. In my installation, I serve the EPUBs through both. The daily.hallada.net/opds server is just a little quicker to navigate and download the EPUBs on the X4 since the BookOrbit OPDS requires diving into a couple layers of folders before you get to the files.
This commit is contained in:
@@ -8,9 +8,9 @@ the articles, enriches them with HackerNews/Lobsters/Reddit social proof, filter
|
||||
300–500 candidates down to ~120 with cheap heuristics, and asks DeepSeek to score,
|
||||
select and introduce 15–25 of them. It assembles two EPUB editions (a standard one
|
||||
and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and
|
||||
drops everything into a [BookOrbit](https://github.com/thallada/bookorbit) watched
|
||||
folder so KOReader can pick it up over OPDS. Each article chapter ends with 👍/👎
|
||||
links that feed back into tomorrow's curation.
|
||||
publishes the lot over its own OPDS catalog — which doubles as a
|
||||
[BookOrbit](https://github.com/thallada/bookorbit) watched folder if you run one.
|
||||
Each article chapter ends with 👍/👎 links that feed back into tomorrow's curation.
|
||||
|
||||
Steady-state cost is roughly **$0.05–0.30/day** in DeepSeek tokens, hard-capped by
|
||||
`max_daily_usd`.
|
||||
@@ -49,7 +49,7 @@ selects, feed excerpts stand in for summaries) instead of losing the day's issue
|
||||
| **Miniflux** with an API key | the only content source | Settings → API Keys. The client is read-only and never mutates read state. |
|
||||
| **DeepSeek API key** | curation + editorial | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. |
|
||||
| A 32+ byte random secret | signs the 👍/👎 rating links | `openssl rand -hex 32` |
|
||||
| **BookOrbit** library + watched folder | delivery to KOReader over OPDS | Create a dedicated "The Daily EPUB" library, enable *Watch folders*, note the folder path. |
|
||||
| **BookOrbit** library + watched folder | *optional* — a richer library UI on top of the same folder | Delivery does not need it: `daily-epub serve` has its own OPDS catalog over `publish.epub_dir`. If you do run it, create a dedicated "The Daily EPUB" library, enable *Watch folders*, and point `publish.epub_dir` at it. |
|
||||
| **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF — see below. It has **no global npm bin** — it is invoked as `node <repo>/cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. |
|
||||
| A reverse proxy for `daily.hallada.net` → `127.0.0.1:3499` | rating links must be reachable from e-readers on the internet | TLS via your existing setup. |
|
||||
|
||||
@@ -71,7 +71,7 @@ sudo install -m0755 target/release/daily-epub /usr/local/bin/
|
||||
|
||||
```
|
||||
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
|
||||
daily-epub serve # rating endpoints + XTC OPDS + static files
|
||||
daily-epub serve # rating endpoints + OPDS catalog + downloads
|
||||
daily-epub profile rebuild # regenerate the taste profile from ratings (weekly inside generate)
|
||||
daily-epub backfill-social # re-poll social scores for recent articles
|
||||
daily-epub db migrate # run migrations (also automatic on every start)
|
||||
@@ -108,7 +108,8 @@ Secrets belong in the environment file, never in the TOML.
|
||||
| `lookback_hours` | `26` | Size of the ingest window ending at the issue day's end (clamped to now). |
|
||||
| `target_article_count` | `20` | Lineup size the selector aims for. `--max-articles` overrides it. |
|
||||
| `prefilter_keep` | `120` | Candidates surviving the heuristic pre-filter. Must be ≥ `target_article_count`. |
|
||||
| `retention_days` | `21` | Published files older than this are deleted from both publish dirs. SQLite history is kept forever. |
|
||||
| `retention_days` | `21` | EPUBs older than this are deleted from `publish.epub_dir`. SQLite history is kept forever. |
|
||||
| `xtc_retention_count` | `5` | How many XTC issues to keep in `publish.xtc_dir`. Counted, not dated: each `.xtch` is ~80–100 MB, so the binding constraint is disk, not age. |
|
||||
| `max_daily_usd` | `2.0` | Hard ceiling on DeepSeek spend **per day**, not per run — a re-run inherits what earlier runs for that date already spent. Tripping it skips remaining LLM work and degrades to excerpts. |
|
||||
| `world_briefing` | `true` | Include the Wikipedia Current Events section. |
|
||||
| `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. |
|
||||
@@ -130,8 +131,8 @@ Secrets belong in the environment file, never in the TOML.
|
||||
| `curation.blocked_domains` | `[]` | Hosts excluded outright. |
|
||||
| `curation.paywall_domains` | `[]` | Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, economist, …). |
|
||||
| `curation.sections` | 8 sections | The **only** section names the model may use. `World Briefing` is reserved and never offered. |
|
||||
| `publish.bookorbit_dir` | `/srv/bookorbit/libraries/daily-epub` | BookOrbit watched folder. Both EPUB editions land here by atomic copy, distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. |
|
||||
| `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts + the generated `xtc.xml` OPDS feed. |
|
||||
| `publish.epub_dir` | `/srv/bookorbit/libraries/daily-epub` | Both EPUB editions land here by atomic copy, and this is the directory the OPDS feed lists. The editions are distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. Point a BookOrbit watched folder at it if you want its UI too. **Renamed from `bookorbit_dir`**; the old key is a hard config error. |
|
||||
| `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts. **Not** listed in the OPDS feed — CrossPoint cannot acquire them — but downloadable at `/files/xtc/<name>` for sideloading. |
|
||||
| `xtc.enabled` | `true` | Set `false` to skip the converter entirely. |
|
||||
| `xtc.command` | `node` | Converter executable. |
|
||||
| `xtc.args` | `["/opt/epub-to-xtc-converter/cli/index.js", "convert"]` | Prefix; the code appends `<input.epub> -o <output> -f <format>` (plus `-c <settings>`). |
|
||||
@@ -140,7 +141,7 @@ Secrets belong in the environment file, never in the TOML.
|
||||
| `server.bind` | `127.0.0.1:3499` | Listen address. |
|
||||
| `server.public_url` | `https://daily.hallada.net` | Base URL the rating links inside the EPUB are built from. |
|
||||
| `server.hmac_secret` | — | **`DAILY_EPUB_SERVER__HMAC_SECRET`** (or `DAILY_EPUB_SECRET`). Without it, generated links are rejected with 403. |
|
||||
| `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`. |
|
||||
| `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/*` and `/files/*`. |
|
||||
|
||||
---
|
||||
|
||||
@@ -185,7 +186,7 @@ sudo systemctl enable --now daily-epub.service daily-epub-generate.timer
|
||||
> ```
|
||||
> ReadWritePaths=/home/thallada/bookorbit/books/daily-epub /var/lib/daily-epub/xtc
|
||||
> ```
|
||||
> If you change `publish.bookorbit_dir` or `publish.xtc_dir` in the config, change
|
||||
> If you change `publish.epub_dir` or `publish.xtc_dir` in the config, change
|
||||
> these lines too and `systemctl daemon-reload`, or publishing fails with
|
||||
> `Read-only file system`.
|
||||
|
||||
@@ -196,7 +197,7 @@ Node for the XTC converter, whose JIT needs W+X pages.
|
||||
|
||||
The rating links baked into every article chapter point at
|
||||
`server.public_url`, so `daily.hallada.net` must resolve and serve TLS from the
|
||||
internet (e-readers tap these links). The XTC OPDS feed rides on the same host.
|
||||
internet (e-readers tap these links). The OPDS catalog rides on the same host.
|
||||
With an existing certificate, a minimal nginx site is:
|
||||
|
||||
```nginx
|
||||
@@ -266,23 +267,43 @@ sudo -u daily-epub node /opt/epub-to-xtc-converter/cli/index.js convert \
|
||||
> Running as `daily-epub` both avoids the trap and proves the *service* can read
|
||||
> everything it needs.
|
||||
|
||||
### How the XTC edition reaches the X4
|
||||
### The OPDS catalog
|
||||
|
||||
The `.xtch` file in `publish.xtc_dir` is **not** meant for BookOrbit — BookOrbit
|
||||
only watches the EPUB folder and wouldn't import the XTC binary format anyway.
|
||||
`daily-epub serve` publishes its own OPDS 1.2 feed for it: point the X4's
|
||||
CrossPoint OPDS browser at `https://daily.hallada.net/opds/xtc.xml` (the bare
|
||||
`https://daily.hallada.net/opds` works too) and it will list the last 14 issues,
|
||||
newest first, with the files served from `/files/xtc/`. (The X4 can also fall
|
||||
back to the "(X4)" EPUB via BookOrbit's own OPDS catalog.)
|
||||
`daily-epub serve` publishes its own OPDS 1.2 acquisition feed at
|
||||
**`https://daily.hallada.net/opds/daily.xml`** (the bare
|
||||
`https://daily.hallada.net/opds`, with or without a trailing slash, serves the
|
||||
same feed — one less thing to type on a seven-button keyboard). Point any OPDS
|
||||
client — the X4's CrossPoint browser, KOReader, Calibre — at it and you get the
|
||||
last 14 issues, newest first, **both editions of each**, with the standard
|
||||
edition listed first and the files served from `/files/epub/`.
|
||||
|
||||
`xtc.xml` is not an issue — it is a generated index, rewritten from scratch at
|
||||
the end of every run by scanning `publish.xtc_dir` for `.xtc`/`.xtch` files. The
|
||||
issues themselves are the dated files beside it; `/files/xtc/` is a route, not a
|
||||
directory on disk. So the feed always carries the back catalogue, capped at the
|
||||
last 14 days by `XTC_FEED_ENTRIES` and bounded by `retention_days` on disk. An
|
||||
empty `<feed>` with no `<entry>` elements means the converter never produced a
|
||||
file — check the run's warnings, not the server.
|
||||
The feed is not a file. It is rendered per request by scanning
|
||||
`publish.epub_dir` for `The Daily EPUB - YYYY-MM-DD*.epub`, so it cannot go
|
||||
stale behind a failed publish and there is no generated index for the retention
|
||||
sweep to step around. Entries are titled exactly like each EPUB's `dc:title`, so
|
||||
the two editions of one issue are distinguishable in the list.
|
||||
|
||||
This is deliberately independent of BookOrbit: it needs only the directory, so
|
||||
BookOrbit is optional, and it puts the day's issue one screen from the X4's home
|
||||
instead of several clicks down a library tree.
|
||||
|
||||
### Why XTC is not in the feed
|
||||
|
||||
XTC files are still generated and still land in `publish.xtc_dir` — they are just
|
||||
not advertised over OPDS, because **CrossPoint's OPDS browser can only acquire
|
||||
EPUBs**. Two independent reasons, both in the firmware:
|
||||
|
||||
- its OPDS parser marks an entry as a book only when an acquisition link's `type`
|
||||
is exactly `application/epub+zip` (a `strcmp`), and drops the entry otherwise —
|
||||
which surfaces as *"No entries found"*;
|
||||
- `OpdsBookBrowserActivity` hardcodes `.epub` as the saved filename regardless of
|
||||
the URL or `Content-Disposition`, and the reader dispatches on extension, so a
|
||||
downloaded `.xtch` would land under a name it then refuses to open.
|
||||
|
||||
XTC remains a first-class format *on the device* — the file browser lists
|
||||
`.xtc`/`.xtch` and there is a dedicated XTC reader — so the artifacts stay
|
||||
downloadable at `/files/xtc/<name>` (same Basic auth) for sideloading by SD card,
|
||||
WebDAV or the device's web upload UI.
|
||||
|
||||
**Budget the disk.** XTCH is a pre-rendered 2-bit page bitmap — 480×800 px is
|
||||
~96 KB per page regardless of what is on it — so an issue is large and its size
|
||||
@@ -294,8 +315,9 @@ tracks the page count, which `font.size` drives:
|
||||
| 30 (`xtc-settings.example.json`) | 820 | 79 MB | ~39 MB |
|
||||
|
||||
Measured on the 20-article issue of 2026-08-15; conversion took ~13 s either way.
|
||||
At `retention_days = 21` that is 1.7–2.2 GB in `publish.xtc_dir`, and it is also
|
||||
what the X4 downloads over WiFi per issue.
|
||||
That is why `xtc_dir` is swept by **count** rather than age: at the default
|
||||
`xtc_retention_count = 5` it holds ~0.4–0.5 GB, while the EPUBs beside it age out
|
||||
on the much longer `retention_days`.
|
||||
|
||||
---
|
||||
|
||||
@@ -329,9 +351,9 @@ ls -la /srv/bookorbit/libraries/daily-epub /var/lib/daily-epub/xtc
|
||||
|
||||
# 6. Delivery
|
||||
# KOReader (Kindle/Palma): browse BookOrbit's OPDS, download, read.
|
||||
# Xteink X4 / CrossPoint: OPDS → https://daily.hallada.net/opds/xtc.xml
|
||||
# Xteink X4 / CrossPoint: OPDS → https://daily.hallada.net/opds/daily.xml
|
||||
curl -s https://daily.hallada.net/healthz
|
||||
curl -s https://daily.hallada.net/opds/xtc.xml | head
|
||||
curl -s https://daily.hallada.net/opds/daily.xml | head
|
||||
curl -s https://daily.hallada.net/issues.json | jq '.[0]'
|
||||
|
||||
# 7. Feedback loop: tap 👍 in KOReader, then
|
||||
@@ -356,7 +378,8 @@ from what you see in step 8.
|
||||
| Run status `degraded` | a best-effort stage failed; the warnings are in the report (`/issues.json`, `runs.error`, the journal). |
|
||||
| No XTC file | `xtc.enabled = false`, Node missing, wrong `xtc.args` path, or `xtc.settings` unset/pointing at a font that does not exist. Non-fatal — the X4 can read the X4 EPUB from BookOrbit instead. The report warning quotes the converter's own error. |
|
||||
| `Font path is required` for a settings file that *does* set `font.path` | The process cannot read the file, and the converter cannot tell that apart from the file not existing. Almost always running the converter as yourself instead of `daily-epub` (see above), or a font path that has moved. `sudo -u daily-epub cat /etc/daily-epub/xtc-settings.json` and `sudo -u daily-epub test -r <font> && echo ok` settle it. |
|
||||
| The X4's OPDS browser says "Failed to fetch feed" | Usually an *empty* feed: `curl -s https://daily.hallada.net/opds/xtc.xml` and count the `<entry>` elements. Zero means no `.xtch` has ever been published — fix the converter, not the server. |
|
||||
| The X4's OPDS browser says "No entries found" | It fetched and parsed the feed but accepted no entry. Every acquisition link must be typed exactly `application/epub+zip`; anything else is dropped silently. `curl -s -u user:pass https://daily.hallada.net/opds/daily.xml \| grep -c "<entry>"` — zero means nothing has been published yet. |
|
||||
| The X4's OPDS browser says "Failed to fetch feed" | The request never completed: wrong URL, TLS, or credentials. "Failed to parse feed" means malformed XML. The three messages are distinct — read which one you got. |
|
||||
| No World Briefing | The portal page for the issue's own date is an empty stub until midday UTC, so the run falls back up to `world::MAX_LOOKBACK_DAYS` days. A warning means even those were empty or Wikipedia was unreachable. |
|
||||
|
||||
---
|
||||
|
||||
+8
-2
@@ -11,7 +11,8 @@ timezone = "America/New_York"
|
||||
lookback_hours = 26
|
||||
target_article_count = 20
|
||||
prefilter_keep = 120
|
||||
retention_days = 21
|
||||
retention_days = 21 # EPUBs, by age
|
||||
xtc_retention_count = 5 # XTC issues, by count (~80-100 MB each)
|
||||
max_daily_usd = 2.0
|
||||
world_briefing = true
|
||||
|
||||
@@ -59,7 +60,12 @@ sections = [
|
||||
]
|
||||
|
||||
[publish]
|
||||
bookorbit_dir = "/srv/bookorbit/libraries/daily-epub"
|
||||
# Where both EPUB editions land, and what the OPDS feed lists. BookOrbit is
|
||||
# optional — it just watches this folder if you run it.
|
||||
# (Renamed from `bookorbit_dir`; the old key is now a hard config error.)
|
||||
epub_dir = "/srv/bookorbit/libraries/daily-epub"
|
||||
# XTC artifacts. Not listed in OPDS (CrossPoint cannot acquire them); reachable
|
||||
# at /files/xtc/<name> for sideloading.
|
||||
xtc_dir = "/var/lib/daily-epub/xtc"
|
||||
|
||||
[xtc]
|
||||
|
||||
@@ -31,6 +31,25 @@ This file records implementation-time decisions and verified external facts. Fol
|
||||
- **Output size.** XTCH is a pre-rendered page bitmap: 480×800 at 2bpp = ~96 KB/page. A
|
||||
20-article issue rendered at `font.size = 34` came to 1,088 pages ≈ **104 MB**, in ~13 s.
|
||||
`retention_days = 21` therefore implies ~2 GB in `publish.xtc_dir`.
|
||||
- **CrossPoint's OPDS browser can only acquire EPUBs.** Verified against the
|
||||
`yokki-vans/InkPointX` sources (an open fork of CrossPoint/CrossInk). Two independent
|
||||
blockers, either one fatal for serving XTC over OPDS:
|
||||
1. `lib/OpdsParser/OpdsParser.cpp` sets an entry's `href` only for an acquisition link
|
||||
whose `type` is **exactly** `application/epub+zip` (`strcmp`), or for a navigation
|
||||
link (`application/atom+xml`). `endElement` then drops any entry with an empty
|
||||
`href`. An `application/octet-stream` acquisition link therefore yields an empty
|
||||
list and the UI reports `STR_NO_ENTRIES` — **"No entries found"**.
|
||||
2. `OpdsBookBrowserActivity::downloadBook` builds the destination filename as
|
||||
`sanitizeFilename(author + " - " + title) + ".epub"` — hardcoded, ignoring the URL
|
||||
and `Content-Disposition` — and `ReaderActivity` dispatches on extension only
|
||||
(`FsHelpers::hasXtcExtension` → `.xtc`/`.xtch`, no magic-byte sniffing). So even a
|
||||
mistyped XTC link downloads into a file the device will not open.
|
||||
The three OPDS failure strings are distinct and worth reading precisely:
|
||||
`STR_FETCH_FEED_FAILED` ("Failed to fetch feed"), `STR_PARSE_FEED_FAILED`
|
||||
("Failed to parse feed") and `STR_NO_ENTRIES` ("No entries found") — only the last is
|
||||
reachable after a *successful* fetch **and** parse.
|
||||
Consequence: the built-in feed serves EPUBs (both editions), XTC is published but not
|
||||
advertised, and `publish.xtc_dir` is swept by count. See the amendment in spec §3.11.
|
||||
- **X4 firmware rendering limits** (from the `epub-to-xtc-converter` optimizer's header, which
|
||||
cites papyrix-reader): 464×788 usable viewport, max image decode 2048×3072, **baseline JPEG
|
||||
only**, no GIF/SVG/WebP, **max 1500 CSS rules and simple selectors only** (`tag`, `.class`,
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
3. Applies cheap heuristic pre-filters, then uses **DeepSeek V4 Flash** to score, select, and organize ~15–25 articles into newspaper sections.
|
||||
4. Generates editorial framing: a front-page "day in brief," section intros, and per-article summaries.
|
||||
5. Builds two EPUB editions (standard + Xteink X4-optimized), converts the X4 edition to XTC/XTCH.
|
||||
6. Publishes into a dedicated **BookOrbit** library via watched folder (→ OPDS for KOReader devices) and serves XTC via a minimal built-in OPDS feed.
|
||||
6. Publishes both EPUB editions into a folder that its own built-in OPDS feed serves (and that **BookOrbit** can optionally watch for a richer library UI). *(Amended: the built-in feed serves EPUBs, not XTC — see §3.11.)*
|
||||
7. Collects 👍/👎 feedback via rating links inside the EPUB to continuously improve curation.
|
||||
|
||||
**Reader profile (bake into curation prompts):** prefers long-form, high-effort, well-written articles on *any* topic; uses social proof (HN/Reddit upvotes+comments) as a quality proxy; wants tech news, light general/US world news (prefers Wikipedia Current Events for world news), Boston-area news, and ultra-niche community news. The full interest list lives in `data/scour-interests.opml` (~220 Scour interests: Rust, systems programming, e-ink, self-hosting, PKM, sci-fi, creative coding, space, running, board games, Boston Tech, etc.) — compile it into the taste profile at build time.
|
||||
@@ -21,7 +21,7 @@
|
||||
|---|---|---|---|
|
||||
| Miniflux | `127.0.0.1:8082` | `miniflux.hallada.net` | Installed via PPA. API auth via `X-Auth-Token` header. |
|
||||
| BookOrbit | `127.0.0.1:3498` | `bookorbit.hallada.net` | NestJS/Vue/Postgres. Supports multiple isolated libraries, per-library watched folders, OPDS at `/api/v1/opds` (Basic auth, `opds_access` permission). |
|
||||
| The Daily EPUB (new) | `127.0.0.1:<port, e.g. 3499>` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + XTC OPDS + static files. |
|
||||
| The Daily EPUB (new) | `127.0.0.1:<port, e.g. 3499>` | `daily.hallada.net` (reverse proxy to be added) | Rating endpoints + OPDS catalog + downloads. |
|
||||
|
||||
### Secrets/config the operator must provide
|
||||
|
||||
@@ -46,7 +46,7 @@ Single Rust binary crate `daily-epub` (workspace not needed yet) with clap subco
|
||||
|
||||
```
|
||||
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
|
||||
daily-epub serve # long-running: rating endpoints + XTC OPDS + static
|
||||
daily-epub serve # long-running: rating endpoints + OPDS catalog + downloads
|
||||
daily-epub profile rebuild # regenerate taste profile from ratings (also runs weekly inside generate)
|
||||
daily-epub backfill-social # re-poll social scores for recent entries (optional helper)
|
||||
daily-epub db migrate # run sqlx migrations (also auto-run on start)
|
||||
@@ -60,7 +60,7 @@ daily-epub db migrate # run sqlx migrations (also auto-run on start)
|
||||
Miniflux ingest → normalize/dedupe → content extraction → social enrichment
|
||||
→ heuristic pre-filter (500 → ~120) → LLM scoring (batched) → LLM selection (~120 → 15–25)
|
||||
→ comment fetching for selected → LLM editorial (summaries, section intros, front page)
|
||||
→ EPUB build (standard + X4 editions) → XTC conversion → publish (BookOrbit folder, XTC dir, OPDS xml)
|
||||
→ EPUB build (standard + X4 editions) → XTC conversion → publish (EPUB dir, XTC dir) → OPDS feed served from the EPUB dir
|
||||
→ retention pruning → run report logged + stored
|
||||
```
|
||||
|
||||
@@ -103,8 +103,8 @@ the-daily-epub/
|
||||
│ │ ├── templates/ # askama XHTML templates + CSS
|
||||
│ │ ├── images.rs # download, resize, grayscale, re-encode
|
||||
│ │ └── x4.rs # X4 edition transforms + XTC CLI invocation
|
||||
│ ├── publish.rs # copy to BookOrbit folder, OPDS xml gen, retention
|
||||
│ ├── server.rs # axum: /r/… ratings, /opds/xtc.xml, /files/…
|
||||
│ ├── publish.rs # copy to the publish dirs, OPDS feed rendering, retention
|
||||
│ ├── server.rs # axum: /r/… ratings, /opds/daily.xml, /files/…
|
||||
│ └── report.rs # run summary (counts, cost, timings)
|
||||
└── tests/ # integration tests with fixture JSON
|
||||
```
|
||||
@@ -248,15 +248,40 @@ TOC: nav depth 2 (sections → articles, discussions nested). Metadata: `dc:titl
|
||||
### 3.11 XTC conversion & publishing (`publish.rs`)
|
||||
|
||||
- Run the `epub-to-xtc-converter` CLI (Node 18+) on the X4 edition: invoke via `tokio::process::Command`, config keys `xtc.command` (default `epub-to-xtc`) and `xtc.args` (verify exact CLI name/flags from the repo README at implementation time; support both `.xtc` 1-bit and `.xtch` 4-level grayscale via config, default XTCH for image quality). Non-zero exit → log error, continue (XTC is a bonus artifact).
|
||||
- **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.bookorbit_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) — KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered.
|
||||
- **XTC delivery:** copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead.
|
||||
- **Publish standard + X4 EPUBs** by atomic copy (`write temp + rename`) into the BookOrbit "The Daily EPUB" library watched folder (`publish.epub_dir`), filenames `The Daily EPUB - 2026-08-15.epub` and `The Daily EPUB - 2026-08-15 (X4).epub`. BookOrbit's watcher auto-imports; the library appears as its own section in BookOrbit's OPDS catalog (`/api/v1/opds`, Basic auth with an OPDS account) — KOReader on Kindle/Palma and CrossPoint on the X4 browse that. Main library stays uncluttered.
|
||||
- **XTC delivery** *(superseded — see the amendment below; kept for the record)*: copy `.xtch/.xtc` into `publish.xtc_dir`; regenerate a static **OPDS 1.2 acquisition feed** (`xtc.xml`, entries typed `application/octet-stream`, newest first, last 14) served by `daily-epub serve` at `/opds/xtc.xml` with files under `/files/xtc/` (optional Basic auth from config). CrossPoint's OPDS browser can fetch these; worst case the X4 uses the X4 EPUB from BookOrbit instead.
|
||||
- **Retention:** delete issue files older than `retention_days` (default 21) from both dirs (BookOrbit's scan removes the DB entries); SQLite issue/rating history is kept forever (it's the training data).
|
||||
|
||||
> **Amended 2026-08-15 (post-M8), after testing against a real X4.** The XTC
|
||||
> delivery bullet above does not work and has been replaced. CrossPoint's OPDS
|
||||
> browser cannot acquire XTC at all — see the implementation notes' "Verified
|
||||
> external facts" for the two firmware reasons — so:
|
||||
>
|
||||
> - The built-in feed lists **EPUBs, both editions**, from `publish.epub_dir`,
|
||||
> with every acquisition link typed exactly `application/epub+zip` and files
|
||||
> served from `/files/epub/{name}`. Canonical path `/opds/daily.xml`, with
|
||||
> `/opds` and `/opds/` as aliases.
|
||||
> - The feed is **rendered per request** from the directory rather than written to
|
||||
> disk, so it cannot go stale behind a failed publish and the retention sweep has
|
||||
> no generated index to step around.
|
||||
> - This makes BookOrbit **optional**: the feed needs only the folder, and it puts
|
||||
> the day's issue one screen from the X4's home rather than several clicks down a
|
||||
> library tree.
|
||||
> - XTC is still generated and still published to `publish.xtc_dir`, just not
|
||||
> advertised. It stays fetchable at `/files/xtc/{name}` for sideloading.
|
||||
> - `publish.xtc_dir` is swept by **count** (`xtc_retention_count`, default 5)
|
||||
> rather than by age: an XTCH issue measured 79–104 MB, so disk is the binding
|
||||
> constraint. EPUBs keep the dated `retention_days` sweep.
|
||||
> - `publish.bookorbit_dir` was renamed **`publish.epub_dir`** to match: the feed
|
||||
> needs the directory, not BookOrbit. The old key is rejected outright
|
||||
> (`deny_unknown_fields`) rather than silently falling back to the default,
|
||||
> which would publish into a directory the feed does not read.
|
||||
|
||||
### 3.12 Server (`server.rs`)
|
||||
|
||||
axum on `127.0.0.1:3499`:
|
||||
- `GET /r/{date}/{article_id}/{vote}?t=` — verify HMAC, upsert rating, tiny HTML response. No auth beyond the token (links live inside a private EPUB; tokens are per-article+vote and unguessable).
|
||||
- `GET /opds/xtc.xml`, `GET /files/xtc/{name}` — optional Basic auth.
|
||||
- `GET /opds/daily.xml` (aliases `/opds`, `/opds/`), `GET /files/epub/{name}`, `GET /files/xtc/{name}` — optional Basic auth. *(Amended: was `/opds/xtc.xml` + `/files/xtc/` only; see §3.11.)*
|
||||
- `GET /healthz`, `GET /issues.json` (recent run reports; handy for debugging).
|
||||
- `tower-http` request tracing; graceful shutdown on SIGTERM.
|
||||
|
||||
@@ -317,7 +342,7 @@ sections = ["Top Stories", "Tech & Engineering", "Science & Space",
|
||||
"Niche Corner", "From the Blogroll"]
|
||||
|
||||
[publish]
|
||||
bookorbit_dir = "/srv/bookorbit/libraries/daily-epub"
|
||||
epub_dir = "/srv/bookorbit/libraries/daily-epub"
|
||||
xtc_dir = "/var/lib/daily-epub/xtc"
|
||||
|
||||
[xtc]
|
||||
@@ -346,7 +371,7 @@ public_url = "https://daily.hallada.net"
|
||||
3. **M3 — Pre-filter + LLM scoring/selection:** end-to-end lineup JSON printed in dry-run; token/cost report. *Verify: lineup is sane; cost < $0.50.*
|
||||
4. **M4 — EPUB standard edition + publish:** full issue EPUB with cover, front page (temporary plain summaries), sections, articles, images; lands in BookOrbit, visible via OPDS on Kindle. *Verify: epubcheck clean; opens in KOReader with working TOC.*
|
||||
5. **M5 — Editorial + comments:** DeepSeek summaries/intros/front page wired in; discussion chapters. *Verify: read an issue; comments legible on e-ink.*
|
||||
6. **M6 — X4 edition + XTC + XTC OPDS:** second edition, converter invocation, static OPDS feed. *Verify: X4 fetches and renders both.*
|
||||
6. **M6 — X4 edition + XTC + OPDS:** second edition, converter invocation, OPDS feed. *Verify: X4 fetches and renders both.*
|
||||
7. **M7 — Feedback loop:** `serve` rating endpoints, links in chapters, feed priors in pre-filter, weekly profile rebuild. *Verify: tap 👍 in KOReader → row in `ratings` → prior changes next run.*
|
||||
8. **M8 — Hardening & ops:** systemd units, retention, cost guardrail, run reports, `issues.json`, README.
|
||||
|
||||
@@ -354,7 +379,7 @@ public_url = "https://daily.hallada.net"
|
||||
|
||||
- `cargo test` — unit tests: URL canonicalization, dedupe clustering, HMAC round-trip, comment-tree truncation, prefilter scoring; integration tests over fixture JSON (recorded Miniflux/Algolia/Reddit responses) with the LLM stage mocked (`--skip-llm` uses prefilter order).
|
||||
- `daily-epub generate --dry-run --out ./out --max-articles 6` with real keys → inspect `./out/*.epub` in Calibre + run `epubcheck` (if installed) → zero errors.
|
||||
- Full live run: `daily-epub generate` → file appears in BookOrbit UI under the Daily EPUB library only → browse BookOrbit OPDS from KOReader (Kindle/Palma), download, read; X4: CrossPoint OPDS → both the X4 EPUB (via BookOrbit) and XTC (via `daily.hallada.net/opds/xtc.xml`).
|
||||
- Full live run: `daily-epub generate` → both editions appear in the publish dir (and in BookOrbit's UI if it is running) → browse `daily.hallada.net/opds/daily.xml` from KOReader (Kindle/Palma) or the X4's CrossPoint, download, read.
|
||||
- Tap a rating link on the Kindle → confirmation page loads → `sqlite3 … 'select * from ratings'` shows the vote.
|
||||
- Watch `runs` for a week: cost per day, selection quality; tune `prefilter_keep`/prompts.
|
||||
|
||||
|
||||
+36
-6
@@ -51,8 +51,13 @@ pub struct Config {
|
||||
pub target_article_count: usize,
|
||||
/// How many articles survive the heuristic pre-filter (§3.5).
|
||||
pub prefilter_keep: usize,
|
||||
/// Days of published files kept in the publish dirs (§3.11).
|
||||
/// Days of published EPUBs kept in `publish.epub_dir` (§3.11).
|
||||
pub retention_days: u32,
|
||||
/// How many XTC issues to keep in `publish.xtc_dir` (§3.11).
|
||||
///
|
||||
/// Counted, not dated, because an XTCH issue is ~80–100 MB of pre-rendered
|
||||
/// page bitmaps: the constraint is disk, not age.
|
||||
pub xtc_retention_count: u32,
|
||||
/// Hard cost ceiling per run (§3.6 guardrail).
|
||||
pub max_daily_usd: f64,
|
||||
/// Include the Wikipedia Current Events section (§3.8).
|
||||
@@ -81,6 +86,7 @@ impl Default for Config {
|
||||
target_article_count: 20,
|
||||
prefilter_keep: 120,
|
||||
retention_days: 21,
|
||||
xtc_retention_count: 5,
|
||||
max_daily_usd: 2.0,
|
||||
world_briefing: true,
|
||||
database_path: PathBuf::from("/var/lib/daily-epub/daily-epub.db"),
|
||||
@@ -195,16 +201,20 @@ impl Default for CurationConfig {
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields, default)]
|
||||
pub struct PublishConfig {
|
||||
/// BookOrbit "The Daily EPUB" library watched folder.
|
||||
pub bookorbit_dir: PathBuf,
|
||||
/// Directory served at `/files/xtc/`.
|
||||
/// Where both EPUB editions land: the source of the OPDS feed, served at
|
||||
/// `/files/epub/`, and a BookOrbit watched folder if one is configured.
|
||||
///
|
||||
/// Renamed from `bookorbit_dir` once the built-in feed started serving this
|
||||
/// directory directly — BookOrbit is optional, the directory is not.
|
||||
pub epub_dir: PathBuf,
|
||||
/// Directory served at `/files/xtc/`. Not listed in the OPDS feed (§3.11).
|
||||
pub xtc_dir: PathBuf,
|
||||
}
|
||||
|
||||
impl Default for PublishConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
bookorbit_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"),
|
||||
epub_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"),
|
||||
xtc_dir: PathBuf::from("/var/lib/daily-epub/xtc"),
|
||||
}
|
||||
}
|
||||
@@ -274,7 +284,7 @@ pub struct ServerConfig {
|
||||
pub public_url: String,
|
||||
/// HMAC key for rating tokens; supply via `DAILY_EPUB_SERVER__HMAC_SECRET`.
|
||||
pub hmac_secret: Option<String>,
|
||||
/// Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`.
|
||||
/// Optional Basic auth for `/opds/*` and `/files/*`.
|
||||
pub basic_auth_user: Option<String>,
|
||||
pub basic_auth_pass: Option<String>,
|
||||
}
|
||||
@@ -431,6 +441,26 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
/// `publish.bookorbit_dir` was renamed to `publish.epub_dir`. A config still
|
||||
/// using the old key must fail loudly and name both — silently falling back
|
||||
/// to the default would publish the issue into the wrong directory, where
|
||||
/// the OPDS feed would then find nothing.
|
||||
#[test]
|
||||
fn the_renamed_publish_key_fails_loudly() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("config.toml");
|
||||
std::fs::write(
|
||||
&path,
|
||||
"[publish]\nbookorbit_dir = \"/srv/books\"\nxtc_dir = \"/srv/xtc\"\n",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let err = Config::load(Some(&path)).expect_err("the stale key must be rejected");
|
||||
let message = err.to_string();
|
||||
assert!(message.contains("bookorbit_dir"), "{message}");
|
||||
assert!(message.contains("epub_dir"), "{message}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shipped_example_config_parses() {
|
||||
let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml");
|
||||
|
||||
+1
-4
@@ -31,7 +31,7 @@ struct Cli {
|
||||
enum Command {
|
||||
/// Build (and publish) one issue.
|
||||
Generate(GenerateArgs),
|
||||
/// Run the rating endpoints, XTC OPDS feed and static files.
|
||||
/// Run the rating endpoints, the OPDS catalog and downloads.
|
||||
Serve,
|
||||
/// Taste-profile maintenance.
|
||||
#[command(subcommand)]
|
||||
@@ -167,9 +167,6 @@ fn print_outcome(outcome: &GenerateOutcome) {
|
||||
if let Some(xtc) = &published.xtc {
|
||||
println!("published: {}", xtc.display());
|
||||
}
|
||||
if let Some(opds) = &published.opds {
|
||||
println!("opds: {}", opds.display());
|
||||
}
|
||||
if published.pruned > 0 {
|
||||
println!("pruned: {} expired files", published.pruned);
|
||||
}
|
||||
|
||||
+1
-1
@@ -510,7 +510,7 @@ async fn run_stages(
|
||||
);
|
||||
None
|
||||
} else {
|
||||
let published = publish::publish_issue(db, config, &issue, &artifacts, xtc.as_deref())
|
||||
let published = publish::publish_issue(config, &issue, &artifacts, xtc.as_deref())
|
||||
.await
|
||||
.context("publishing the issue")?;
|
||||
record_issue(db, &issue, &published)
|
||||
|
||||
+299
-153
@@ -1,6 +1,9 @@
|
||||
//! Publishing: BookOrbit watched folder, XTC delivery, OPDS feed, retention
|
||||
//! Publishing: the EPUB library folder, XTC delivery, OPDS feed, retention
|
||||
//! (spec §3.11).
|
||||
//!
|
||||
//! `publish.epub_dir` holds both EPUB editions and is what the OPDS feed lists;
|
||||
//! BookOrbit may watch the same folder but nothing here depends on it.
|
||||
//!
|
||||
//! Everything here is deliberately dumb about *how* artifacts were produced: the
|
||||
//! EPUB/XTC stages hand over finished files, this module only copies, indexes and
|
||||
//! prunes them. Copies are atomic (temp file in the destination directory, then
|
||||
@@ -8,11 +11,15 @@
|
||||
//! half-written book.
|
||||
//!
|
||||
//! [`crate::pipeline`] ends a non-dry run with one call —
|
||||
//! `publish_issue(db, config, &issue, &artifacts, xtc_path.as_deref())`, where
|
||||
//! `publish_issue(config, &issue, &artifacts, xtc_path.as_deref())`, where
|
||||
//! `artifacts` are the `epub::build_all` outputs and `xtc_path` is
|
||||
//! `epub::x4::convert`'s output (`None` when the converter is disabled or
|
||||
//! failed) — and feeds the returned [`Published`] paths into
|
||||
//! `db.upsert_issue(..., epub_path, x4_path, xtc_path, ...)`.
|
||||
//!
|
||||
//! The OPDS feed ([`build_opds`]) is rendered per request by
|
||||
//! [`crate::server`] rather than written here, and lists **EPUBs only** — see
|
||||
//! its docs for why XTC cannot be delivered over OPDS.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
@@ -26,14 +33,19 @@ use crate::config::Config;
|
||||
use crate::db::Db;
|
||||
use crate::types::{Artifact, Edition, Issue};
|
||||
|
||||
/// Filename of the generated static OPDS feed (§3.11).
|
||||
pub const XTC_OPDS_FILENAME: &str = "xtc.xml";
|
||||
/// Number of issues listed in the XTC OPDS feed (§3.11).
|
||||
pub const XTC_FEED_ENTRIES: usize = 14;
|
||||
/// Number of issue *days* listed in the OPDS feed; each contributes one entry
|
||||
/// per edition (§3.11).
|
||||
pub const OPDS_FEED_ISSUES: usize = 14;
|
||||
/// Every published file starts with this (the retention sweep keys off it).
|
||||
pub const FILE_PREFIX: &str = "The Daily EPUB - ";
|
||||
/// Extensions the retention sweep is allowed to delete (§3.11).
|
||||
pub const PRUNABLE_EXTENSIONS: [&str; 3] = ["epub", "xtc", "xtch"];
|
||||
/// Extensions of the XTC artifacts, for the counted sweep of `xtc_dir` (§3.11).
|
||||
pub const XTC_EXTENSIONS: [&str; 2] = ["xtc", "xtch"];
|
||||
/// Canonical path of the OPDS feed, relative to `server.public_url` (§3.11).
|
||||
pub const OPDS_PATH: &str = "/opds/daily.xml";
|
||||
/// The one acquisition type CrossPoint's OPDS parser accepts (§3.11).
|
||||
pub const EPUB_CONTENT_TYPE: &str = "application/epub+zip";
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum PublishError {
|
||||
@@ -61,8 +73,6 @@ pub struct Published {
|
||||
pub epubs: Vec<Artifact>,
|
||||
/// The XTC artifact's published location, when the converter produced one.
|
||||
pub xtc: Option<PathBuf>,
|
||||
/// The regenerated OPDS feed.
|
||||
pub opds: Option<PathBuf>,
|
||||
/// How many expired files the retention sweep removed.
|
||||
pub pruned: usize,
|
||||
}
|
||||
@@ -136,7 +146,7 @@ async fn ensure_dir(dir: &Path) -> Result<(), PublishError> {
|
||||
.map_err(PublishError::at(dir))
|
||||
}
|
||||
|
||||
/// Copy both EPUB editions into the BookOrbit watched folder (§3.11).
|
||||
/// Copy both EPUB editions into `publish.epub_dir` (§3.11).
|
||||
///
|
||||
/// Returns the published paths in the same order as `artifacts`.
|
||||
pub async fn publish_epubs(
|
||||
@@ -144,7 +154,7 @@ pub async fn publish_epubs(
|
||||
issue: &Issue,
|
||||
cfg: &Config,
|
||||
) -> Result<Vec<PathBuf>, PublishError> {
|
||||
let dir = &cfg.publish.bookorbit_dir;
|
||||
let dir = &cfg.publish.epub_dir;
|
||||
ensure_dir(dir).await?;
|
||||
let mut published = Vec::with_capacity(artifacts.len());
|
||||
for artifact in artifacts {
|
||||
@@ -159,7 +169,7 @@ pub async fn publish_epubs(
|
||||
edition = ?artifact.edition,
|
||||
dest = %dest.display(),
|
||||
bytes = artifact.bytes,
|
||||
"published edition to the BookOrbit library"
|
||||
"published edition to the EPUB library"
|
||||
);
|
||||
published.push(dest);
|
||||
}
|
||||
@@ -180,12 +190,11 @@ pub async fn publish_xtc(xtc: &Path, cfg: &Config) -> Result<PathBuf, PublishErr
|
||||
Ok(dest)
|
||||
}
|
||||
|
||||
/// Publish everything one run produced, refresh the OPDS feed and prune (§3.11).
|
||||
/// Publish everything one run produced and prune (§3.11).
|
||||
///
|
||||
/// `xtc` is `None` when the converter is disabled or failed — that is not an
|
||||
/// error, the X4 falls back to the EPUB edition from BookOrbit.
|
||||
/// error, since the OPDS feed lists the EPUB editions either way.
|
||||
pub async fn publish_issue(
|
||||
db: &Db,
|
||||
cfg: &Config,
|
||||
issue: &Issue,
|
||||
artifacts: &[Artifact],
|
||||
@@ -208,67 +217,74 @@ pub async fn publish_issue(
|
||||
Some(src) => Some(publish_xtc(src, cfg).await?),
|
||||
None => None,
|
||||
};
|
||||
let opds = Some(write_xtc_opds(db, cfg).await?);
|
||||
// The OPDS feed is rendered per request from the publish directory, so
|
||||
// there is nothing to write here (§3.11).
|
||||
let pruned = prune(cfg, issue.meta.date).await?;
|
||||
|
||||
Ok(Published {
|
||||
epubs,
|
||||
xtc,
|
||||
opds,
|
||||
pruned,
|
||||
})
|
||||
Ok(Published { epubs, xtc, pruned })
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// OPDS 1.2 acquisition feed (§3.11)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// One published XTC file, as listed in the feed.
|
||||
/// One published EPUB, as listed in the feed.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
struct XtcFile {
|
||||
struct EpubFile {
|
||||
name: String,
|
||||
date: Option<Date>,
|
||||
edition: Edition,
|
||||
modified: Timestamp,
|
||||
bytes: u64,
|
||||
}
|
||||
|
||||
/// Regenerate the static OPDS 1.2 acquisition feed for the XTC directory:
|
||||
/// newest first, last [`XTC_FEED_ENTRIES`], entries typed
|
||||
/// `application/octet-stream` (§3.11).
|
||||
pub async fn write_xtc_opds(db: &Db, cfg: &Config) -> Result<PathBuf, PublishError> {
|
||||
let dir = &cfg.publish.xtc_dir;
|
||||
/// Render the OPDS 1.2 acquisition feed for the EPUB publish directory (§3.11).
|
||||
///
|
||||
/// Rendered per request rather than written to disk: the directory is the only
|
||||
/// source of truth, so the feed cannot go stale behind a failed publish, and
|
||||
/// there is no generated file for the retention sweep to step around.
|
||||
///
|
||||
/// XTC artifacts are deliberately **not** listed. CrossPoint's OPDS browser only
|
||||
/// acquires links typed `application/epub+zip` and always saves the result with
|
||||
/// a `.epub` extension, which its reader dispatches on — so an XTC offered here
|
||||
/// would either be invisible or download into a file that cannot be opened.
|
||||
pub async fn build_opds(db: &Db, cfg: &Config) -> Result<String, PublishError> {
|
||||
let dir = &cfg.publish.epub_dir;
|
||||
ensure_dir(dir).await?;
|
||||
let files = scan_xtc_dir(dir).await?;
|
||||
let files = scan_epub_dir(dir).await?;
|
||||
let numbers = issue_numbers(db, &files).await;
|
||||
let feed = render_opds(&files, &numbers, &cfg.server.public_url, Timestamp::now());
|
||||
|
||||
let dest = dir.join(XTC_OPDS_FILENAME);
|
||||
write_atomic(&dest, feed.as_bytes()).await?;
|
||||
tracing::info!(entries = files.len(), dest = %dest.display(), "wrote the XTC OPDS feed");
|
||||
Ok(dest)
|
||||
Ok(render_opds(
|
||||
&files,
|
||||
&numbers,
|
||||
&cfg.server.public_url,
|
||||
Timestamp::now(),
|
||||
))
|
||||
}
|
||||
|
||||
/// XTC artifacts in `dir`, newest first, capped at [`XTC_FEED_ENTRIES`].
|
||||
async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
|
||||
/// Published EPUBs in `dir`, newest first, limited to the last
|
||||
/// [`OPDS_FEED_ISSUES`] issue days (both editions of each).
|
||||
async fn scan_epub_dir(dir: &Path) -> Result<Vec<EpubFile>, PublishError> {
|
||||
let mut entries = tokio::fs::read_dir(dir)
|
||||
.await
|
||||
.map_err(PublishError::at(dir))?;
|
||||
let mut files = Vec::new();
|
||||
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
let extension = Path::new(&name)
|
||||
let path = Path::new(&name);
|
||||
let extension = path
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.unwrap_or_default()
|
||||
.to_ascii_lowercase();
|
||||
if !matches!(extension.as_str(), "xtc" | "xtch") {
|
||||
// Only our own issues: a shared library may hold other people's books.
|
||||
if extension != "epub" || date_from_filename(&name).is_none() {
|
||||
continue;
|
||||
}
|
||||
let meta = match entry.metadata().await {
|
||||
Ok(meta) if meta.is_file() => meta,
|
||||
Ok(_) => continue,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, name, "skipping unreadable XTC file");
|
||||
tracing::warn!(error = %e, name, "skipping unreadable EPUB");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
@@ -277,29 +293,47 @@ async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
|
||||
.ok()
|
||||
.and_then(|m| Timestamp::try_from(m).ok())
|
||||
.unwrap_or_else(Timestamp::now);
|
||||
files.push(XtcFile {
|
||||
let stem = path
|
||||
.file_stem()
|
||||
.and_then(|s| s.to_str())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
files.push(EpubFile {
|
||||
date: date_from_filename(&name),
|
||||
edition: Edition::from_file_stem(&stem),
|
||||
name,
|
||||
modified,
|
||||
bytes: meta.len(),
|
||||
});
|
||||
}
|
||||
// Newest first: by issue date when the filename carries one, else by mtime.
|
||||
// Newest issue first, and within an issue the standard edition leads.
|
||||
files.sort_by(|a, b| {
|
||||
b.date
|
||||
.cmp(&a.date)
|
||||
.then(a.edition.cmp(&b.edition))
|
||||
.then(b.modified.cmp(&a.modified))
|
||||
.then(a.name.cmp(&b.name))
|
||||
});
|
||||
files.truncate(XTC_FEED_ENTRIES);
|
||||
// Cap by issue day, not by file: an issue is two entries and truncating
|
||||
// mid-issue would list one edition without the other.
|
||||
let mut kept_dates: Vec<Option<Date>> = Vec::new();
|
||||
files.retain(|file| {
|
||||
if !kept_dates.contains(&file.date) {
|
||||
kept_dates.push(file.date);
|
||||
}
|
||||
kept_dates.iter().position(|d| *d == file.date) < Some(OPDS_FEED_ISSUES)
|
||||
});
|
||||
Ok(files)
|
||||
}
|
||||
|
||||
/// Issue numbers for the dated files, best-effort (the feed is still valid
|
||||
/// without them). Uses the `db` escape hatch — no bespoke helper in `db.rs`.
|
||||
async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
|
||||
async fn issue_numbers(db: &Db, files: &[EpubFile]) -> BTreeMap<Date, i64> {
|
||||
let mut numbers = BTreeMap::new();
|
||||
for date in files.iter().filter_map(|f| f.date) {
|
||||
if numbers.contains_key(&date) {
|
||||
continue;
|
||||
}
|
||||
let row = sqlx::query("SELECT issue_number FROM issues WHERE date = ?")
|
||||
.bind(date.to_string())
|
||||
.fetch_optional(db.pool())
|
||||
@@ -317,13 +351,13 @@ async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
|
||||
|
||||
/// Render the Atom/OPDS document (§3.11).
|
||||
fn render_opds(
|
||||
files: &[XtcFile],
|
||||
files: &[EpubFile],
|
||||
numbers: &BTreeMap<Date, i64>,
|
||||
public_url: &str,
|
||||
now: Timestamp,
|
||||
) -> String {
|
||||
let base = public_url.trim_end_matches('/');
|
||||
let self_href = format!("{base}/opds/xtc.xml");
|
||||
let self_href = format!("{base}{OPDS_PATH}");
|
||||
let updated = files.first().map(|f| f.modified).unwrap_or(now);
|
||||
|
||||
let mut out = String::with_capacity(1024 + files.len() * 512);
|
||||
@@ -333,8 +367,8 @@ fn render_opds(
|
||||
xmlns:dc=\"http://purl.org/dc/terms/\" \
|
||||
xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
|
||||
);
|
||||
out.push_str(" <id>urn:daily-epub:xtc</id>\n");
|
||||
out.push_str(" <title>The Daily EPUB — XTC editions</title>\n");
|
||||
out.push_str(" <id>urn:daily-epub:issues</id>\n");
|
||||
out.push_str(" <title>The Daily EPUB</title>\n");
|
||||
out.push_str(&format!(" <updated>{}</updated>\n", rfc3339(updated)));
|
||||
out.push_str(" <author><name>The Daily EPUB</name></author>\n");
|
||||
out.push_str(&format!(
|
||||
@@ -349,19 +383,25 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
|
||||
));
|
||||
|
||||
for file in files {
|
||||
// Matches the EPUB's own `dc:title`, so the two editions of one issue
|
||||
// are told apart in the list rather than showing as the same book.
|
||||
let title = match file.date {
|
||||
Some(date) => format!("The Daily EPUB — {date}"),
|
||||
Some(date) => format!("The Daily EPUB — {date}{}", file.edition.file_suffix()),
|
||||
None => file.name.clone(),
|
||||
};
|
||||
let summary = match file.date.and_then(|d| numbers.get(&d)) {
|
||||
Some(n) => format!("Issue #{n} · {}", human_bytes(file.bytes)),
|
||||
None => human_bytes(file.bytes),
|
||||
let edition_note = match file.edition {
|
||||
Edition::Standard => "Standard",
|
||||
Edition::X4 => "Xteink X4",
|
||||
};
|
||||
let href = format!("{base}/files/xtc/{}", percent_encode(&file.name));
|
||||
let summary = match file.date.and_then(|d| numbers.get(&d)) {
|
||||
Some(n) => format!("Issue #{n} · {edition_note} · {}", human_bytes(file.bytes)),
|
||||
None => format!("{edition_note} · {}", human_bytes(file.bytes)),
|
||||
};
|
||||
let href = format!("{base}/files/epub/{}", percent_encode(&file.name));
|
||||
out.push_str(" <entry>\n");
|
||||
out.push_str(&format!(" <title>{}</title>\n", xml_escape(&title)));
|
||||
out.push_str(&format!(
|
||||
" <id>urn:daily-epub:xtc:{}</id>\n",
|
||||
" <id>urn:daily-epub:issue:{}</id>\n",
|
||||
xml_escape(&percent_encode(&file.name))
|
||||
));
|
||||
out.push_str(&format!(
|
||||
@@ -376,9 +416,12 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
|
||||
" <summary>{}</summary>\n",
|
||||
xml_escape(&summary)
|
||||
));
|
||||
// The type must be exactly `application/epub+zip`: CrossPoint's OPDS
|
||||
// parser compares it with `strcmp` and silently drops entries whose
|
||||
// acquisition link is anything else, reporting "No entries found".
|
||||
out.push_str(&format!(
|
||||
" <link rel=\"http://opds-spec.org/acquisition\" href=\"{}\" \
|
||||
type=\"application/octet-stream\" length=\"{}\"/>\n",
|
||||
type=\"{EPUB_CONTENT_TYPE}\" length=\"{}\"/>\n",
|
||||
xml_escape(&href),
|
||||
file.bytes
|
||||
));
|
||||
@@ -430,43 +473,82 @@ fn percent_encode(s: &str) -> String {
|
||||
out
|
||||
}
|
||||
|
||||
async fn write_atomic(dest: &Path, bytes: &[u8]) -> Result<(), PublishError> {
|
||||
let dir = dest.parent().unwrap_or_else(|| Path::new("."));
|
||||
let tmp = dir.join(format!(
|
||||
".{}.{}.tmp",
|
||||
dest.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
.unwrap_or("daily-epub"),
|
||||
std::process::id()
|
||||
));
|
||||
tokio::fs::write(&tmp, bytes)
|
||||
.await
|
||||
.map_err(PublishError::at(&tmp))?;
|
||||
tokio::fs::rename(&tmp, dest)
|
||||
.await
|
||||
.map_err(PublishError::at(dest))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Retention (§3.11)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Delete issue files older than `retention_days` from both publish dirs.
|
||||
/// SQLite history is kept forever — it's the training data (§3.11).
|
||||
/// Retention sweep over both publish dirs. SQLite history is kept forever —
|
||||
/// it's the training data (§3.11).
|
||||
///
|
||||
/// The two directories are swept on different rules: EPUBs age out after
|
||||
/// `retention_days`, while XTC is capped at `xtc_retention_count` issues because
|
||||
/// each one is ~80–100 MB of pre-rendered page bitmaps and the binding
|
||||
/// constraint is disk rather than age.
|
||||
///
|
||||
/// Only files named `The Daily EPUB - YYYY-MM-DD*.{epub,xtc,xtch}` are ever
|
||||
/// considered; anything else in those directories (including `xtc.xml` and other
|
||||
/// people's books) is left strictly alone.
|
||||
/// considered; anything else in those directories (other people's books) is left
|
||||
/// strictly alone.
|
||||
pub async fn prune(cfg: &Config, today: Date) -> Result<usize, PublishError> {
|
||||
let cutoff = today
|
||||
.checked_sub(jiff::Span::new().days(i64::from(cfg.retention_days)))
|
||||
.unwrap_or(today);
|
||||
let mut removed = 0;
|
||||
for dir in [&cfg.publish.bookorbit_dir, &cfg.publish.xtc_dir] {
|
||||
removed += prune_dir(dir, cutoff).await?;
|
||||
}
|
||||
let mut removed = prune_dir(&cfg.publish.epub_dir, cutoff).await?;
|
||||
if removed > 0 {
|
||||
tracing::info!(removed, %cutoff, "retention sweep removed expired issues");
|
||||
tracing::info!(removed, %cutoff, "retention sweep removed expired EPUBs");
|
||||
}
|
||||
let xtc_removed = prune_xtc_dir(&cfg.publish.xtc_dir, cfg.xtc_retention_count as usize).await?;
|
||||
if xtc_removed > 0 {
|
||||
tracing::info!(
|
||||
removed = xtc_removed,
|
||||
keep = cfg.xtc_retention_count,
|
||||
"retention sweep trimmed the XTC directory"
|
||||
);
|
||||
}
|
||||
removed += xtc_removed;
|
||||
Ok(removed)
|
||||
}
|
||||
|
||||
/// Keep only the newest `keep` XTC issues, deleting the rest (§3.11).
|
||||
///
|
||||
/// Counted rather than dated so the directory has a hard size ceiling no matter
|
||||
/// how often `generate` runs.
|
||||
async fn prune_xtc_dir(dir: &Path, keep: usize) -> Result<usize, PublishError> {
|
||||
if !dir.exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
let mut entries = tokio::fs::read_dir(dir)
|
||||
.await
|
||||
.map_err(PublishError::at(dir))?;
|
||||
let mut ours: Vec<(Date, PathBuf)> = Vec::new();
|
||||
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
let extension = Path::new(&name)
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.unwrap_or_default()
|
||||
.to_ascii_lowercase();
|
||||
if !XTC_EXTENSIONS.contains(&extension.as_str()) {
|
||||
continue;
|
||||
}
|
||||
let Some(date) = date_from_filename(&name) else {
|
||||
continue;
|
||||
};
|
||||
if !entry.metadata().await.map(|m| m.is_file()).unwrap_or(false) {
|
||||
continue;
|
||||
}
|
||||
ours.push((date, entry.path()));
|
||||
}
|
||||
// Newest first, then drop everything past the cap.
|
||||
ours.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(&b.1)));
|
||||
let mut removed = 0;
|
||||
for (date, path) in ours.into_iter().skip(keep) {
|
||||
match tokio::fs::remove_file(&path).await {
|
||||
Ok(()) => {
|
||||
tracing::info!(path = %path.display(), %date, "pruned an XTC issue past the cap");
|
||||
removed += 1;
|
||||
}
|
||||
Err(e) => tracing::warn!(error = %e, path = %path.display(), "could not prune file"),
|
||||
}
|
||||
}
|
||||
Ok(removed)
|
||||
}
|
||||
@@ -520,7 +602,7 @@ mod tests {
|
||||
|
||||
fn cfg(dir: &Path) -> Config {
|
||||
let mut cfg = Config::default();
|
||||
cfg.publish.bookorbit_dir = dir.join("bookorbit");
|
||||
cfg.publish.epub_dir = dir.join("bookorbit");
|
||||
cfg.publish.xtc_dir = dir.join("xtc");
|
||||
cfg.server.public_url = "https://daily.hallada.net".into();
|
||||
cfg
|
||||
@@ -561,7 +643,7 @@ mod tests {
|
||||
Some(date("2026-08-15"))
|
||||
);
|
||||
for foreign in [
|
||||
"xtc.xml",
|
||||
"The Daily EPUB - 2026-08-15.xml",
|
||||
"Moby Dick.epub",
|
||||
"The Daily EPUB - notadate.epub",
|
||||
"The Daily EPUB - 2026-08-15.txt",
|
||||
@@ -634,10 +716,10 @@ mod tests {
|
||||
paths,
|
||||
vec![
|
||||
cfg.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-08-15.epub"),
|
||||
cfg.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-08-15 (X4).epub"),
|
||||
]
|
||||
);
|
||||
@@ -673,21 +755,37 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn epub_file(name: &str, edition: Edition, modified: &str, bytes: u64) -> EpubFile {
|
||||
EpubFile {
|
||||
date: date_from_filename(name),
|
||||
edition,
|
||||
name: name.into(),
|
||||
modified: ts(modified),
|
||||
bytes,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opds_feed_is_newest_first_with_acquisition_links() {
|
||||
let files = vec![
|
||||
XtcFile {
|
||||
name: "The Daily EPUB - 2026-08-15 (X4).xtch".into(),
|
||||
date: Some(date("2026-08-15")),
|
||||
modified: ts("2026-08-15T05:40:00Z"),
|
||||
bytes: 2_500_000,
|
||||
},
|
||||
XtcFile {
|
||||
name: "The Daily EPUB - 2026-08-14 (X4).xtch".into(),
|
||||
date: Some(date("2026-08-14")),
|
||||
modified: ts("2026-08-14T05:40:00Z"),
|
||||
bytes: 4096,
|
||||
},
|
||||
epub_file(
|
||||
"The Daily EPUB - 2026-08-15.epub",
|
||||
Edition::Standard,
|
||||
"2026-08-15T05:40:00Z",
|
||||
6_500_000,
|
||||
),
|
||||
epub_file(
|
||||
"The Daily EPUB - 2026-08-15 (X4).epub",
|
||||
Edition::X4,
|
||||
"2026-08-15T05:40:00Z",
|
||||
1_700_000,
|
||||
),
|
||||
epub_file(
|
||||
"The Daily EPUB - 2026-08-14.epub",
|
||||
Edition::Standard,
|
||||
"2026-08-14T05:40:00Z",
|
||||
4096,
|
||||
),
|
||||
];
|
||||
let mut numbers = BTreeMap::new();
|
||||
numbers.insert(date("2026-08-15"), 12);
|
||||
@@ -700,37 +798,68 @@ mod tests {
|
||||
|
||||
assert!(feed.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>"));
|
||||
assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\""));
|
||||
assert!(feed.contains("<id>urn:daily-epub:xtc</id>"));
|
||||
assert!(feed.contains("<id>urn:daily-epub:issues</id>"));
|
||||
assert!(feed.contains("<updated>2026-08-15T05:40:00Z</updated>"));
|
||||
assert_eq!(feed.matches("<entry>").count(), 2);
|
||||
assert_eq!(feed.matches("</entry>").count(), 2);
|
||||
// Newest first.
|
||||
let i15 = feed.find("The Daily EPUB — 2026-08-15").unwrap();
|
||||
let i14 = feed.find("The Daily EPUB — 2026-08-14").unwrap();
|
||||
assert!(i15 < i14);
|
||||
// Acquisition link, encoded filename, absolute public URL, size.
|
||||
assert_eq!(feed.matches("<entry>").count(), 3);
|
||||
assert_eq!(feed.matches("</entry>").count(), 3);
|
||||
|
||||
// Newest issue first; the two editions of one issue are distinguishable
|
||||
// by title, matching each EPUB's own `dc:title`.
|
||||
let i15 = feed
|
||||
.find("<title>The Daily EPUB — 2026-08-15</title>")
|
||||
.unwrap();
|
||||
let i15_x4 = feed
|
||||
.find("<title>The Daily EPUB — 2026-08-15 (X4)</title>")
|
||||
.unwrap();
|
||||
let i14 = feed
|
||||
.find("<title>The Daily EPUB — 2026-08-14</title>")
|
||||
.unwrap();
|
||||
assert!(i15 < i15_x4 && i15_x4 < i14);
|
||||
|
||||
// CrossPoint compares the acquisition type with `strcmp` against
|
||||
// `application/epub+zip` and drops the entry on any mismatch (§3.11).
|
||||
assert_eq!(
|
||||
feed.matches("type=\"application/epub+zip\"").count(),
|
||||
3,
|
||||
"{feed}"
|
||||
);
|
||||
assert!(!feed.contains("application/octet-stream"));
|
||||
assert!(feed.contains(
|
||||
"<link rel=\"http://opds-spec.org/acquisition\" \
|
||||
href=\"https://daily.hallada.net/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch\" \
|
||||
type=\"application/octet-stream\" length=\"2500000\"/>"
|
||||
href=\"https://daily.hallada.net/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub\" \
|
||||
type=\"application/epub+zip\" length=\"1700000\"/>"
|
||||
));
|
||||
assert!(feed.contains("Issue #12 · 2.4 MB"));
|
||||
assert!(feed.contains("Issue #12 · Standard · 6.2 MB"));
|
||||
assert!(feed.contains("Issue #12 · Xteink X4 · 1.6 MB"));
|
||||
assert!(
|
||||
feed.contains("<link rel=\"self\" href=\"https://daily.hallada.net/opds/daily.xml\"")
|
||||
);
|
||||
assert!(feed.trim_end().ends_with("</feed>"));
|
||||
assert!(!feed.contains("&<"), "unescaped markup leaked in");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn write_xtc_opds_lists_only_xtc_files_capped_at_fourteen() {
|
||||
async fn the_feed_lists_both_editions_of_the_last_fourteen_issues() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cfg = cfg(dir.path());
|
||||
std::fs::create_dir_all(&cfg.publish.epub_dir).unwrap();
|
||||
std::fs::create_dir_all(&cfg.publish.xtc_dir).unwrap();
|
||||
for day in 1..=20 {
|
||||
let name = format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch");
|
||||
std::fs::write(cfg.publish.xtc_dir.join(name), b"x").unwrap();
|
||||
for edition in Edition::ALL {
|
||||
let name = issue_filename(date(&format!("2026-08-{day:02}")), edition, "epub");
|
||||
std::fs::write(cfg.publish.epub_dir.join(name), b"x").unwrap();
|
||||
}
|
||||
// Non-XTC neighbours must be ignored.
|
||||
std::fs::write(cfg.publish.xtc_dir.join("README.txt"), b"x").unwrap();
|
||||
std::fs::write(cfg.publish.xtc_dir.join("cover.epub"), b"x").unwrap();
|
||||
}
|
||||
// Other people's books and our own XTC artifacts must be ignored.
|
||||
std::fs::write(cfg.publish.epub_dir.join("Moby Dick.epub"), b"x").unwrap();
|
||||
std::fs::write(cfg.publish.epub_dir.join("metadata.db"), b"x").unwrap();
|
||||
std::fs::write(
|
||||
cfg.publish
|
||||
.xtc_dir
|
||||
.join("The Daily EPUB - 2026-08-20 (X4).xtch"),
|
||||
b"x",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
|
||||
.await
|
||||
@@ -748,64 +877,75 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let path = write_xtc_opds(&db, &cfg).await.unwrap();
|
||||
assert_eq!(path, cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME));
|
||||
let feed = std::fs::read_to_string(&path).unwrap();
|
||||
assert_eq!(feed.matches("<entry>").count(), XTC_FEED_ENTRIES);
|
||||
let feed = build_opds(&db, &cfg).await.unwrap();
|
||||
// Capped by issue day, so both editions of each of the last 14 survive.
|
||||
assert_eq!(feed.matches("<entry>").count(), OPDS_FEED_ISSUES * 2);
|
||||
assert!(feed.contains("2026-08-20"));
|
||||
assert!(!feed.contains("2026-08-06"), "older than the last 14");
|
||||
assert!(!feed.contains("README"));
|
||||
assert!(!feed.contains("cover.epub"));
|
||||
assert!(!feed.contains("Moby Dick"));
|
||||
assert!(!feed.contains("metadata.db"));
|
||||
assert!(feed.contains("Issue #20"));
|
||||
|
||||
// Regenerating replaces the file in place.
|
||||
write_xtc_opds(&db, &cfg).await.unwrap();
|
||||
let leftovers: Vec<String> = std::fs::read_dir(&cfg.publish.xtc_dir)
|
||||
.unwrap()
|
||||
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
|
||||
.filter(|n| n.ends_with(".tmp"))
|
||||
.collect();
|
||||
assert!(leftovers.is_empty(), "{leftovers:?}");
|
||||
// XTC is never offered: CrossPoint cannot acquire it (§3.11).
|
||||
assert!(!feed.contains(".xtch"), "{feed}");
|
||||
assert!(!feed.contains("/files/xtc/"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn prune_only_deletes_old_matching_files() {
|
||||
async fn prune_ages_out_epubs_but_counts_xtc() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let mut conf = cfg(dir.path());
|
||||
conf.retention_days = 21;
|
||||
std::fs::create_dir_all(&conf.publish.bookorbit_dir).unwrap();
|
||||
conf.xtc_retention_count = 5;
|
||||
std::fs::create_dir_all(&conf.publish.epub_dir).unwrap();
|
||||
std::fs::create_dir_all(&conf.publish.xtc_dir).unwrap();
|
||||
|
||||
let keep_epub = conf
|
||||
.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-08-14.epub");
|
||||
let old_epub = conf
|
||||
.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-07-01.epub");
|
||||
let old_x4 = conf
|
||||
.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-07-01 (X4).epub");
|
||||
let foreign = conf.publish.bookorbit_dir.join("Moby Dick.epub");
|
||||
let old_xtc = conf
|
||||
.publish
|
||||
.xtc_dir
|
||||
.join("The Daily EPUB - 2026-07-01 (X4).xtch");
|
||||
let feed = conf.publish.xtc_dir.join(XTC_OPDS_FILENAME);
|
||||
for path in [&keep_epub, &old_epub, &old_x4, &foreign, &old_xtc, &feed] {
|
||||
let foreign = conf.publish.epub_dir.join("Moby Dick.epub");
|
||||
for path in [&keep_epub, &old_epub, &old_x4, &foreign] {
|
||||
std::fs::write(path, b"x").unwrap();
|
||||
}
|
||||
|
||||
// Eight consecutive XTC issues, all recent: age would keep every one,
|
||||
// the count cap keeps the newest five.
|
||||
let xtc: Vec<PathBuf> = (8..=15)
|
||||
.map(|day| {
|
||||
let path = conf
|
||||
.publish
|
||||
.xtc_dir
|
||||
.join(format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch"));
|
||||
std::fs::write(&path, b"x").unwrap();
|
||||
path
|
||||
})
|
||||
.collect();
|
||||
let foreign_xtc = conf.publish.xtc_dir.join("Someone Else.xtch");
|
||||
std::fs::write(&foreign_xtc, b"x").unwrap();
|
||||
|
||||
// 2 expired EPUBs + 3 XTC issues past the cap of 5.
|
||||
let removed = prune(&conf, date("2026-08-15")).await.unwrap();
|
||||
assert_eq!(removed, 3);
|
||||
assert_eq!(removed, 5);
|
||||
assert!(keep_epub.exists());
|
||||
assert!(foreign.exists(), "never touch other people's books");
|
||||
assert!(feed.exists(), "the OPDS feed is not an issue file");
|
||||
assert!(foreign_xtc.exists(), "nor their XTC files");
|
||||
assert!(!old_epub.exists());
|
||||
assert!(!old_x4.exists());
|
||||
assert!(!old_xtc.exists());
|
||||
// The five newest XTC issues (11th–15th) survive; 8th–10th are gone.
|
||||
for path in &xtc[..3] {
|
||||
assert!(!path.exists(), "{} should be pruned", path.display());
|
||||
}
|
||||
for path in &xtc[3..] {
|
||||
assert!(path.exists(), "{} should be kept", path.display());
|
||||
}
|
||||
|
||||
// Idempotent, and tolerant of missing directories.
|
||||
assert_eq!(prune(&conf, date("2026-08-15")).await.unwrap(), 0);
|
||||
@@ -844,25 +984,31 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
|
||||
];
|
||||
let issue = fake_issue(date("2026-08-15"));
|
||||
|
||||
let published = publish_issue(&db, &cfg, &issue, &artifacts, Some(&xtc_src))
|
||||
let published = publish_issue(&cfg, &issue, &artifacts, Some(&xtc_src))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(published.epubs.len(), 2);
|
||||
assert!(published.epubs.iter().all(|a| a.path.exists()));
|
||||
assert_eq!(published.epubs[1].edition, Edition::X4);
|
||||
assert!(published.xtc.as_ref().is_some_and(|p| p.exists()));
|
||||
assert!(published.opds.as_ref().is_some_and(|p| p.exists()));
|
||||
assert_eq!(published.pruned, 0);
|
||||
|
||||
let feed = std::fs::read_to_string(cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)).unwrap();
|
||||
assert!(feed.contains("out.xtch"));
|
||||
// The feed is derived from the publish directory, so both editions show
|
||||
// up without publish having written anything.
|
||||
let feed = build_opds(&db, &cfg).await.unwrap();
|
||||
assert!(
|
||||
feed.contains("The Daily EPUB — 2026-08-15</title>"),
|
||||
"{feed}"
|
||||
);
|
||||
assert!(
|
||||
feed.contains("The Daily EPUB — 2026-08-15 (X4)</title>"),
|
||||
"{feed}"
|
||||
);
|
||||
assert!(!feed.contains("out.xtch"));
|
||||
|
||||
// No XTC artifact is fine — the feed is still regenerated.
|
||||
let published = publish_issue(&db, &cfg, &issue, &artifacts, None)
|
||||
.await
|
||||
.unwrap();
|
||||
// No XTC artifact is fine — the EPUBs are what the feed lists anyway.
|
||||
let published = publish_issue(&cfg, &issue, &artifacts, None).await.unwrap();
|
||||
assert!(published.xtc.is_none());
|
||||
assert!(published.opds.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
+89
-35
@@ -1,4 +1,4 @@
|
||||
//! axum server: rating endpoints, XTC OPDS, static files (spec §3.9, §3.12).
|
||||
//! axum server: rating endpoints, the OPDS catalog, downloads (spec §3.9, §3.12).
|
||||
//!
|
||||
//! Rating links must work from an e-reader's built-in browser, so every rating
|
||||
//! endpoint is a `GET` and the response is a tiny e-ink-sized HTML page.
|
||||
@@ -7,8 +7,9 @@
|
||||
//! | route | behaviour |
|
||||
//! |---|---|
|
||||
//! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC, upsert rating, rebuild feed priors |
|
||||
//! | `GET /opds/xtc.xml` | static OPDS 1.2 acquisition feed from `publish.xtc_dir` |
|
||||
//! | `GET /files/xtc/{name}` | XTC artifact download (no path traversal) |
|
||||
//! | `GET /opds/daily.xml` (also `/opds`, `/opds/`) | OPDS 1.2 acquisition feed over `publish.epub_dir` |
|
||||
//! | `GET /files/epub/{name}` | EPUB download — what the feed's acquisition links point at |
|
||||
//! | `GET /files/xtc/{name}` | XTC artifact download, unlisted (no path traversal) |
|
||||
//! | `GET /healthz` | liveness |
|
||||
//! | `GET /issues.json` | the last 30 run reports, newest first |
|
||||
//!
|
||||
@@ -81,16 +82,18 @@ pub use crate::auth::{constant_time_eq, rating_token, rating_url, verify_token};
|
||||
// Router (§3.12)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/xtc.xml`,
|
||||
/// `/files/xtc/{name}`, `/healthz`, `/issues.json`, with `tower-http` tracing (§3.12).
|
||||
/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/daily.xml`,
|
||||
/// `/files/epub/{name}`, `/files/xtc/{name}`, `/healthz`, `/issues.json`, with
|
||||
/// `tower-http` tracing (§3.12).
|
||||
pub fn router(state: AppState) -> Router {
|
||||
Router::new()
|
||||
.route("/r/{date}/{article_id}/{vote}", get(handle_rating))
|
||||
.route("/opds/xtc.xml", get(handle_opds))
|
||||
.route(crate::publish::OPDS_PATH, get(handle_opds))
|
||||
// OPDS browsers are typed into by hand on a 6" e-ink keyboard: serve the
|
||||
// same feed from the catalog root so a URL without the filename works.
|
||||
.route("/opds", get(handle_opds))
|
||||
.route("/opds/", get(handle_opds))
|
||||
.route("/files/epub/{name}", get(handle_epub_file))
|
||||
.route("/files/xtc/{name}", get(handle_xtc_file))
|
||||
.route("/healthz", get(handle_healthz))
|
||||
.route("/issues.json", get(handle_issues_json))
|
||||
@@ -284,44 +287,64 @@ async fn handle_rating(
|
||||
)
|
||||
}
|
||||
|
||||
/// `GET /opds/xtc.xml` — the static feed written by [`crate::publish`] (§3.11).
|
||||
/// `GET /opds/daily.xml` — both EPUB editions of the last issues, newest first,
|
||||
/// rendered from the publish directory on each request (§3.11).
|
||||
async fn handle_opds(State(state): State<AppState>, headers: HeaderMap) -> Response {
|
||||
if let Some(challenge) = check_basic_auth(&state.config, &headers) {
|
||||
return challenge;
|
||||
}
|
||||
let path = state
|
||||
.config
|
||||
.publish
|
||||
.xtc_dir
|
||||
.join(crate::publish::XTC_OPDS_FILENAME);
|
||||
match tokio::fs::read(&path).await {
|
||||
Ok(bytes) => (
|
||||
match crate::publish::build_opds(&state.db, &state.config).await {
|
||||
Ok(feed) => (
|
||||
StatusCode::OK,
|
||||
[
|
||||
(header::CONTENT_TYPE, OPDS_CONTENT_TYPE),
|
||||
(header::CACHE_CONTROL, "no-cache"),
|
||||
],
|
||||
bytes,
|
||||
feed,
|
||||
)
|
||||
.into_response(),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, path = %path.display(), "no XTC OPDS feed yet");
|
||||
(StatusCode::NOT_FOUND, "no feed yet").into_response()
|
||||
tracing::error!(error = %e, "could not build the OPDS feed");
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"could not build the feed",
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `GET /files/epub/{name}` — download one published EPUB; this is what the
|
||||
/// OPDS acquisition links point at (§3.11).
|
||||
async fn handle_epub_file(
|
||||
State(state): State<AppState>,
|
||||
Path(name): Path<String>,
|
||||
headers: HeaderMap,
|
||||
) -> Response {
|
||||
let dir = state.config.publish.epub_dir.clone();
|
||||
serve_file(&state, &dir, &name, &headers).await
|
||||
}
|
||||
|
||||
/// `GET /files/xtc/{name}` — download one XTC artifact (§3.11).
|
||||
///
|
||||
/// Not listed in the OPDS feed — CrossPoint's browser cannot acquire XTC — but
|
||||
/// kept so the artifacts can still be fetched by URL for sideloading.
|
||||
async fn handle_xtc_file(
|
||||
State(state): State<AppState>,
|
||||
Path(name): Path<String>,
|
||||
headers: HeaderMap,
|
||||
) -> Response {
|
||||
if let Some(challenge) = check_basic_auth(&state.config, &headers) {
|
||||
let dir = state.config.publish.xtc_dir.clone();
|
||||
serve_file(&state, &dir, &name, &headers).await
|
||||
}
|
||||
|
||||
/// Stream one file out of `dir`, behind the OPDS Basic auth (§3.11).
|
||||
async fn serve_file(state: &AppState, dir: &FsPath, name: &str, headers: &HeaderMap) -> Response {
|
||||
if let Some(challenge) = check_basic_auth(&state.config, headers) {
|
||||
return challenge;
|
||||
}
|
||||
let Some(path) = safe_join(&state.config.publish.xtc_dir, &name) else {
|
||||
tracing::warn!(name, "rejected an unsafe XTC file name");
|
||||
let Some(path) = safe_join(dir, name) else {
|
||||
tracing::warn!(name, "rejected an unsafe file name");
|
||||
return (StatusCode::BAD_REQUEST, "bad file name").into_response();
|
||||
};
|
||||
// An XTCH issue is a pre-rendered page bitmap per page — ~100 MB for a full
|
||||
@@ -332,12 +355,14 @@ async fn handle_xtc_file(
|
||||
(file, len)
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, path = %path.display(), "XTC file not found");
|
||||
tracing::warn!(error = %e, path = %path.display(), "file not found");
|
||||
return (StatusCode::NOT_FOUND, "not found").into_response();
|
||||
}
|
||||
};
|
||||
let content_type = if name.ends_with(".xml") {
|
||||
OPDS_CONTENT_TYPE
|
||||
// CrossPoint dispatches on the saved file's extension, not on this header,
|
||||
// but Calibre and KOReader both use it.
|
||||
let content_type = if name.ends_with(".epub") {
|
||||
crate::publish::EPUB_CONTENT_TYPE
|
||||
} else {
|
||||
"application/octet-stream"
|
||||
};
|
||||
@@ -629,7 +654,9 @@ mod tests {
|
||||
async fn start(with_auth: bool) -> TestServer {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let xtc_dir = dir.path().join("xtc");
|
||||
let epub_dir = dir.path().join("epub");
|
||||
std::fs::create_dir_all(&xtc_dir).unwrap();
|
||||
std::fs::create_dir_all(&epub_dir).unwrap();
|
||||
let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -637,6 +664,8 @@ mod tests {
|
||||
let mut config = Config::default();
|
||||
config.server.hmac_secret = Some(VECTOR_SECRET.into());
|
||||
config.publish.xtc_dir = xtc_dir;
|
||||
config.publish.epub_dir = epub_dir;
|
||||
config.server.public_url = "https://daily.hallada.net".into();
|
||||
if with_auth {
|
||||
config.server.basic_auth_user = Some("opds".into());
|
||||
config.server.basic_auth_pass = Some("hunter2".into());
|
||||
@@ -663,6 +692,10 @@ mod tests {
|
||||
self._dir.path().join("xtc")
|
||||
}
|
||||
|
||||
fn epub_dir(&self) -> PathBuf {
|
||||
self._dir.path().join("epub")
|
||||
}
|
||||
|
||||
async fn seed_article(&self) -> ArticleId {
|
||||
let entry = crate::types::Entry {
|
||||
id: 1,
|
||||
@@ -847,11 +880,10 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn opds_and_files_are_served_behind_basic_auth() {
|
||||
let server = TestServer::start(true).await;
|
||||
std::fs::write(
|
||||
server.xtc_dir().join(crate::publish::XTC_OPDS_FILENAME),
|
||||
"<feed/>",
|
||||
)
|
||||
.unwrap();
|
||||
for edition in crate::types::Edition::ALL {
|
||||
let name = crate::publish::issue_filename(date(), edition, "epub");
|
||||
std::fs::write(server.epub_dir().join(name), b"EPUB").unwrap();
|
||||
}
|
||||
std::fs::write(
|
||||
server
|
||||
.xtc_dir()
|
||||
@@ -861,7 +893,7 @@ mod tests {
|
||||
.unwrap();
|
||||
|
||||
let res = client()
|
||||
.get(format!("{}/opds/xtc.xml", server.base))
|
||||
.get(format!("{}/opds/daily.xml", server.base))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -875,7 +907,7 @@ mod tests {
|
||||
);
|
||||
|
||||
let res = client()
|
||||
.get(format!("{}/opds/xtc.xml", server.base))
|
||||
.get(format!("{}/opds/daily.xml", server.base))
|
||||
.basic_auth("opds", Some("wrong"))
|
||||
.send()
|
||||
.await
|
||||
@@ -883,7 +915,7 @@ mod tests {
|
||||
assert_eq!(res.status(), 401);
|
||||
|
||||
let res = client()
|
||||
.get(format!("{}/opds/xtc.xml", server.base))
|
||||
.get(format!("{}/opds/daily.xml", server.base))
|
||||
.basic_auth("opds", Some("hunter2"))
|
||||
.send()
|
||||
.await
|
||||
@@ -895,7 +927,11 @@ mod tests {
|
||||
.unwrap()
|
||||
.starts_with("application/atom+xml")
|
||||
);
|
||||
assert_eq!(res.text().await.unwrap(), "<feed/>");
|
||||
let feed = res.text().await.unwrap();
|
||||
assert_eq!(feed.matches("<entry>").count(), 2, "{feed}");
|
||||
assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}");
|
||||
// XTC exists on disk but is never advertised (§3.11).
|
||||
assert!(!feed.contains(".xtch"), "{feed}");
|
||||
|
||||
// The catalog root serves the same feed, behind the same auth.
|
||||
for alias in ["/opds", "/opds/"] {
|
||||
@@ -912,9 +948,29 @@ mod tests {
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(res.status(), 200, "{alias}");
|
||||
assert_eq!(res.text().await.unwrap(), "<feed/>", "{alias}");
|
||||
assert_eq!(res.text().await.unwrap(), feed, "{alias}");
|
||||
}
|
||||
|
||||
// The acquisition link resolves, typed as an EPUB.
|
||||
let res = client()
|
||||
.get(format!(
|
||||
"{}/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20(X4).epub",
|
||||
server.base
|
||||
))
|
||||
.basic_auth("opds", Some("hunter2"))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(res.status(), 200);
|
||||
// CrossPoint needs the size up front to show download progress.
|
||||
assert_eq!(res.content_length(), Some(4));
|
||||
assert_eq!(
|
||||
res.headers()[header::CONTENT_TYPE].to_str().unwrap(),
|
||||
"application/epub+zip"
|
||||
);
|
||||
assert_eq!(res.bytes().await.unwrap().as_ref(), b"EPUB");
|
||||
|
||||
// XTC is still fetchable by URL for sideloading, just not listed.
|
||||
let res = client()
|
||||
.get(format!(
|
||||
"{}/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20(X4).xtch",
|
||||
@@ -925,8 +981,6 @@ mod tests {
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(res.status(), 200);
|
||||
// CrossPoint needs the size up front to show download progress.
|
||||
assert_eq!(res.content_length(), Some(4));
|
||||
assert_eq!(res.bytes().await.unwrap().as_ref(), b"XTCH");
|
||||
|
||||
// Ratings are not behind auth (the token is the credential).
|
||||
|
||||
+19
-1
@@ -411,7 +411,10 @@ pub const WORLD_BRIEFING_SECTION: &str = "World Briefing";
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Which of the two editions is being built (§3.10).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
///
|
||||
/// The declaration order is the listing order: standard first, then X4. The
|
||||
/// OPDS feed sorts on it.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Edition {
|
||||
/// 1200px images, full CSS.
|
||||
@@ -421,6 +424,9 @@ pub enum Edition {
|
||||
}
|
||||
|
||||
impl Edition {
|
||||
/// Every edition, in the order they are listed and built (§3.10).
|
||||
pub const ALL: [Edition; 2] = [Edition::Standard, Edition::X4];
|
||||
|
||||
/// Filename suffix: `""` / `" (X4)"` (§3.11).
|
||||
pub fn file_suffix(self) -> &'static str {
|
||||
match self {
|
||||
@@ -428,6 +434,18 @@ impl Edition {
|
||||
Edition::X4 => " (X4)",
|
||||
}
|
||||
}
|
||||
|
||||
/// Recover the edition from a published filename's stem (§3.11).
|
||||
///
|
||||
/// The OPDS feed is rebuilt by scanning the publish directory, so the
|
||||
/// filename is the only record of which edition a file is.
|
||||
pub fn from_file_stem(stem: &str) -> Edition {
|
||||
if stem.ends_with(Edition::X4.file_suffix()) {
|
||||
Edition::X4
|
||||
} else {
|
||||
Edition::Standard
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Issue-level metadata rendered on the cover, front page and OPF (§3.10).
|
||||
|
||||
+19
-11
@@ -59,7 +59,7 @@ fn test_config(root: &Path) -> Config {
|
||||
prefilter_keep: 20,
|
||||
world_briefing: false,
|
||||
publish: PublishConfig {
|
||||
bookorbit_dir: root.join("bookorbit"),
|
||||
epub_dir: root.join("bookorbit"),
|
||||
xtc_dir: root.join("xtc"),
|
||||
},
|
||||
xtc: XtcConfig {
|
||||
@@ -305,35 +305,44 @@ async fn assemble_build_publish(
|
||||
);
|
||||
|
||||
// --- Publish (§3.11) ---
|
||||
let published = publish::publish_issue(db, cfg, &issue, &artifacts, None)
|
||||
let published = publish::publish_issue(cfg, &issue, &artifacts, None)
|
||||
.await
|
||||
.expect("publish");
|
||||
assert_eq!(published.epubs.len(), 2);
|
||||
for artifact in &published.epubs {
|
||||
assert!(artifact.path.starts_with(&cfg.publish.bookorbit_dir));
|
||||
assert!(artifact.path.starts_with(&cfg.publish.epub_dir));
|
||||
assert!(artifact.path.exists(), "{}", artifact.path.display());
|
||||
}
|
||||
assert!(
|
||||
cfg.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-08-15.epub")
|
||||
.exists()
|
||||
);
|
||||
assert!(
|
||||
cfg.publish
|
||||
.bookorbit_dir
|
||||
.epub_dir
|
||||
.join("The Daily EPUB - 2026-08-15 (X4).epub")
|
||||
.exists()
|
||||
);
|
||||
assert!(published.xtc.is_none(), "the converter is disabled here");
|
||||
|
||||
// The OPDS feed is regenerated on every publish, even with no XTC files yet.
|
||||
let opds = published.opds.clone().expect("an OPDS feed was written");
|
||||
assert_eq!(opds, cfg.publish.xtc_dir.join("xtc.xml"));
|
||||
let feed = std::fs::read_to_string(&opds).expect("read the OPDS feed");
|
||||
// The OPDS feed is derived from what was just published — both editions,
|
||||
// typed so CrossPoint will accept them (§3.11).
|
||||
let feed = publish::build_opds(db, cfg).await.expect("build the feed");
|
||||
assert!(feed.starts_with("<?xml"), "{feed}");
|
||||
assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\""));
|
||||
assert!(feed.contains(&cfg.server.public_url));
|
||||
assert_eq!(feed.matches("<entry>").count(), 2, "{feed}");
|
||||
assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}");
|
||||
assert!(
|
||||
feed.contains("The Daily EPUB — 2026-08-15</title>"),
|
||||
"{feed}"
|
||||
);
|
||||
assert!(
|
||||
feed.contains("The Daily EPUB — 2026-08-15 (X4)</title>"),
|
||||
"{feed}"
|
||||
);
|
||||
|
||||
// --- Record (§3.13) ---
|
||||
let epub_path = published
|
||||
@@ -438,7 +447,6 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
|
||||
|
||||
// Re-running the same date replaces rather than duplicates (notes §12).
|
||||
let republished = publish::publish_issue(
|
||||
&db,
|
||||
&cfg,
|
||||
&issue,
|
||||
&[
|
||||
@@ -458,7 +466,7 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
|
||||
.await
|
||||
.expect("republish");
|
||||
assert_eq!(republished.epubs.len(), 2);
|
||||
let files: Vec<String> = std::fs::read_dir(&cfg.publish.bookorbit_dir)
|
||||
let files: Vec<String> = std::fs::read_dir(&cfg.publish.epub_dir)
|
||||
.expect("read bookorbit dir")
|
||||
.filter_map(|e| e.ok().map(|e| e.file_name().to_string_lossy().into_owned()))
|
||||
.collect();
|
||||
|
||||
+48
-26
@@ -41,6 +41,7 @@ impl Server {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let xtc_dir = dir.path().join("xtc");
|
||||
std::fs::create_dir_all(&xtc_dir).expect("xtc dir");
|
||||
std::fs::create_dir_all(dir.path().join("bookorbit")).expect("epub dir");
|
||||
let log = std::fs::File::create(dir.path().join("server.log")).expect("log file");
|
||||
|
||||
let port = free_port();
|
||||
@@ -59,10 +60,7 @@ impl Server {
|
||||
format!("http://127.0.0.1:{port}"),
|
||||
)
|
||||
.env("DAILY_EPUB_PUBLISH__XTC_DIR", &xtc_dir)
|
||||
.env(
|
||||
"DAILY_EPUB_PUBLISH__BOOKORBIT_DIR",
|
||||
dir.path().join("bookorbit"),
|
||||
)
|
||||
.env("DAILY_EPUB_PUBLISH__EPUB_DIR", dir.path().join("bookorbit"))
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::from(log));
|
||||
if basic_auth {
|
||||
@@ -80,6 +78,10 @@ impl Server {
|
||||
self.dir.path().join("xtc")
|
||||
}
|
||||
|
||||
fn epub_dir(&self) -> std::path::PathBuf {
|
||||
self.dir.path().join("bookorbit")
|
||||
}
|
||||
|
||||
fn wait_until_ready(&self) {
|
||||
let deadline = Instant::now() + Duration::from_secs(30);
|
||||
while Instant::now() < deadline {
|
||||
@@ -217,22 +219,38 @@ fn binary_serves_health_issues_and_rating_endpoints() {
|
||||
#[test]
|
||||
fn binary_serves_opds_and_files_behind_basic_auth() {
|
||||
let server = Server::start(true);
|
||||
let name = "The Daily EPUB - 2026-08-15 (X4).xtch";
|
||||
write(&server.xtc_dir().join("xtc.xml"), FEED);
|
||||
write(&server.xtc_dir().join(name), "XTCH");
|
||||
write(
|
||||
&server.epub_dir().join("The Daily EPUB - 2026-08-15.epub"),
|
||||
"STANDARD",
|
||||
);
|
||||
write(
|
||||
&server
|
||||
.epub_dir()
|
||||
.join("The Daily EPUB - 2026-08-15 (X4).epub"),
|
||||
"X4EPUB",
|
||||
);
|
||||
write(
|
||||
&server
|
||||
.xtc_dir()
|
||||
.join("The Daily EPUB - 2026-08-15 (X4).xtch"),
|
||||
"XTCH",
|
||||
);
|
||||
write(&server.dir.path().join("secret"), "top secret");
|
||||
|
||||
// No credentials → challenge.
|
||||
let res = server.get("/opds/xtc.xml");
|
||||
let res = server.get("/opds/daily.xml");
|
||||
assert_eq!(res.status, 401);
|
||||
assert_eq!(
|
||||
res.header("www-authenticate"),
|
||||
Some("Basic realm=\"The Daily EPUB\", charset=\"UTF-8\"")
|
||||
);
|
||||
assert_eq!(server.get_auth("/opds/xtc.xml", "bm9wZTpub3Bl").status, 401);
|
||||
assert_eq!(
|
||||
server.get_auth("/opds/daily.xml", "bm9wZTpub3Bl").status,
|
||||
401
|
||||
);
|
||||
|
||||
// Correct credentials → the feed, typed as OPDS.
|
||||
let res = server.get_auth("/opds/xtc.xml", BASIC_AUTH);
|
||||
// Correct credentials → a feed built from the publish dir, typed as OPDS.
|
||||
let res = server.get_auth("/opds/daily.xml", BASIC_AUTH);
|
||||
assert_eq!(res.status, 200);
|
||||
assert!(
|
||||
res.header("content-type")
|
||||
@@ -242,8 +260,27 @@ fn binary_serves_opds_and_files_behind_basic_auth() {
|
||||
res.headers
|
||||
);
|
||||
assert!(res.body.contains("opds-spec.org/acquisition"));
|
||||
// Both editions, and only as `application/epub+zip` — the one acquisition
|
||||
// type CrossPoint's parser accepts.
|
||||
assert_eq!(res.body.matches("<entry>").count(), 2, "{}", res.body);
|
||||
assert_eq!(
|
||||
res.body.matches("type=\"application/epub+zip\"").count(),
|
||||
2,
|
||||
"{}",
|
||||
res.body
|
||||
);
|
||||
assert!(!res.body.contains(".xtch"), "{}", res.body);
|
||||
|
||||
// The acquisition link in the feed resolves to the file itself.
|
||||
let res = server.get_auth(
|
||||
"/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub",
|
||||
BASIC_AUTH,
|
||||
);
|
||||
assert_eq!(res.status, 200);
|
||||
assert_eq!(res.body, "X4EPUB");
|
||||
assert_eq!(res.header("content-type"), Some("application/epub+zip"));
|
||||
|
||||
// XTC stays reachable by URL for sideloading, just unlisted.
|
||||
let res = server.get_auth(
|
||||
"/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch",
|
||||
BASIC_AUTH,
|
||||
@@ -272,18 +309,3 @@ fn binary_serves_opds_and_files_behind_basic_auth() {
|
||||
fn write(path: &Path, body: &str) {
|
||||
std::fs::write(path, body).unwrap_or_else(|e| panic!("writing {}: {e}", path.display()));
|
||||
}
|
||||
|
||||
/// A feed shaped like the one `publish::write_xtc_opds` generates.
|
||||
const FEED: &str = r#"<?xml version="1.0" encoding="utf-8"?>
|
||||
<feed xmlns="http://www.w3.org/2005/Atom">
|
||||
<id>urn:daily-epub:xtc</id>
|
||||
<title>The Daily EPUB — XTC editions</title>
|
||||
<updated>2026-08-15T05:40:00Z</updated>
|
||||
<entry>
|
||||
<title>The Daily EPUB — 2026-08-15</title>
|
||||
<id>urn:daily-epub:xtc:x</id>
|
||||
<updated>2026-08-15T05:40:00Z</updated>
|
||||
<link rel="http://opds-spec.org/acquisition" href="http://127.0.0.1/files/xtc/x.xtch" type="application/octet-stream" length="4"/>
|
||||
</entry>
|
||||
</feed>
|
||||
"#;
|
||||
|
||||
Reference in New Issue
Block a user