thallada/the-daily-epub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Serve EPUBs instead XTC in daily OPDS

Browse Source 
It turns out it is currently impossible for the Xteink X4 to download
XTC files from an OPDS server. So I switched the built-in OPDS server in
this binary (daily.hallada.net/opds) to serve the EPUB files instead of
the XTC files. The X4 is pretty capable of reading the X4 edition of the
EPUB that was optimized for it anyways, and I prefer the flexibility of
EPUB, so I might end up eventually deleting the XTC conversion. For now,
I kept the conversion step (in case I ever want to ever try manually
copying them to the device) and I keep a limited number of XTC issues on
the server since they are quite big in filesize. The BookOrbit
integration is now optional since the built-in OPDS server is able to
serve the EPUBs. In my installation, I serve the EPUBs through both. The
daily.hallada.net/opds server is just a little quicker to navigate and
download the EPUBs on the X4 since the BookOrbit OPDS requires diving
into a couple layers of folders before you get to the files.
This commit is contained in:
thallada
2026-08-15 20:24:20 +00:00
parent 9e30c1dcdf
commit 9e27a32fb6
12 changed files with 630 additions and 282 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/publish.rs
+299 -153
View File
@@ -1,6 +1,9 @@
//! Publishing: BookOrbit watched folder, XTC delivery, OPDS feed, retention
//! Publishing: the EPUB library folder, XTC delivery, OPDS feed, retention
//! (spec §3.11).
//!
//! `publish.epub_dir` holds both EPUB editions and is what the OPDS feed lists;
//! BookOrbit may watch the same folder but nothing here depends on it.
//!
//! Everything here is deliberately dumb about *how* artifacts were produced: the
//! EPUB/XTC stages hand over finished files, this module only copies, indexes and
//! prunes them. Copies are atomic (temp file in the destination directory, then
@@ -8,11 +11,15 @@
//! half-written book.
//!
//! [`crate::pipeline`] ends a non-dry run with one call —
//! `publish_issue(db, config, &issue, &artifacts, xtc_path.as_deref())`, where
//! `publish_issue(config, &issue, &artifacts, xtc_path.as_deref())`, where
//! `artifacts` are the `epub::build_all` outputs and `xtc_path` is
//! `epub::x4::convert`'s output (`None` when the converter is disabled or
//! failed) — and feeds the returned [`Published`] paths into
//! `db.upsert_issue(..., epub_path, x4_path, xtc_path, ...)`.
//!
//! The OPDS feed ([`build_opds`]) is rendered per request by
//! [`crate::server`] rather than written here, and lists **EPUBs only** — see
//! its docs for why XTC cannot be delivered over OPDS.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
@@ -26,14 +33,19 @@ use crate::config::Config;
use crate::db::Db;
use crate::types::{Artifact, Edition, Issue};
/// Filename of the generated static OPDS feed (§3.11).
pub const XTC_OPDS_FILENAME: &str = "xtc.xml";
/// Number of issues listed in the XTC OPDS feed (§3.11).
pub const XTC_FEED_ENTRIES: usize = 14;
/// Number of issue *days* listed in the OPDS feed; each contributes one entry
/// per edition (§3.11).
pub const OPDS_FEED_ISSUES: usize = 14;
/// Every published file starts with this (the retention sweep keys off it).
pub const FILE_PREFIX: &str = "The Daily EPUB - ";
/// Extensions the retention sweep is allowed to delete (§3.11).
pub const PRUNABLE_EXTENSIONS: [&str; 3] = ["epub", "xtc", "xtch"];
/// Extensions of the XTC artifacts, for the counted sweep of `xtc_dir` (§3.11).
pub const XTC_EXTENSIONS: [&str; 2] = ["xtc", "xtch"];
/// Canonical path of the OPDS feed, relative to `server.public_url` (§3.11).
pub const OPDS_PATH: &str = "/opds/daily.xml";
/// The one acquisition type CrossPoint's OPDS parser accepts (§3.11).
pub const EPUB_CONTENT_TYPE: &str = "application/epub+zip";
#[derive(Debug, thiserror::Error)]
pub enum PublishError {
@@ -61,8 +73,6 @@ pub struct Published {
pub epubs: Vec<Artifact>,
/// The XTC artifact's published location, when the converter produced one.
pub xtc: Option<PathBuf>,
/// The regenerated OPDS feed.
pub opds: Option<PathBuf>,
/// How many expired files the retention sweep removed.
pub pruned: usize,
}
@@ -136,7 +146,7 @@ async fn ensure_dir(dir: &Path) -> Result<(), PublishError> {
.map_err(PublishError::at(dir))
}
/// Copy both EPUB editions into the BookOrbit watched folder (§3.11).
/// Copy both EPUB editions into `publish.epub_dir` (§3.11).
///
/// Returns the published paths in the same order as `artifacts`.
pub async fn publish_epubs(
@@ -144,7 +154,7 @@ pub async fn publish_epubs(
issue: &Issue,
cfg: &Config,
) -> Result<Vec<PathBuf>, PublishError> {
let dir = &cfg.publish.bookorbit_dir;
let dir = &cfg.publish.epub_dir;
ensure_dir(dir).await?;
let mut published = Vec::with_capacity(artifacts.len());
for artifact in artifacts {
@@ -159,7 +169,7 @@ pub async fn publish_epubs(
edition = ?artifact.edition,
dest = %dest.display(),
bytes = artifact.bytes,
"published edition to the BookOrbit library"
"published edition to the EPUB library"
);
published.push(dest);
}
@@ -180,12 +190,11 @@ pub async fn publish_xtc(xtc: &Path, cfg: &Config) -> Result<PathBuf, PublishErr
Ok(dest)
}
/// Publish everything one run produced, refresh the OPDS feed and prune (§3.11).
/// Publish everything one run produced and prune (§3.11).
///
/// `xtc` is `None` when the converter is disabled or failed — that is not an
/// error, the X4 falls back to the EPUB edition from BookOrbit.
/// error, since the OPDS feed lists the EPUB editions either way.
pub async fn publish_issue(
db: &Db,
cfg: &Config,
issue: &Issue,
artifacts: &[Artifact],
@@ -208,67 +217,74 @@ pub async fn publish_issue(
Some(src) => Some(publish_xtc(src, cfg).await?),
None => None,
};
let opds = Some(write_xtc_opds(db, cfg).await?);
// The OPDS feed is rendered per request from the publish directory, so
// there is nothing to write here (§3.11).
let pruned = prune(cfg, issue.meta.date).await?;
Ok(Published {
epubs,
xtc,
opds,
pruned,
})
Ok(Published { epubs, xtc, pruned })
}
// ---------------------------------------------------------------------------
// OPDS 1.2 acquisition feed (§3.11)
// ---------------------------------------------------------------------------
/// One published XTC file, as listed in the feed.
/// One published EPUB, as listed in the feed.
#[derive(Debug, Clone, PartialEq, Eq)]
struct XtcFile {
struct EpubFile {
name: String,
date: Option<Date>,
edition: Edition,
modified: Timestamp,
bytes: u64,
}
/// Regenerate the static OPDS 1.2 acquisition feed for the XTC directory:
/// newest first, last [`XTC_FEED_ENTRIES`], entries typed
/// `application/octet-stream` (§3.11).
pub async fn write_xtc_opds(db: &Db, cfg: &Config) -> Result<PathBuf, PublishError> {
let dir = &cfg.publish.xtc_dir;
/// Render the OPDS 1.2 acquisition feed for the EPUB publish directory (§3.11).
///
/// Rendered per request rather than written to disk: the directory is the only
/// source of truth, so the feed cannot go stale behind a failed publish, and
/// there is no generated file for the retention sweep to step around.
///
/// XTC artifacts are deliberately **not** listed. CrossPoint's OPDS browser only
/// acquires links typed `application/epub+zip` and always saves the result with
/// a `.epub` extension, which its reader dispatches on — so an XTC offered here
/// would either be invisible or download into a file that cannot be opened.
pub async fn build_opds(db: &Db, cfg: &Config) -> Result<String, PublishError> {
let dir = &cfg.publish.epub_dir;
ensure_dir(dir).await?;
let files = scan_xtc_dir(dir).await?;
let files = scan_epub_dir(dir).await?;
let numbers = issue_numbers(db, &files).await;
let feed = render_opds(&files, &numbers, &cfg.server.public_url, Timestamp::now());
let dest = dir.join(XTC_OPDS_FILENAME);
write_atomic(&dest, feed.as_bytes()).await?;
tracing::info!(entries = files.len(), dest = %dest.display(), "wrote the XTC OPDS feed");
Ok(dest)
Ok(render_opds(
&files,
&numbers,
&cfg.server.public_url,
Timestamp::now(),
))
}
/// XTC artifacts in `dir`, newest first, capped at [`XTC_FEED_ENTRIES`].
async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
/// Published EPUBs in `dir`, newest first, limited to the last
/// [`OPDS_FEED_ISSUES`] issue days (both editions of each).
async fn scan_epub_dir(dir: &Path) -> Result<Vec<EpubFile>, PublishError> {
let mut entries = tokio::fs::read_dir(dir)
.await
.map_err(PublishError::at(dir))?;
let mut files = Vec::new();
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
let name = entry.file_name().to_string_lossy().into_owned();
let extension = Path::new(&name)
let path = Path::new(&name);
let extension = path
.extension()
.and_then(|e| e.to_str())
.unwrap_or_default()
.to_ascii_lowercase();
if !matches!(extension.as_str(), "xtc" | "xtch") {
// Only our own issues: a shared library may hold other people's books.
if extension != "epub" || date_from_filename(&name).is_none() {
continue;
}
let meta = match entry.metadata().await {
Ok(meta) if meta.is_file() => meta,
Ok(_) => continue,
Err(e) => {
tracing::warn!(error = %e, name, "skipping unreadable XTC file");
tracing::warn!(error = %e, name, "skipping unreadable EPUB");
continue;
}
};
@@ -277,29 +293,47 @@ async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
.ok()
.and_then(|m| Timestamp::try_from(m).ok())
.unwrap_or_else(Timestamp::now);
files.push(XtcFile {
let stem = path
.file_stem()
.and_then(|s| s.to_str())
.unwrap_or_default()
.to_string();
files.push(EpubFile {
date: date_from_filename(&name),
edition: Edition::from_file_stem(&stem),
name,
modified,
bytes: meta.len(),
});
}
// Newest first: by issue date when the filename carries one, else by mtime.
// Newest issue first, and within an issue the standard edition leads.
files.sort_by(|a, b| {
b.date
.cmp(&a.date)
.then(a.edition.cmp(&b.edition))
.then(b.modified.cmp(&a.modified))
.then(a.name.cmp(&b.name))
});
files.truncate(XTC_FEED_ENTRIES);
// Cap by issue day, not by file: an issue is two entries and truncating
// mid-issue would list one edition without the other.
let mut kept_dates: Vec<Option<Date>> = Vec::new();
files.retain(|file| {
if !kept_dates.contains(&file.date) {
kept_dates.push(file.date);
}
kept_dates.iter().position(|d| *d == file.date) < Some(OPDS_FEED_ISSUES)
});
Ok(files)
}
/// Issue numbers for the dated files, best-effort (the feed is still valid
/// without them). Uses the `db` escape hatch — no bespoke helper in `db.rs`.
async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
async fn issue_numbers(db: &Db, files: &[EpubFile]) -> BTreeMap<Date, i64> {
let mut numbers = BTreeMap::new();
for date in files.iter().filter_map(|f| f.date) {
if numbers.contains_key(&date) {
continue;
}
let row = sqlx::query("SELECT issue_number FROM issues WHERE date = ?")
.bind(date.to_string())
.fetch_optional(db.pool())
@@ -317,13 +351,13 @@ async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
/// Render the Atom/OPDS document (§3.11).
fn render_opds(
files: &[XtcFile],
files: &[EpubFile],
numbers: &BTreeMap<Date, i64>,
public_url: &str,
now: Timestamp,
) -> String {
let base = public_url.trim_end_matches('/');
let self_href = format!("{base}/opds/xtc.xml");
let self_href = format!("{base}{OPDS_PATH}");
let updated = files.first().map(|f| f.modified).unwrap_or(now);
let mut out = String::with_capacity(1024 + files.len() * 512);
@@ -333,8 +367,8 @@ fn render_opds(
xmlns:dc=\"http://purl.org/dc/terms/\" \
xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
);
out.push_str(" <id>urn:daily-epub:xtc</id>\n");
out.push_str(" <title>The Daily EPUB — XTC editions</title>\n");
out.push_str(" <id>urn:daily-epub:issues</id>\n");
out.push_str(" <title>The Daily EPUB</title>\n");
out.push_str(&format!(" <updated>{}</updated>\n", rfc3339(updated)));
out.push_str(" <author><name>The Daily EPUB</name></author>\n");
out.push_str(&format!(
@@ -349,19 +383,25 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
));
for file in files {
// Matches the EPUB's own `dc:title`, so the two editions of one issue
// are told apart in the list rather than showing as the same book.
let title = match file.date {
Some(date) => format!("The Daily EPUB — {date}"),
Some(date) => format!("The Daily EPUB — {date}{}", file.edition.file_suffix()),
None => file.name.clone(),
};
let summary = match file.date.and_then(|d| numbers.get(&d)) {
Some(n) => format!("Issue #{n} · {}", human_bytes(file.bytes)),
None => human_bytes(file.bytes),
let edition_note = match file.edition {
Edition::Standard => "Standard",
Edition::X4 => "Xteink X4",
};
let href = format!("{base}/files/xtc/{}", percent_encode(&file.name));
let summary = match file.date.and_then(|d| numbers.get(&d)) {
Some(n) => format!("Issue #{n} · {edition_note} · {}", human_bytes(file.bytes)),
None => format!("{edition_note} · {}", human_bytes(file.bytes)),
};
let href = format!("{base}/files/epub/{}", percent_encode(&file.name));
out.push_str(" <entry>\n");
out.push_str(&format!(" <title>{}</title>\n", xml_escape(&title)));
out.push_str(&format!(
" <id>urn:daily-epub:xtc:{}</id>\n",
" <id>urn:daily-epub:issue:{}</id>\n",
xml_escape(&percent_encode(&file.name))
));
out.push_str(&format!(
@@ -376,9 +416,12 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
" <summary>{}</summary>\n",
xml_escape(&summary)
));
// The type must be exactly `application/epub+zip`: CrossPoint's OPDS
// parser compares it with `strcmp` and silently drops entries whose
// acquisition link is anything else, reporting "No entries found".
out.push_str(&format!(
" <link rel=\"http://opds-spec.org/acquisition\" href=\"{}\" \
type=\"application/octet-stream\" length=\"{}\"/>\n",
type=\"{EPUB_CONTENT_TYPE}\" length=\"{}\"/>\n",
xml_escape(&href),
file.bytes
));
@@ -430,43 +473,82 @@ fn percent_encode(s: &str) -> String {
out
}
async fn write_atomic(dest: &Path, bytes: &[u8]) -> Result<(), PublishError> {
let dir = dest.parent().unwrap_or_else(|| Path::new("."));
let tmp = dir.join(format!(
".{}.{}.tmp",
dest.file_name()
.and_then(|n| n.to_str())
.unwrap_or("daily-epub"),
std::process::id()
));
tokio::fs::write(&tmp, bytes)
.await
.map_err(PublishError::at(&tmp))?;
tokio::fs::rename(&tmp, dest)
.await
.map_err(PublishError::at(dest))
}
// ---------------------------------------------------------------------------
// Retention (§3.11)
// ---------------------------------------------------------------------------
/// Delete issue files older than `retention_days` from both publish dirs.
/// SQLite history is kept forever — it's the training data (§3.11).
/// Retention sweep over both publish dirs. SQLite history is kept forever —
/// it's the training data (§3.11).
///
/// The two directories are swept on different rules: EPUBs age out after
/// `retention_days`, while XTC is capped at `xtc_retention_count` issues because
/// each one is ~80–100 MB of pre-rendered page bitmaps and the binding
/// constraint is disk rather than age.
///
/// Only files named `The Daily EPUB - YYYY-MM-DD*.{epub,xtc,xtch}` are ever
/// considered; anything else in those directories (including `xtc.xml` and other
/// people's books) is left strictly alone.
/// considered; anything else in those directories (other people's books) is left
/// strictly alone.
pub async fn prune(cfg: &Config, today: Date) -> Result<usize, PublishError> {
let cutoff = today
.checked_sub(jiff::Span::new().days(i64::from(cfg.retention_days)))
.unwrap_or(today);
let mut removed = 0;
for dir in [&cfg.publish.bookorbit_dir, &cfg.publish.xtc_dir] {
removed += prune_dir(dir, cutoff).await?;
}
let mut removed = prune_dir(&cfg.publish.epub_dir, cutoff).await?;
if removed > 0 {
tracing::info!(removed, %cutoff, "retention sweep removed expired issues");
tracing::info!(removed, %cutoff, "retention sweep removed expired EPUBs");
}
let xtc_removed = prune_xtc_dir(&cfg.publish.xtc_dir, cfg.xtc_retention_count as usize).await?;
if xtc_removed > 0 {
tracing::info!(
removed = xtc_removed,
keep = cfg.xtc_retention_count,
"retention sweep trimmed the XTC directory"
);
}
removed += xtc_removed;
Ok(removed)
}
/// Keep only the newest `keep` XTC issues, deleting the rest (§3.11).
///
/// Counted rather than dated so the directory has a hard size ceiling no matter
/// how often `generate` runs.
async fn prune_xtc_dir(dir: &Path, keep: usize) -> Result<usize, PublishError> {
if !dir.exists() {
return Ok(0);
}
let mut entries = tokio::fs::read_dir(dir)
.await
.map_err(PublishError::at(dir))?;
let mut ours: Vec<(Date, PathBuf)> = Vec::new();
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
let name = entry.file_name().to_string_lossy().into_owned();
let extension = Path::new(&name)
.extension()
.and_then(|e| e.to_str())
.unwrap_or_default()
.to_ascii_lowercase();
if !XTC_EXTENSIONS.contains(&extension.as_str()) {
continue;
}
let Some(date) = date_from_filename(&name) else {
continue;
};
if !entry.metadata().await.map(|m| m.is_file()).unwrap_or(false) {
continue;
}
ours.push((date, entry.path()));
}
// Newest first, then drop everything past the cap.
ours.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(&b.1)));
let mut removed = 0;
for (date, path) in ours.into_iter().skip(keep) {
match tokio::fs::remove_file(&path).await {
Ok(()) => {
tracing::info!(path = %path.display(), %date, "pruned an XTC issue past the cap");
removed += 1;
}
Err(e) => tracing::warn!(error = %e, path = %path.display(), "could not prune file"),
}
}
Ok(removed)
}
@@ -520,7 +602,7 @@ mod tests {
fn cfg(dir: &Path) -> Config {
let mut cfg = Config::default();
cfg.publish.bookorbit_dir = dir.join("bookorbit");
cfg.publish.epub_dir = dir.join("bookorbit");
cfg.publish.xtc_dir = dir.join("xtc");
cfg.server.public_url = "https://daily.hallada.net".into();
cfg
@@ -561,7 +643,7 @@ mod tests {
Some(date("2026-08-15"))
);
for foreign in [
"xtc.xml",
"The Daily EPUB - 2026-08-15.xml",
"Moby Dick.epub",
"The Daily EPUB - notadate.epub",
"The Daily EPUB - 2026-08-15.txt",
@@ -634,10 +716,10 @@ mod tests {
paths,
vec![
cfg.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-08-15.epub"),
cfg.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-08-15 (X4).epub"),
]
);
@@ -673,21 +755,37 @@ mod tests {
}
}
fn epub_file(name: &str, edition: Edition, modified: &str, bytes: u64) -> EpubFile {
EpubFile {
date: date_from_filename(name),
edition,
name: name.into(),
modified: ts(modified),
bytes,
}
}
#[test]
fn opds_feed_is_newest_first_with_acquisition_links() {
let files = vec![
XtcFile {
name: "The Daily EPUB - 2026-08-15 (X4).xtch".into(),
date: Some(date("2026-08-15")),
modified: ts("2026-08-15T05:40:00Z"),
bytes: 2_500_000,
},
XtcFile {
name: "The Daily EPUB - 2026-08-14 (X4).xtch".into(),
date: Some(date("2026-08-14")),
modified: ts("2026-08-14T05:40:00Z"),
bytes: 4096,
},
epub_file(
"The Daily EPUB - 2026-08-15.epub",
Edition::Standard,
"2026-08-15T05:40:00Z",
6_500_000,
),
epub_file(
"The Daily EPUB - 2026-08-15 (X4).epub",
Edition::X4,
"2026-08-15T05:40:00Z",
1_700_000,
),
epub_file(
"The Daily EPUB - 2026-08-14.epub",
Edition::Standard,
"2026-08-14T05:40:00Z",
4096,
),
];
let mut numbers = BTreeMap::new();
numbers.insert(date("2026-08-15"), 12);
@@ -700,37 +798,68 @@ mod tests {
assert!(feed.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>"));
assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\""));
assert!(feed.contains("<id>urn:daily-epub:xtc</id>"));
assert!(feed.contains("<id>urn:daily-epub:issues</id>"));
assert!(feed.contains("<updated>2026-08-15T05:40:00Z</updated>"));
assert_eq!(feed.matches("<entry>").count(), 2);
assert_eq!(feed.matches("</entry>").count(), 2);
// Newest first.
let i15 = feed.find("The Daily EPUB — 2026-08-15").unwrap();
let i14 = feed.find("The Daily EPUB — 2026-08-14").unwrap();
assert!(i15 < i14);
// Acquisition link, encoded filename, absolute public URL, size.
assert_eq!(feed.matches("<entry>").count(), 3);
assert_eq!(feed.matches("</entry>").count(), 3);
// Newest issue first; the two editions of one issue are distinguishable
// by title, matching each EPUB's own `dc:title`.
let i15 = feed
.find("<title>The Daily EPUB — 2026-08-15</title>")
.unwrap();
let i15_x4 = feed
.find("<title>The Daily EPUB — 2026-08-15 (X4)</title>")
.unwrap();
let i14 = feed
.find("<title>The Daily EPUB — 2026-08-14</title>")
.unwrap();
assert!(i15 < i15_x4 && i15_x4 < i14);
// CrossPoint compares the acquisition type with `strcmp` against
// `application/epub+zip` and drops the entry on any mismatch (§3.11).
assert_eq!(
feed.matches("type=\"application/epub+zip\"").count(),
3,
"{feed}"
);
assert!(!feed.contains("application/octet-stream"));
assert!(feed.contains(
"<link rel=\"http://opds-spec.org/acquisition\" \
href=\"https://daily.hallada.net/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch\" \
type=\"application/octet-stream\" length=\"2500000\"/>"
href=\"https://daily.hallada.net/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub\" \
type=\"application/epub+zip\" length=\"1700000\"/>"
));
assert!(feed.contains("Issue #12 · 2.4 MB"));
assert!(feed.contains("Issue #12 · Standard · 6.2 MB"));
assert!(feed.contains("Issue #12 · Xteink X4 · 1.6 MB"));
assert!(
feed.contains("<link rel=\"self\" href=\"https://daily.hallada.net/opds/daily.xml\"")
);
assert!(feed.trim_end().ends_with("</feed>"));
assert!(!feed.contains("&<"), "unescaped markup leaked in");
}
#[tokio::test]
async fn write_xtc_opds_lists_only_xtc_files_capped_at_fourteen() {
async fn the_feed_lists_both_editions_of_the_last_fourteen_issues() {
let dir = tempfile::tempdir().unwrap();
let cfg = cfg(dir.path());
std::fs::create_dir_all(&cfg.publish.epub_dir).unwrap();
std::fs::create_dir_all(&cfg.publish.xtc_dir).unwrap();
for day in 1..=20 {
let name = format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch");
std::fs::write(cfg.publish.xtc_dir.join(name), b"x").unwrap();
for edition in Edition::ALL {
let name = issue_filename(date(&format!("2026-08-{day:02}")), edition, "epub");
std::fs::write(cfg.publish.epub_dir.join(name), b"x").unwrap();
}
}
// Non-XTC neighbours must be ignored.
std::fs::write(cfg.publish.xtc_dir.join("README.txt"), b"x").unwrap();
std::fs::write(cfg.publish.xtc_dir.join("cover.epub"), b"x").unwrap();
// Other people's books and our own XTC artifacts must be ignored.
std::fs::write(cfg.publish.epub_dir.join("Moby Dick.epub"), b"x").unwrap();
std::fs::write(cfg.publish.epub_dir.join("metadata.db"), b"x").unwrap();
std::fs::write(
cfg.publish
.xtc_dir
.join("The Daily EPUB - 2026-08-20 (X4).xtch"),
b"x",
)
.unwrap();
let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
.await
@@ -748,64 +877,75 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
.await
.unwrap();
let path = write_xtc_opds(&db, &cfg).await.unwrap();
assert_eq!(path, cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME));
let feed = std::fs::read_to_string(&path).unwrap();
assert_eq!(feed.matches("<entry>").count(), XTC_FEED_ENTRIES);
let feed = build_opds(&db, &cfg).await.unwrap();
// Capped by issue day, so both editions of each of the last 14 survive.
assert_eq!(feed.matches("<entry>").count(), OPDS_FEED_ISSUES * 2);
assert!(feed.contains("2026-08-20"));
assert!(!feed.contains("2026-08-06"), "older than the last 14");
assert!(!feed.contains("README"));
assert!(!feed.contains("cover.epub"));
assert!(!feed.contains("Moby Dick"));
assert!(!feed.contains("metadata.db"));
assert!(feed.contains("Issue #20"));
// Regenerating replaces the file in place.
write_xtc_opds(&db, &cfg).await.unwrap();
let leftovers: Vec<String> = std::fs::read_dir(&cfg.publish.xtc_dir)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
.filter(|n| n.ends_with(".tmp"))
.collect();
assert!(leftovers.is_empty(), "{leftovers:?}");
// XTC is never offered: CrossPoint cannot acquire it (§3.11).
assert!(!feed.contains(".xtch"), "{feed}");
assert!(!feed.contains("/files/xtc/"));
}
#[tokio::test]
async fn prune_only_deletes_old_matching_files() {
async fn prune_ages_out_epubs_but_counts_xtc() {
let dir = tempfile::tempdir().unwrap();
let mut conf = cfg(dir.path());
conf.retention_days = 21;
std::fs::create_dir_all(&conf.publish.bookorbit_dir).unwrap();
conf.xtc_retention_count = 5;
std::fs::create_dir_all(&conf.publish.epub_dir).unwrap();
std::fs::create_dir_all(&conf.publish.xtc_dir).unwrap();
let keep_epub = conf
.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-08-14.epub");
let old_epub = conf
.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-07-01.epub");
let old_x4 = conf
.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-07-01 (X4).epub");
let foreign = conf.publish.bookorbit_dir.join("Moby Dick.epub");
let old_xtc = conf
.publish
.xtc_dir
.join("The Daily EPUB - 2026-07-01 (X4).xtch");
let feed = conf.publish.xtc_dir.join(XTC_OPDS_FILENAME);
for path in [&keep_epub, &old_epub, &old_x4, &foreign, &old_xtc, &feed] {
let foreign = conf.publish.epub_dir.join("Moby Dick.epub");
for path in [&keep_epub, &old_epub, &old_x4, &foreign] {
std::fs::write(path, b"x").unwrap();
}
// Eight consecutive XTC issues, all recent: age would keep every one,
// the count cap keeps the newest five.
let xtc: Vec<PathBuf> = (8..=15)
.map(|day| {
let path = conf
.publish
.xtc_dir
.join(format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch"));
std::fs::write(&path, b"x").unwrap();
path
})
.collect();
let foreign_xtc = conf.publish.xtc_dir.join("Someone Else.xtch");
std::fs::write(&foreign_xtc, b"x").unwrap();
// 2 expired EPUBs + 3 XTC issues past the cap of 5.
let removed = prune(&conf, date("2026-08-15")).await.unwrap();
assert_eq!(removed, 3);
assert_eq!(removed, 5);
assert!(keep_epub.exists());
assert!(foreign.exists(), "never touch other people's books");
assert!(feed.exists(), "the OPDS feed is not an issue file");
assert!(foreign_xtc.exists(), "nor their XTC files");
assert!(!old_epub.exists());
assert!(!old_x4.exists());
assert!(!old_xtc.exists());
// The five newest XTC issues (11th–15th) survive; 8th–10th are gone.
for path in &xtc[..3] {
assert!(!path.exists(), "{} should be pruned", path.display());
}
for path in &xtc[3..] {
assert!(path.exists(), "{} should be kept", path.display());
}
// Idempotent, and tolerant of missing directories.
assert_eq!(prune(&conf, date("2026-08-15")).await.unwrap(), 0);
@@ -844,25 +984,31 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
];
let issue = fake_issue(date("2026-08-15"));
let published = publish_issue(&db, &cfg, &issue, &artifacts, Some(&xtc_src))
let published = publish_issue(&cfg, &issue, &artifacts, Some(&xtc_src))
.await
.unwrap();
assert_eq!(published.epubs.len(), 2);
assert!(published.epubs.iter().all(|a| a.path.exists()));
assert_eq!(published.epubs[1].edition, Edition::X4);
assert!(published.xtc.as_ref().is_some_and(|p| p.exists()));
assert!(published.opds.as_ref().is_some_and(|p| p.exists()));
assert_eq!(published.pruned, 0);
let feed = std::fs::read_to_string(cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)).unwrap();
assert!(feed.contains("out.xtch"));
// The feed is derived from the publish directory, so both editions show
// up without publish having written anything.
let feed = build_opds(&db, &cfg).await.unwrap();
assert!(
feed.contains("The Daily EPUB — 2026-08-15</title>"),
"{feed}"
);
assert!(
feed.contains("The Daily EPUB — 2026-08-15 (X4)</title>"),
"{feed}"
);
assert!(!feed.contains("out.xtch"));
// No XTC artifact is fine — the feed is still regenerated.
let published = publish_issue(&db, &cfg, &issue, &artifacts, None)
.await
.unwrap();
// No XTC artifact is fine — the EPUBs are what the feed lists anyway.
let published = publish_issue(&cfg, &issue, &artifacts, None).await.unwrap();
assert!(published.xtc.is_none());
assert!(published.opds.is_some());
}
#[test]