thallada/the-daily-epub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Serve EPUBs instead XTC in daily OPDS

Browse Source 
It turns out it is currently impossible for the Xteink X4 to download
XTC files from an OPDS server. So I switched the built-in OPDS server in
this binary (daily.hallada.net/opds) to serve the EPUB files instead of
the XTC files. The X4 is pretty capable of reading the X4 edition of the
EPUB that was optimized for it anyways, and I prefer the flexibility of
EPUB, so I might end up eventually deleting the XTC conversion. For now,
I kept the conversion step (in case I ever want to ever try manually
copying them to the device) and I keep a limited number of XTC issues on
the server since they are quite big in filesize. The BookOrbit
integration is now optional since the built-in OPDS server is able to
serve the EPUBs. In my installation, I serve the EPUBs through both. The
daily.hallada.net/opds server is just a little quicker to navigate and
download the EPUBs on the X4 since the BookOrbit OPDS requires diving
into a couple layers of folders before you get to the files.
This commit is contained in:
thallada
2026-08-15 20:24:20 +00:00
parent 9e30c1dcdf
commit 9e27a32fb6
12 changed files with 630 additions and 282 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/config.rs
+36 -6
View File
@@ -51,8 +51,13 @@ pub struct Config {
pub target_article_count: usize,
/// How many articles survive the heuristic pre-filter (§3.5).
pub prefilter_keep: usize,
/// Days of published files kept in the publish dirs (§3.11).
/// Days of published EPUBs kept in `publish.epub_dir` (§3.11).
pub retention_days: u32,
/// How many XTC issues to keep in `publish.xtc_dir` (§3.11).
///
/// Counted, not dated, because an XTCH issue is ~80–100 MB of pre-rendered
/// page bitmaps: the constraint is disk, not age.
pub xtc_retention_count: u32,
/// Hard cost ceiling per run (§3.6 guardrail).
pub max_daily_usd: f64,
/// Include the Wikipedia Current Events section (§3.8).
@@ -81,6 +86,7 @@ impl Default for Config {
target_article_count: 20,
prefilter_keep: 120,
retention_days: 21,
xtc_retention_count: 5,
max_daily_usd: 2.0,
world_briefing: true,
database_path: PathBuf::from("/var/lib/daily-epub/daily-epub.db"),
@@ -195,16 +201,20 @@ impl Default for CurationConfig {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields, default)]
pub struct PublishConfig {
/// BookOrbit "The Daily EPUB" library watched folder.
pub bookorbit_dir: PathBuf,
/// Directory served at `/files/xtc/`.
/// Where both EPUB editions land: the source of the OPDS feed, served at
/// `/files/epub/`, and a BookOrbit watched folder if one is configured.
///
/// Renamed from `bookorbit_dir` once the built-in feed started serving this
/// directory directly — BookOrbit is optional, the directory is not.
pub epub_dir: PathBuf,
/// Directory served at `/files/xtc/`. Not listed in the OPDS feed (§3.11).
pub xtc_dir: PathBuf,
}
impl Default for PublishConfig {
fn default() -> Self {
Self {
bookorbit_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"),
epub_dir: PathBuf::from("/srv/bookorbit/libraries/daily-epub"),
xtc_dir: PathBuf::from("/var/lib/daily-epub/xtc"),
}
}
@@ -274,7 +284,7 @@ pub struct ServerConfig {
pub public_url: String,
/// HMAC key for rating tokens; supply via `DAILY_EPUB_SERVER__HMAC_SECRET`.
pub hmac_secret: Option<String>,
/// Optional Basic auth for `/opds/xtc.xml` and `/files/xtc/`.
/// Optional Basic auth for `/opds/*` and `/files/*`.
pub basic_auth_user: Option<String>,
pub basic_auth_pass: Option<String>,
}
@@ -431,6 +441,26 @@ mod tests {
));
}
/// `publish.bookorbit_dir` was renamed to `publish.epub_dir`. A config still
/// using the old key must fail loudly and name both — silently falling back
/// to the default would publish the issue into the wrong directory, where
/// the OPDS feed would then find nothing.
#[test]
fn the_renamed_publish_key_fails_loudly() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(
&path,
"[publish]\nbookorbit_dir = \"/srv/books\"\nxtc_dir = \"/srv/xtc\"\n",
)
.unwrap();
let err = Config::load(Some(&path)).expect_err("the stale key must be rejected");
let message = err.to_string();
assert!(message.contains("bookorbit_dir"), "{message}");
assert!(message.contains("epub_dir"), "{message}");
}
#[test]
fn shipped_example_config_parses() {
let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml");
src/main.rs
+1 -4
View File
@@ -31,7 +31,7 @@ struct Cli {
enum Command {
/// Build (and publish) one issue.
Generate(GenerateArgs),
/// Run the rating endpoints, XTC OPDS feed and static files.
/// Run the rating endpoints, the OPDS catalog and downloads.
Serve,
/// Taste-profile maintenance.
#[command(subcommand)]
@@ -167,9 +167,6 @@ fn print_outcome(outcome: &GenerateOutcome) {
if let Some(xtc) = &published.xtc {
println!("published: {}", xtc.display());
}
if let Some(opds) = &published.opds {
println!("opds: {}", opds.display());
}
if published.pruned > 0 {
println!("pruned: {} expired files", published.pruned);
}
src/pipeline.rs
+1 -1
View File
@@ -510,7 +510,7 @@ async fn run_stages(
);
None
} else {
let published = publish::publish_issue(db, config, &issue, &artifacts, xtc.as_deref())
let published = publish::publish_issue(config, &issue, &artifacts, xtc.as_deref())
.await
.context("publishing the issue")?;
record_issue(db, &issue, &published)
src/publish.rs
+299 -153
View File
@@ -1,6 +1,9 @@
//! Publishing: BookOrbit watched folder, XTC delivery, OPDS feed, retention
//! Publishing: the EPUB library folder, XTC delivery, OPDS feed, retention
//! (spec §3.11).
//!
//! `publish.epub_dir` holds both EPUB editions and is what the OPDS feed lists;
//! BookOrbit may watch the same folder but nothing here depends on it.
//!
//! Everything here is deliberately dumb about *how* artifacts were produced: the
//! EPUB/XTC stages hand over finished files, this module only copies, indexes and
//! prunes them. Copies are atomic (temp file in the destination directory, then
@@ -8,11 +11,15 @@
//! half-written book.
//!
//! [`crate::pipeline`] ends a non-dry run with one call —
//! `publish_issue(db, config, &issue, &artifacts, xtc_path.as_deref())`, where
//! `publish_issue(config, &issue, &artifacts, xtc_path.as_deref())`, where
//! `artifacts` are the `epub::build_all` outputs and `xtc_path` is
//! `epub::x4::convert`'s output (`None` when the converter is disabled or
//! failed) — and feeds the returned [`Published`] paths into
//! `db.upsert_issue(..., epub_path, x4_path, xtc_path, ...)`.
//!
//! The OPDS feed ([`build_opds`]) is rendered per request by
//! [`crate::server`] rather than written here, and lists **EPUBs only** — see
//! its docs for why XTC cannot be delivered over OPDS.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
@@ -26,14 +33,19 @@ use crate::config::Config;
use crate::db::Db;
use crate::types::{Artifact, Edition, Issue};
/// Filename of the generated static OPDS feed (§3.11).
pub const XTC_OPDS_FILENAME: &str = "xtc.xml";
/// Number of issues listed in the XTC OPDS feed (§3.11).
pub const XTC_FEED_ENTRIES: usize = 14;
/// Number of issue *days* listed in the OPDS feed; each contributes one entry
/// per edition (§3.11).
pub const OPDS_FEED_ISSUES: usize = 14;
/// Every published file starts with this (the retention sweep keys off it).
pub const FILE_PREFIX: &str = "The Daily EPUB - ";
/// Extensions the retention sweep is allowed to delete (§3.11).
pub const PRUNABLE_EXTENSIONS: [&str; 3] = ["epub", "xtc", "xtch"];
/// Extensions of the XTC artifacts, for the counted sweep of `xtc_dir` (§3.11).
pub const XTC_EXTENSIONS: [&str; 2] = ["xtc", "xtch"];
/// Canonical path of the OPDS feed, relative to `server.public_url` (§3.11).
pub const OPDS_PATH: &str = "/opds/daily.xml";
/// The one acquisition type CrossPoint's OPDS parser accepts (§3.11).
pub const EPUB_CONTENT_TYPE: &str = "application/epub+zip";
#[derive(Debug, thiserror::Error)]
pub enum PublishError {
@@ -61,8 +73,6 @@ pub struct Published {
pub epubs: Vec<Artifact>,
/// The XTC artifact's published location, when the converter produced one.
pub xtc: Option<PathBuf>,
/// The regenerated OPDS feed.
pub opds: Option<PathBuf>,
/// How many expired files the retention sweep removed.
pub pruned: usize,
}
@@ -136,7 +146,7 @@ async fn ensure_dir(dir: &Path) -> Result<(), PublishError> {
.map_err(PublishError::at(dir))
}
/// Copy both EPUB editions into the BookOrbit watched folder (§3.11).
/// Copy both EPUB editions into `publish.epub_dir` (§3.11).
///
/// Returns the published paths in the same order as `artifacts`.
pub async fn publish_epubs(
@@ -144,7 +154,7 @@ pub async fn publish_epubs(
issue: &Issue,
cfg: &Config,
) -> Result<Vec<PathBuf>, PublishError> {
let dir = &cfg.publish.bookorbit_dir;
let dir = &cfg.publish.epub_dir;
ensure_dir(dir).await?;
let mut published = Vec::with_capacity(artifacts.len());
for artifact in artifacts {
@@ -159,7 +169,7 @@ pub async fn publish_epubs(
edition = ?artifact.edition,
dest = %dest.display(),
bytes = artifact.bytes,
"published edition to the BookOrbit library"
"published edition to the EPUB library"
);
published.push(dest);
}
@@ -180,12 +190,11 @@ pub async fn publish_xtc(xtc: &Path, cfg: &Config) -> Result<PathBuf, PublishErr
Ok(dest)
}
/// Publish everything one run produced, refresh the OPDS feed and prune (§3.11).
/// Publish everything one run produced and prune (§3.11).
///
/// `xtc` is `None` when the converter is disabled or failed — that is not an
/// error, the X4 falls back to the EPUB edition from BookOrbit.
/// error, since the OPDS feed lists the EPUB editions either way.
pub async fn publish_issue(
db: &Db,
cfg: &Config,
issue: &Issue,
artifacts: &[Artifact],
@@ -208,67 +217,74 @@ pub async fn publish_issue(
Some(src) => Some(publish_xtc(src, cfg).await?),
None => None,
};
let opds = Some(write_xtc_opds(db, cfg).await?);
// The OPDS feed is rendered per request from the publish directory, so
// there is nothing to write here (§3.11).
let pruned = prune(cfg, issue.meta.date).await?;
Ok(Published {
epubs,
xtc,
opds,
pruned,
})
Ok(Published { epubs, xtc, pruned })
}
// ---------------------------------------------------------------------------
// OPDS 1.2 acquisition feed (§3.11)
// ---------------------------------------------------------------------------
/// One published XTC file, as listed in the feed.
/// One published EPUB, as listed in the feed.
#[derive(Debug, Clone, PartialEq, Eq)]
struct XtcFile {
struct EpubFile {
name: String,
date: Option<Date>,
edition: Edition,
modified: Timestamp,
bytes: u64,
}
/// Regenerate the static OPDS 1.2 acquisition feed for the XTC directory:
/// newest first, last [`XTC_FEED_ENTRIES`], entries typed
/// `application/octet-stream` (§3.11).
pub async fn write_xtc_opds(db: &Db, cfg: &Config) -> Result<PathBuf, PublishError> {
let dir = &cfg.publish.xtc_dir;
/// Render the OPDS 1.2 acquisition feed for the EPUB publish directory (§3.11).
///
/// Rendered per request rather than written to disk: the directory is the only
/// source of truth, so the feed cannot go stale behind a failed publish, and
/// there is no generated file for the retention sweep to step around.
///
/// XTC artifacts are deliberately **not** listed. CrossPoint's OPDS browser only
/// acquires links typed `application/epub+zip` and always saves the result with
/// a `.epub` extension, which its reader dispatches on — so an XTC offered here
/// would either be invisible or download into a file that cannot be opened.
pub async fn build_opds(db: &Db, cfg: &Config) -> Result<String, PublishError> {
let dir = &cfg.publish.epub_dir;
ensure_dir(dir).await?;
let files = scan_xtc_dir(dir).await?;
let files = scan_epub_dir(dir).await?;
let numbers = issue_numbers(db, &files).await;
let feed = render_opds(&files, &numbers, &cfg.server.public_url, Timestamp::now());
let dest = dir.join(XTC_OPDS_FILENAME);
write_atomic(&dest, feed.as_bytes()).await?;
tracing::info!(entries = files.len(), dest = %dest.display(), "wrote the XTC OPDS feed");
Ok(dest)
Ok(render_opds(
&files,
&numbers,
&cfg.server.public_url,
Timestamp::now(),
))
}
/// XTC artifacts in `dir`, newest first, capped at [`XTC_FEED_ENTRIES`].
async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
/// Published EPUBs in `dir`, newest first, limited to the last
/// [`OPDS_FEED_ISSUES`] issue days (both editions of each).
async fn scan_epub_dir(dir: &Path) -> Result<Vec<EpubFile>, PublishError> {
let mut entries = tokio::fs::read_dir(dir)
.await
.map_err(PublishError::at(dir))?;
let mut files = Vec::new();
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
let name = entry.file_name().to_string_lossy().into_owned();
let extension = Path::new(&name)
let path = Path::new(&name);
let extension = path
.extension()
.and_then(|e| e.to_str())
.unwrap_or_default()
.to_ascii_lowercase();
if !matches!(extension.as_str(), "xtc" | "xtch") {
// Only our own issues: a shared library may hold other people's books.
if extension != "epub" || date_from_filename(&name).is_none() {
continue;
}
let meta = match entry.metadata().await {
Ok(meta) if meta.is_file() => meta,
Ok(_) => continue,
Err(e) => {
tracing::warn!(error = %e, name, "skipping unreadable XTC file");
tracing::warn!(error = %e, name, "skipping unreadable EPUB");
continue;
}
};
@@ -277,29 +293,47 @@ async fn scan_xtc_dir(dir: &Path) -> Result<Vec<XtcFile>, PublishError> {
.ok()
.and_then(|m| Timestamp::try_from(m).ok())
.unwrap_or_else(Timestamp::now);
files.push(XtcFile {
let stem = path
.file_stem()
.and_then(|s| s.to_str())
.unwrap_or_default()
.to_string();
files.push(EpubFile {
date: date_from_filename(&name),
edition: Edition::from_file_stem(&stem),
name,
modified,
bytes: meta.len(),
});
}
// Newest first: by issue date when the filename carries one, else by mtime.
// Newest issue first, and within an issue the standard edition leads.
files.sort_by(|a, b| {
b.date
.cmp(&a.date)
.then(a.edition.cmp(&b.edition))
.then(b.modified.cmp(&a.modified))
.then(a.name.cmp(&b.name))
});
files.truncate(XTC_FEED_ENTRIES);
// Cap by issue day, not by file: an issue is two entries and truncating
// mid-issue would list one edition without the other.
let mut kept_dates: Vec<Option<Date>> = Vec::new();
files.retain(|file| {
if !kept_dates.contains(&file.date) {
kept_dates.push(file.date);
}
kept_dates.iter().position(|d| *d == file.date) < Some(OPDS_FEED_ISSUES)
});
Ok(files)
}
/// Issue numbers for the dated files, best-effort (the feed is still valid
/// without them). Uses the `db` escape hatch — no bespoke helper in `db.rs`.
async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
async fn issue_numbers(db: &Db, files: &[EpubFile]) -> BTreeMap<Date, i64> {
let mut numbers = BTreeMap::new();
for date in files.iter().filter_map(|f| f.date) {
if numbers.contains_key(&date) {
continue;
}
let row = sqlx::query("SELECT issue_number FROM issues WHERE date = ?")
.bind(date.to_string())
.fetch_optional(db.pool())
@@ -317,13 +351,13 @@ async fn issue_numbers(db: &Db, files: &[XtcFile]) -> BTreeMap<Date, i64> {
/// Render the Atom/OPDS document (§3.11).
fn render_opds(
files: &[XtcFile],
files: &[EpubFile],
numbers: &BTreeMap<Date, i64>,
public_url: &str,
now: Timestamp,
) -> String {
let base = public_url.trim_end_matches('/');
let self_href = format!("{base}/opds/xtc.xml");
let self_href = format!("{base}{OPDS_PATH}");
let updated = files.first().map(|f| f.modified).unwrap_or(now);
let mut out = String::with_capacity(1024 + files.len() * 512);
@@ -333,8 +367,8 @@ fn render_opds(
xmlns:dc=\"http://purl.org/dc/terms/\" \
xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
);
out.push_str(" <id>urn:daily-epub:xtc</id>\n");
out.push_str(" <title>The Daily EPUB — XTC editions</title>\n");
out.push_str(" <id>urn:daily-epub:issues</id>\n");
out.push_str(" <title>The Daily EPUB</title>\n");
out.push_str(&format!(" <updated>{}</updated>\n", rfc3339(updated)));
out.push_str(" <author><name>The Daily EPUB</name></author>\n");
out.push_str(&format!(
@@ -349,19 +383,25 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
));
for file in files {
// Matches the EPUB's own `dc:title`, so the two editions of one issue
// are told apart in the list rather than showing as the same book.
let title = match file.date {
Some(date) => format!("The Daily EPUB — {date}"),
Some(date) => format!("The Daily EPUB — {date}{}", file.edition.file_suffix()),
None => file.name.clone(),
};
let summary = match file.date.and_then(|d| numbers.get(&d)) {
Some(n) => format!("Issue #{n} · {}", human_bytes(file.bytes)),
None => human_bytes(file.bytes),
let edition_note = match file.edition {
Edition::Standard => "Standard",
Edition::X4 => "Xteink X4",
};
let href = format!("{base}/files/xtc/{}", percent_encode(&file.name));
let summary = match file.date.and_then(|d| numbers.get(&d)) {
Some(n) => format!("Issue #{n} · {edition_note} · {}", human_bytes(file.bytes)),
None => format!("{edition_note} · {}", human_bytes(file.bytes)),
};
let href = format!("{base}/files/epub/{}", percent_encode(&file.name));
out.push_str(" <entry>\n");
out.push_str(&format!(" <title>{}</title>\n", xml_escape(&title)));
out.push_str(&format!(
" <id>urn:daily-epub:xtc:{}</id>\n",
" <id>urn:daily-epub:issue:{}</id>\n",
xml_escape(&percent_encode(&file.name))
));
out.push_str(&format!(
@@ -376,9 +416,12 @@ xmlns:opds=\"http://opds-spec.org/2010/catalog\">\n",
" <summary>{}</summary>\n",
xml_escape(&summary)
));
// The type must be exactly `application/epub+zip`: CrossPoint's OPDS
// parser compares it with `strcmp` and silently drops entries whose
// acquisition link is anything else, reporting "No entries found".
out.push_str(&format!(
" <link rel=\"http://opds-spec.org/acquisition\" href=\"{}\" \
type=\"application/octet-stream\" length=\"{}\"/>\n",
type=\"{EPUB_CONTENT_TYPE}\" length=\"{}\"/>\n",
xml_escape(&href),
file.bytes
));
@@ -430,43 +473,82 @@ fn percent_encode(s: &str) -> String {
out
}
async fn write_atomic(dest: &Path, bytes: &[u8]) -> Result<(), PublishError> {
let dir = dest.parent().unwrap_or_else(|| Path::new("."));
let tmp = dir.join(format!(
".{}.{}.tmp",
dest.file_name()
.and_then(|n| n.to_str())
.unwrap_or("daily-epub"),
std::process::id()
));
tokio::fs::write(&tmp, bytes)
.await
.map_err(PublishError::at(&tmp))?;
tokio::fs::rename(&tmp, dest)
.await
.map_err(PublishError::at(dest))
}
// ---------------------------------------------------------------------------
// Retention (§3.11)
// ---------------------------------------------------------------------------
/// Delete issue files older than `retention_days` from both publish dirs.
/// SQLite history is kept forever — it's the training data (§3.11).
/// Retention sweep over both publish dirs. SQLite history is kept forever —
/// it's the training data (§3.11).
///
/// The two directories are swept on different rules: EPUBs age out after
/// `retention_days`, while XTC is capped at `xtc_retention_count` issues because
/// each one is ~80–100 MB of pre-rendered page bitmaps and the binding
/// constraint is disk rather than age.
///
/// Only files named `The Daily EPUB - YYYY-MM-DD*.{epub,xtc,xtch}` are ever
/// considered; anything else in those directories (including `xtc.xml` and other
/// people's books) is left strictly alone.
/// considered; anything else in those directories (other people's books) is left
/// strictly alone.
pub async fn prune(cfg: &Config, today: Date) -> Result<usize, PublishError> {
let cutoff = today
.checked_sub(jiff::Span::new().days(i64::from(cfg.retention_days)))
.unwrap_or(today);
let mut removed = 0;
for dir in [&cfg.publish.bookorbit_dir, &cfg.publish.xtc_dir] {
removed += prune_dir(dir, cutoff).await?;
}
let mut removed = prune_dir(&cfg.publish.epub_dir, cutoff).await?;
if removed > 0 {
tracing::info!(removed, %cutoff, "retention sweep removed expired issues");
tracing::info!(removed, %cutoff, "retention sweep removed expired EPUBs");
}
let xtc_removed = prune_xtc_dir(&cfg.publish.xtc_dir, cfg.xtc_retention_count as usize).await?;
if xtc_removed > 0 {
tracing::info!(
removed = xtc_removed,
keep = cfg.xtc_retention_count,
"retention sweep trimmed the XTC directory"
);
}
removed += xtc_removed;
Ok(removed)
}
/// Keep only the newest `keep` XTC issues, deleting the rest (§3.11).
///
/// Counted rather than dated so the directory has a hard size ceiling no matter
/// how often `generate` runs.
async fn prune_xtc_dir(dir: &Path, keep: usize) -> Result<usize, PublishError> {
if !dir.exists() {
return Ok(0);
}
let mut entries = tokio::fs::read_dir(dir)
.await
.map_err(PublishError::at(dir))?;
let mut ours: Vec<(Date, PathBuf)> = Vec::new();
while let Some(entry) = entries.next_entry().await.map_err(PublishError::at(dir))? {
let name = entry.file_name().to_string_lossy().into_owned();
let extension = Path::new(&name)
.extension()
.and_then(|e| e.to_str())
.unwrap_or_default()
.to_ascii_lowercase();
if !XTC_EXTENSIONS.contains(&extension.as_str()) {
continue;
}
let Some(date) = date_from_filename(&name) else {
continue;
};
if !entry.metadata().await.map(|m| m.is_file()).unwrap_or(false) {
continue;
}
ours.push((date, entry.path()));
}
// Newest first, then drop everything past the cap.
ours.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(&b.1)));
let mut removed = 0;
for (date, path) in ours.into_iter().skip(keep) {
match tokio::fs::remove_file(&path).await {
Ok(()) => {
tracing::info!(path = %path.display(), %date, "pruned an XTC issue past the cap");
removed += 1;
}
Err(e) => tracing::warn!(error = %e, path = %path.display(), "could not prune file"),
}
}
Ok(removed)
}
@@ -520,7 +602,7 @@ mod tests {
fn cfg(dir: &Path) -> Config {
let mut cfg = Config::default();
cfg.publish.bookorbit_dir = dir.join("bookorbit");
cfg.publish.epub_dir = dir.join("bookorbit");
cfg.publish.xtc_dir = dir.join("xtc");
cfg.server.public_url = "https://daily.hallada.net".into();
cfg
@@ -561,7 +643,7 @@ mod tests {
Some(date("2026-08-15"))
);
for foreign in [
"xtc.xml",
"The Daily EPUB - 2026-08-15.xml",
"Moby Dick.epub",
"The Daily EPUB - notadate.epub",
"The Daily EPUB - 2026-08-15.txt",
@@ -634,10 +716,10 @@ mod tests {
paths,
vec![
cfg.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-08-15.epub"),
cfg.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-08-15 (X4).epub"),
]
);
@@ -673,21 +755,37 @@ mod tests {
}
}
fn epub_file(name: &str, edition: Edition, modified: &str, bytes: u64) -> EpubFile {
EpubFile {
date: date_from_filename(name),
edition,
name: name.into(),
modified: ts(modified),
bytes,
}
}
#[test]
fn opds_feed_is_newest_first_with_acquisition_links() {
let files = vec![
XtcFile {
name: "The Daily EPUB - 2026-08-15 (X4).xtch".into(),
date: Some(date("2026-08-15")),
modified: ts("2026-08-15T05:40:00Z"),
bytes: 2_500_000,
},
XtcFile {
name: "The Daily EPUB - 2026-08-14 (X4).xtch".into(),
date: Some(date("2026-08-14")),
modified: ts("2026-08-14T05:40:00Z"),
bytes: 4096,
},
epub_file(
"The Daily EPUB - 2026-08-15.epub",
Edition::Standard,
"2026-08-15T05:40:00Z",
6_500_000,
),
epub_file(
"The Daily EPUB - 2026-08-15 (X4).epub",
Edition::X4,
"2026-08-15T05:40:00Z",
1_700_000,
),
epub_file(
"The Daily EPUB - 2026-08-14.epub",
Edition::Standard,
"2026-08-14T05:40:00Z",
4096,
),
];
let mut numbers = BTreeMap::new();
numbers.insert(date("2026-08-15"), 12);
@@ -700,37 +798,68 @@ mod tests {
assert!(feed.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>"));
assert!(feed.contains("<feed xmlns=\"http://www.w3.org/2005/Atom\""));
assert!(feed.contains("<id>urn:daily-epub:xtc</id>"));
assert!(feed.contains("<id>urn:daily-epub:issues</id>"));
assert!(feed.contains("<updated>2026-08-15T05:40:00Z</updated>"));
assert_eq!(feed.matches("<entry>").count(), 2);
assert_eq!(feed.matches("</entry>").count(), 2);
// Newest first.
let i15 = feed.find("The Daily EPUB — 2026-08-15").unwrap();
let i14 = feed.find("The Daily EPUB — 2026-08-14").unwrap();
assert!(i15 < i14);
// Acquisition link, encoded filename, absolute public URL, size.
assert_eq!(feed.matches("<entry>").count(), 3);
assert_eq!(feed.matches("</entry>").count(), 3);
// Newest issue first; the two editions of one issue are distinguishable
// by title, matching each EPUB's own `dc:title`.
let i15 = feed
.find("<title>The Daily EPUB — 2026-08-15</title>")
.unwrap();
let i15_x4 = feed
.find("<title>The Daily EPUB — 2026-08-15 (X4)</title>")
.unwrap();
let i14 = feed
.find("<title>The Daily EPUB — 2026-08-14</title>")
.unwrap();
assert!(i15 < i15_x4 && i15_x4 < i14);
// CrossPoint compares the acquisition type with `strcmp` against
// `application/epub+zip` and drops the entry on any mismatch (§3.11).
assert_eq!(
feed.matches("type=\"application/epub+zip\"").count(),
3,
"{feed}"
);
assert!(!feed.contains("application/octet-stream"));
assert!(feed.contains(
"<link rel=\"http://opds-spec.org/acquisition\" \
href=\"https://daily.hallada.net/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.xtch\" \
type=\"application/octet-stream\" length=\"2500000\"/>"
href=\"https://daily.hallada.net/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20%28X4%29.epub\" \
type=\"application/epub+zip\" length=\"1700000\"/>"
));
assert!(feed.contains("Issue #12 · 2.4 MB"));
assert!(feed.contains("Issue #12 · Standard · 6.2 MB"));
assert!(feed.contains("Issue #12 · Xteink X4 · 1.6 MB"));
assert!(
feed.contains("<link rel=\"self\" href=\"https://daily.hallada.net/opds/daily.xml\"")
);
assert!(feed.trim_end().ends_with("</feed>"));
assert!(!feed.contains("&<"), "unescaped markup leaked in");
}
#[tokio::test]
async fn write_xtc_opds_lists_only_xtc_files_capped_at_fourteen() {
async fn the_feed_lists_both_editions_of_the_last_fourteen_issues() {
let dir = tempfile::tempdir().unwrap();
let cfg = cfg(dir.path());
std::fs::create_dir_all(&cfg.publish.epub_dir).unwrap();
std::fs::create_dir_all(&cfg.publish.xtc_dir).unwrap();
for day in 1..=20 {
let name = format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch");
std::fs::write(cfg.publish.xtc_dir.join(name), b"x").unwrap();
for edition in Edition::ALL {
let name = issue_filename(date(&format!("2026-08-{day:02}")), edition, "epub");
std::fs::write(cfg.publish.epub_dir.join(name), b"x").unwrap();
}
}
// Non-XTC neighbours must be ignored.
std::fs::write(cfg.publish.xtc_dir.join("README.txt"), b"x").unwrap();
std::fs::write(cfg.publish.xtc_dir.join("cover.epub"), b"x").unwrap();
// Other people's books and our own XTC artifacts must be ignored.
std::fs::write(cfg.publish.epub_dir.join("Moby Dick.epub"), b"x").unwrap();
std::fs::write(cfg.publish.epub_dir.join("metadata.db"), b"x").unwrap();
std::fs::write(
cfg.publish
.xtc_dir
.join("The Daily EPUB - 2026-08-20 (X4).xtch"),
b"x",
)
.unwrap();
let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
.await
@@ -748,64 +877,75 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
.await
.unwrap();
let path = write_xtc_opds(&db, &cfg).await.unwrap();
assert_eq!(path, cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME));
let feed = std::fs::read_to_string(&path).unwrap();
assert_eq!(feed.matches("<entry>").count(), XTC_FEED_ENTRIES);
let feed = build_opds(&db, &cfg).await.unwrap();
// Capped by issue day, so both editions of each of the last 14 survive.
assert_eq!(feed.matches("<entry>").count(), OPDS_FEED_ISSUES * 2);
assert!(feed.contains("2026-08-20"));
assert!(!feed.contains("2026-08-06"), "older than the last 14");
assert!(!feed.contains("README"));
assert!(!feed.contains("cover.epub"));
assert!(!feed.contains("Moby Dick"));
assert!(!feed.contains("metadata.db"));
assert!(feed.contains("Issue #20"));
// Regenerating replaces the file in place.
write_xtc_opds(&db, &cfg).await.unwrap();
let leftovers: Vec<String> = std::fs::read_dir(&cfg.publish.xtc_dir)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
.filter(|n| n.ends_with(".tmp"))
.collect();
assert!(leftovers.is_empty(), "{leftovers:?}");
// XTC is never offered: CrossPoint cannot acquire it (§3.11).
assert!(!feed.contains(".xtch"), "{feed}");
assert!(!feed.contains("/files/xtc/"));
}
#[tokio::test]
async fn prune_only_deletes_old_matching_files() {
async fn prune_ages_out_epubs_but_counts_xtc() {
let dir = tempfile::tempdir().unwrap();
let mut conf = cfg(dir.path());
conf.retention_days = 21;
std::fs::create_dir_all(&conf.publish.bookorbit_dir).unwrap();
conf.xtc_retention_count = 5;
std::fs::create_dir_all(&conf.publish.epub_dir).unwrap();
std::fs::create_dir_all(&conf.publish.xtc_dir).unwrap();
let keep_epub = conf
.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-08-14.epub");
let old_epub = conf
.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-07-01.epub");
let old_x4 = conf
.publish
.bookorbit_dir
.epub_dir
.join("The Daily EPUB - 2026-07-01 (X4).epub");
let foreign = conf.publish.bookorbit_dir.join("Moby Dick.epub");
let old_xtc = conf
.publish
.xtc_dir
.join("The Daily EPUB - 2026-07-01 (X4).xtch");
let feed = conf.publish.xtc_dir.join(XTC_OPDS_FILENAME);
for path in [&keep_epub, &old_epub, &old_x4, &foreign, &old_xtc, &feed] {
let foreign = conf.publish.epub_dir.join("Moby Dick.epub");
for path in [&keep_epub, &old_epub, &old_x4, &foreign] {
std::fs::write(path, b"x").unwrap();
}
// Eight consecutive XTC issues, all recent: age would keep every one,
// the count cap keeps the newest five.
let xtc: Vec<PathBuf> = (8..=15)
.map(|day| {
let path = conf
.publish
.xtc_dir
.join(format!("The Daily EPUB - 2026-08-{day:02} (X4).xtch"));
std::fs::write(&path, b"x").unwrap();
path
})
.collect();
let foreign_xtc = conf.publish.xtc_dir.join("Someone Else.xtch");
std::fs::write(&foreign_xtc, b"x").unwrap();
// 2 expired EPUBs + 3 XTC issues past the cap of 5.
let removed = prune(&conf, date("2026-08-15")).await.unwrap();
assert_eq!(removed, 3);
assert_eq!(removed, 5);
assert!(keep_epub.exists());
assert!(foreign.exists(), "never touch other people's books");
assert!(feed.exists(), "the OPDS feed is not an issue file");
assert!(foreign_xtc.exists(), "nor their XTC files");
assert!(!old_epub.exists());
assert!(!old_x4.exists());
assert!(!old_xtc.exists());
// The five newest XTC issues (11th–15th) survive; 8th–10th are gone.
for path in &xtc[..3] {
assert!(!path.exists(), "{} should be pruned", path.display());
}
for path in &xtc[3..] {
assert!(path.exists(), "{} should be kept", path.display());
}
// Idempotent, and tolerant of missing directories.
assert_eq!(prune(&conf, date("2026-08-15")).await.unwrap(), 0);
@@ -844,25 +984,31 @@ type=\"application/octet-stream\" length=\"2500000\"/>"
];
let issue = fake_issue(date("2026-08-15"));
let published = publish_issue(&db, &cfg, &issue, &artifacts, Some(&xtc_src))
let published = publish_issue(&cfg, &issue, &artifacts, Some(&xtc_src))
.await
.unwrap();
assert_eq!(published.epubs.len(), 2);
assert!(published.epubs.iter().all(|a| a.path.exists()));
assert_eq!(published.epubs[1].edition, Edition::X4);
assert!(published.xtc.as_ref().is_some_and(|p| p.exists()));
assert!(published.opds.as_ref().is_some_and(|p| p.exists()));
assert_eq!(published.pruned, 0);
let feed = std::fs::read_to_string(cfg.publish.xtc_dir.join(XTC_OPDS_FILENAME)).unwrap();
assert!(feed.contains("out.xtch"));
// The feed is derived from the publish directory, so both editions show
// up without publish having written anything.
let feed = build_opds(&db, &cfg).await.unwrap();
assert!(
feed.contains("The Daily EPUB — 2026-08-15</title>"),
"{feed}"
);
assert!(
feed.contains("The Daily EPUB — 2026-08-15 (X4)</title>"),
"{feed}"
);
assert!(!feed.contains("out.xtch"));
// No XTC artifact is fine — the feed is still regenerated.
let published = publish_issue(&db, &cfg, &issue, &artifacts, None)
.await
.unwrap();
// No XTC artifact is fine — the EPUBs are what the feed lists anyway.
let published = publish_issue(&cfg, &issue, &artifacts, None).await.unwrap();
assert!(published.xtc.is_none());
assert!(published.opds.is_some());
}
#[test]
src/server.rs
+89 -35
View File
@@ -1,4 +1,4 @@
//! axum server: rating endpoints, XTC OPDS, static files (spec §3.9, §3.12).
//! axum server: rating endpoints, the OPDS catalog, downloads (spec §3.9, §3.12).
//!
//! Rating links must work from an e-reader's built-in browser, so every rating
//! endpoint is a `GET` and the response is a tiny e-ink-sized HTML page.
@@ -7,8 +7,9 @@
//! | route | behaviour |
//! |---|---|
//! | `GET /r/{date}/{article_id}/{vote}?t=` | verify HMAC, upsert rating, rebuild feed priors |
//! | `GET /opds/xtc.xml` | static OPDS 1.2 acquisition feed from `publish.xtc_dir` |
//! | `GET /files/xtc/{name}` | XTC artifact download (no path traversal) |
//! | `GET /opds/daily.xml` (also `/opds`, `/opds/`) | OPDS 1.2 acquisition feed over `publish.epub_dir` |
//! | `GET /files/epub/{name}` | EPUB download — what the feed's acquisition links point at |
//! | `GET /files/xtc/{name}` | XTC artifact download, unlisted (no path traversal) |
//! | `GET /healthz` | liveness |
//! | `GET /issues.json` | the last 30 run reports, newest first |
//!
@@ -81,16 +82,18 @@ pub use crate::auth::{constant_time_eq, rating_token, rating_url, verify_token};
// Router (§3.12)
// ---------------------------------------------------------------------------
/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/xtc.xml`,
/// `/files/xtc/{name}`, `/healthz`, `/issues.json`, with `tower-http` tracing (§3.12).
/// Build the router: `/r/{date}/{article_id}/{vote}`, `/opds/daily.xml`,
/// `/files/epub/{name}`, `/files/xtc/{name}`, `/healthz`, `/issues.json`, with
/// `tower-http` tracing (§3.12).
pub fn router(state: AppState) -> Router {
Router::new()
.route("/r/{date}/{article_id}/{vote}", get(handle_rating))
.route("/opds/xtc.xml", get(handle_opds))
.route(crate::publish::OPDS_PATH, get(handle_opds))
// OPDS browsers are typed into by hand on a 6" e-ink keyboard: serve the
// same feed from the catalog root so a URL without the filename works.
.route("/opds", get(handle_opds))
.route("/opds/", get(handle_opds))
.route("/files/epub/{name}", get(handle_epub_file))
.route("/files/xtc/{name}", get(handle_xtc_file))
.route("/healthz", get(handle_healthz))
.route("/issues.json", get(handle_issues_json))
@@ -284,44 +287,64 @@ async fn handle_rating(
)
}
/// `GET /opds/xtc.xml` — the static feed written by [`crate::publish`] (§3.11).
/// `GET /opds/daily.xml` — both EPUB editions of the last issues, newest first,
/// rendered from the publish directory on each request (§3.11).
async fn handle_opds(State(state): State<AppState>, headers: HeaderMap) -> Response {
if let Some(challenge) = check_basic_auth(&state.config, &headers) {
return challenge;
}
let path = state
.config
.publish
.xtc_dir
.join(crate::publish::XTC_OPDS_FILENAME);
match tokio::fs::read(&path).await {
Ok(bytes) => (
match crate::publish::build_opds(&state.db, &state.config).await {
Ok(feed) => (
StatusCode::OK,
[
(header::CONTENT_TYPE, OPDS_CONTENT_TYPE),
(header::CACHE_CONTROL, "no-cache"),
],
bytes,
feed,
)
.into_response(),
Err(e) => {
tracing::warn!(error = %e, path = %path.display(), "no XTC OPDS feed yet");
(StatusCode::NOT_FOUND, "no feed yet").into_response()
tracing::error!(error = %e, "could not build the OPDS feed");
(
StatusCode::INTERNAL_SERVER_ERROR,
"could not build the feed",
)
.into_response()
}
}
}
/// `GET /files/epub/{name}` — download one published EPUB; this is what the
/// OPDS acquisition links point at (§3.11).
async fn handle_epub_file(
State(state): State<AppState>,
Path(name): Path<String>,
headers: HeaderMap,
) -> Response {
let dir = state.config.publish.epub_dir.clone();
serve_file(&state, &dir, &name, &headers).await
}
/// `GET /files/xtc/{name}` — download one XTC artifact (§3.11).
///
/// Not listed in the OPDS feed — CrossPoint's browser cannot acquire XTC — but
/// kept so the artifacts can still be fetched by URL for sideloading.
async fn handle_xtc_file(
State(state): State<AppState>,
Path(name): Path<String>,
headers: HeaderMap,
) -> Response {
if let Some(challenge) = check_basic_auth(&state.config, &headers) {
let dir = state.config.publish.xtc_dir.clone();
serve_file(&state, &dir, &name, &headers).await
}
/// Stream one file out of `dir`, behind the OPDS Basic auth (§3.11).
async fn serve_file(state: &AppState, dir: &FsPath, name: &str, headers: &HeaderMap) -> Response {
if let Some(challenge) = check_basic_auth(&state.config, headers) {
return challenge;
}
let Some(path) = safe_join(&state.config.publish.xtc_dir, &name) else {
tracing::warn!(name, "rejected an unsafe XTC file name");
let Some(path) = safe_join(dir, name) else {
tracing::warn!(name, "rejected an unsafe file name");
return (StatusCode::BAD_REQUEST, "bad file name").into_response();
};
// An XTCH issue is a pre-rendered page bitmap per page — ~100 MB for a full
@@ -332,12 +355,14 @@ async fn handle_xtc_file(
(file, len)
}
Err(e) => {
tracing::warn!(error = %e, path = %path.display(), "XTC file not found");
tracing::warn!(error = %e, path = %path.display(), "file not found");
return (StatusCode::NOT_FOUND, "not found").into_response();
}
};
let content_type = if name.ends_with(".xml") {
OPDS_CONTENT_TYPE
// CrossPoint dispatches on the saved file's extension, not on this header,
// but Calibre and KOReader both use it.
let content_type = if name.ends_with(".epub") {
crate::publish::EPUB_CONTENT_TYPE
} else {
"application/octet-stream"
};
@@ -629,7 +654,9 @@ mod tests {
async fn start(with_auth: bool) -> TestServer {
let dir = tempfile::tempdir().unwrap();
let xtc_dir = dir.path().join("xtc");
let epub_dir = dir.path().join("epub");
std::fs::create_dir_all(&xtc_dir).unwrap();
std::fs::create_dir_all(&epub_dir).unwrap();
let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
.await
.unwrap();
@@ -637,6 +664,8 @@ mod tests {
let mut config = Config::default();
config.server.hmac_secret = Some(VECTOR_SECRET.into());
config.publish.xtc_dir = xtc_dir;
config.publish.epub_dir = epub_dir;
config.server.public_url = "https://daily.hallada.net".into();
if with_auth {
config.server.basic_auth_user = Some("opds".into());
config.server.basic_auth_pass = Some("hunter2".into());
@@ -663,6 +692,10 @@ mod tests {
self._dir.path().join("xtc")
}
fn epub_dir(&self) -> PathBuf {
self._dir.path().join("epub")
}
async fn seed_article(&self) -> ArticleId {
let entry = crate::types::Entry {
id: 1,
@@ -847,11 +880,10 @@ mod tests {
#[tokio::test]
async fn opds_and_files_are_served_behind_basic_auth() {
let server = TestServer::start(true).await;
std::fs::write(
server.xtc_dir().join(crate::publish::XTC_OPDS_FILENAME),
"<feed/>",
)
.unwrap();
for edition in crate::types::Edition::ALL {
let name = crate::publish::issue_filename(date(), edition, "epub");
std::fs::write(server.epub_dir().join(name), b"EPUB").unwrap();
}
std::fs::write(
server
.xtc_dir()
@@ -861,7 +893,7 @@ mod tests {
.unwrap();
let res = client()
.get(format!("{}/opds/xtc.xml", server.base))
.get(format!("{}/opds/daily.xml", server.base))
.send()
.await
.unwrap();
@@ -875,7 +907,7 @@ mod tests {
);
let res = client()
.get(format!("{}/opds/xtc.xml", server.base))
.get(format!("{}/opds/daily.xml", server.base))
.basic_auth("opds", Some("wrong"))
.send()
.await
@@ -883,7 +915,7 @@ mod tests {
assert_eq!(res.status(), 401);
let res = client()
.get(format!("{}/opds/xtc.xml", server.base))
.get(format!("{}/opds/daily.xml", server.base))
.basic_auth("opds", Some("hunter2"))
.send()
.await
@@ -895,7 +927,11 @@ mod tests {
.unwrap()
.starts_with("application/atom+xml")
);
assert_eq!(res.text().await.unwrap(), "<feed/>");
let feed = res.text().await.unwrap();
assert_eq!(feed.matches("<entry>").count(), 2, "{feed}");
assert_eq!(feed.matches("application/epub+zip").count(), 2, "{feed}");
// XTC exists on disk but is never advertised (§3.11).
assert!(!feed.contains(".xtch"), "{feed}");
// The catalog root serves the same feed, behind the same auth.
for alias in ["/opds", "/opds/"] {
@@ -912,9 +948,29 @@ mod tests {
.await
.unwrap();
assert_eq!(res.status(), 200, "{alias}");
assert_eq!(res.text().await.unwrap(), "<feed/>", "{alias}");
assert_eq!(res.text().await.unwrap(), feed, "{alias}");
}
// The acquisition link resolves, typed as an EPUB.
let res = client()
.get(format!(
"{}/files/epub/The%20Daily%20EPUB%20-%202026-08-15%20(X4).epub",
server.base
))
.basic_auth("opds", Some("hunter2"))
.send()
.await
.unwrap();
assert_eq!(res.status(), 200);
// CrossPoint needs the size up front to show download progress.
assert_eq!(res.content_length(), Some(4));
assert_eq!(
res.headers()[header::CONTENT_TYPE].to_str().unwrap(),
"application/epub+zip"
);
assert_eq!(res.bytes().await.unwrap().as_ref(), b"EPUB");
// XTC is still fetchable by URL for sideloading, just not listed.
let res = client()
.get(format!(
"{}/files/xtc/The%20Daily%20EPUB%20-%202026-08-15%20(X4).xtch",
@@ -925,8 +981,6 @@ mod tests {
.await
.unwrap();
assert_eq!(res.status(), 200);
// CrossPoint needs the size up front to show download progress.
assert_eq!(res.content_length(), Some(4));
assert_eq!(res.bytes().await.unwrap().as_ref(), b"XTCH");
// Ratings are not behind auth (the token is the credential).
src/types.rs
+19 -1
View File
@@ -411,7 +411,10 @@ pub const WORLD_BRIEFING_SECTION: &str = "World Briefing";
// ---------------------------------------------------------------------------
/// Which of the two editions is being built (§3.10).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
///
/// The declaration order is the listing order: standard first, then X4. The
/// OPDS feed sorts on it.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum Edition {
/// 1200px images, full CSS.
@@ -421,6 +424,9 @@ pub enum Edition {
}
impl Edition {
/// Every edition, in the order they are listed and built (§3.10).
pub const ALL: [Edition; 2] = [Edition::Standard, Edition::X4];
/// Filename suffix: `""` / `" (X4)"` (§3.11).
pub fn file_suffix(self) -> &'static str {
match self {
@@ -428,6 +434,18 @@ impl Edition {
Edition::X4 => " (X4)",
}
}
/// Recover the edition from a published filename's stem (§3.11).
///
/// The OPDS feed is rebuilt by scanning the publish directory, so the
/// filename is the only record of which edition a file is.
pub fn from_file_stem(stem: &str) -> Edition {
if stem.ends_with(Edition::X4.file_suffix()) {
Edition::X4
} else {
Edition::Standard
}
}
}
/// Issue-level metadata rendered on the cover, front page and OPF (§3.10).