Compare commits

...
4 Commits
Author SHA1 Message Date
thalladaandClaude Opus 5.5 7851d8e231 Add an X4 preview tool that renders pages with the real XTC converter
Builds both editions around one article (a URL through real extraction, or
an HTML file), runs the XTC converter on the X4 edition and decodes every
page to PNG. --convert re-renders an existing EPUB, for iterating on a
stylesheet. The decoder follows the converter's encoder, which writes the
level's low bit to the first plane, rather than the spec's table.

The device itself reads the X4 EPUB through its firmware, which lays out
differently. Check those layouts with the firmware's own host tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 19:53:06 +00:00
thalladaandClaude Opus 5.5 5abf062c24 Add rebuild: republish an issue's EPUBs from its stored content
`daily-epub rebuild --date D` (and the rebuild-YYYY-MM-DD job) takes the
issues.issue_json snapshot and restores each article's body from its row.
It then builds both editions and the XTC, publishes them under the same
names and records the issue as a run would. Nothing is curated again or
refreshed from live rows, so headlines, summaries, social lines and rating
links stay as published; only images are downloaded again. It takes the
run lock, opens the database without migrating, and the job is marked
dangerous like generate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 19:53:06 +00:00
thalladaandClaude Opus 5.5 9e521ddaef Lay out X4 code blocks as one paragraph per line
The X4 firmwares (CrossPoint and forks such as CrossInk) ignore
white-space and treat <pre> as inline. Its margins never applied, so the
paragraph after a listing started on the next line. CrossInk also glued
the first code line onto the paragraph before it. The simplifier now makes
each listing line a <p class="code-line">, the one path both firmwares lay
out identically. The first and last lines carry the block's margins.
Inline tags open across a line break are closed and reopened on the next
line.

A blank source line used to be a paragraph of its own. With the firmware's
default paragraph spacing that read as a double newline, so blank lines
now become a half-line gap (p.code-gap) on the line before.

The X4 theme matches the standard one but uses only the device's exact
levels (#000, #555, #aaa), so glyphs are not dithered. Inline code is
display: inline for the XTC converter's CREngine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 19:53:06 +00:00
thalladaandClaude Opus 5.5 50551114cd Richer monochrome syntax theme and code-block layout in the standard EPUB
Following zenbones and lackluster, the theme keeps what the code names in
ink (functions bold, types italic) and moves the language's own syntax to
grey (keywords bold grey, literals and punctuation grey). Comments are the
lightest. Weight and slant carry the distinctions, so nothing depends on
colour.

Code blocks stay left-aligned inside a <figure>, which centres for image
captions, and have explicit margins. Inline code is display: inline because
CREngine readers (KOReader, PocketBook) make it a block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 19:53:06 +00:00
7 changed files with 651 additions and 30 deletions

No files matched your search

+131
View File
@@ -0,0 +1,131 @@
//! Preview an article the way the Xteink X4 will show it.
//!
//! Browsers are a poor stand-in for the X4: its pages are rendered ahead of
//! time by the XTC converter's CREngine, which has its own defaults (`code` is a
//! block, `pre` has no margin) and only four grey levels. This builds both
//! EPUB editions around one article, runs the real converter on the X4 one and
//! decodes every page to `page-NNN.png`.
//!
//! ```text
//! cargo run --example x4_preview -- <article URL | HTML file> <out dir>
//! cargo run --example x4_preview -- --convert <x4.epub> <out dir>
//! ```
//!
//! A URL goes through the real extraction path; an HTML file is used as the
//! stored body as-is. `--convert` skips the build, for iterating on a
//! stylesheet inside an unzipped EPUB. Needs the converter at
//! `/opt/epub-to-xtc-converter` and runs it with `xtc-settings.example.json`.
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use daily_epub::config::Config;
use daily_epub::epub;
use daily_epub::extract::Extractor;
use daily_epub::types::Edition;
const CONVERTER: &str = "/opt/epub-to-xtc-converter/cli/index.js";
const SETTINGS: &str = "xtc-settings.example.json";
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let args: Vec<String> = std::env::args().skip(1).collect();
match args.as_slice() {
[flag, epub, out] if flag == "--convert" => {
let pages = convert(Path::new(epub), Path::new(out))?;
println!("{pages} pages in {out}");
}
[source, out] => {
let out = PathBuf::from(out);
std::fs::create_dir_all(&out)?;
let html = if source.starts_with("http") {
let extractor = Extractor::new(reqwest::Client::new(), vec![]);
let page = extractor.fetch_readable(source).await?;
extractor.finish_readable(source, &page).content_html
} else {
std::fs::read_to_string(source)?
};
std::fs::write(out.join("stored.html"), &html)?;
let mut issue = epub::fixtures::issue();
issue.lineup.picks.truncate(1);
issue.lineup.picks[0].article.content_html = html;
let config = Config::default();
for edition in [Edition::Standard, Edition::X4] {
let artifact =
epub::build_edition_with_images(&issue, edition, &config, &out, &[])?;
println!("{edition:?}: {}", artifact.path.display());
if edition == Edition::X4 {
let pages = convert(&artifact.path, &out)?;
println!("{pages} X4 pages in {}", out.display());
}
}
}
_ => anyhow::bail!(
"usage: x4_preview <URL | HTML file> <out dir> | --convert <epub> <out dir>"
),
}
Ok(())
}
/// Run the converter on `epub` and decode its pages into `out`.
fn convert(epub: &Path, out: &Path) -> anyhow::Result<usize> {
std::fs::create_dir_all(out)?;
let xtch = out.join("x4.xtch");
let status = Command::new("node")
.args([CONVERTER, "convert"])
.arg(epub)
.arg("-o")
.arg(&xtch)
.args(["-f", "xtch", "-c", SETTINGS])
.stdout(Stdio::null())
.status()?;
anyhow::ensure!(status.success(), "converter failed: {status}");
decode_xtch(&xtch, out)
}
fn u16_at(bytes: &[u8], at: usize) -> usize {
u16::from_le_bytes([bytes[at], bytes[at + 1]]) as usize
}
fn u64_at(bytes: &[u8], at: usize) -> usize {
u64::from_le_bytes(bytes[at..at + 8].try_into().unwrap()) as usize
}
/// Decode every page of an XTCH container to a PNG.
///
/// Layout per the converter's `docs/xtc-format-spec.md`: the page index is at
/// the offset in header bytes 0x18..0x20, 16 bytes per page (offset, size,
/// width, height). Each page is a 22-byte header and two bit planes stored
/// column by column, right to left, 8 vertical pixels per byte. The plane
/// order follows the encoder (`cli/encoder.js`), not the spec's table: the
/// first plane holds the level's low bit, and 0b01 is dark grey.
fn decode_xtch(path: &Path, out: &Path) -> anyhow::Result<usize> {
let bytes = std::fs::read(path)?;
let pages = u16_at(&bytes, 0x06);
let index = u64_at(&bytes, 0x18);
for page in 0..pages {
let entry = index + page * 16;
let (offset, width, height) = (
u64_at(&bytes, entry),
u16_at(&bytes, entry + 12),
u16_at(&bytes, entry + 14),
);
let plane = (width * height).div_ceil(8);
let column = height.div_ceil(8);
let data = &bytes[offset + 22..];
let mut image = image::GrayImage::new(width as u32, height as u32);
for x in 0..width {
for y in 0..height {
let byte = (width - 1 - x) * column + y / 8;
let bit = 7 - (y % 8);
let low = (data[byte] >> bit) & 1;
let high = (data[plane + byte] >> bit) & 1;
let shade = [255, 85, 170, 0][usize::from(high << 1 | low)];
image.put_pixel(x as u32, y as u32, image::Luma([shade]));
}
}
image.save(out.join(format!("page-{page:03}.png")))?;
}
Ok(pages)
}
+76 -10
View File
@@ -160,7 +160,16 @@ figcaption,
text-align: center;
}
pre, code {
/* The firmwares (CrossPoint, CrossInk) ignore `white-space` and treat `pre`
as inline, so the X4 simplifier makes each listing line a `p.code-line`;
the first and last lines carry the block's spacing, and a blank source
line becomes a `p.code-gap` margin. `figure` centres for
image captions, so lines reset alignment. The XTC converter's CREngine
defaults `code` to a block. */
pre {
display: block;
margin: 0;
text-align: left;
font-family: monospace;
font-size: 0.85em;
white-space: pre-wrap;
@@ -169,24 +178,81 @@ pre, code {
-webkit-hyphens: none;
}
/* Syntax tokens (src/highlight.rs), monochrome for e-ink: weight and slant
carry the structure, one grey for comments. Flat selectors on purpose. */
.hl-keyword,
.hl-tag,
p.code-line {
margin: 0;
text-indent: 0;
text-align: left;
hyphens: none;
-webkit-hyphens: none;
}
p.code-first {
margin-top: 0.6em;
}
p.code-last {
margin-bottom: 0.6em;
}
/* Where the source had a blank line: half a line, not a whole one. */
p.code-gap {
margin-bottom: 0.5em;
}
code {
display: inline;
font-family: monospace;
font-size: 0.85em;
hyphens: none;
-webkit-hyphens: none;
}
/* Syntax tokens (src/highlight.rs), as in style.css but on the X4's four
levels only (#000, #555, #aaa, white), so glyphs are not dithered: ink for
names, dark grey for keywords, literals and punctuation, light grey for
comments. */
.hl-function,
.hl-macro,
.hl-title,
.hl-strong {
font-weight: bold;
}
.hl-comment {
font-style: italic;
color: #444444;
}
.hl-type,
.hl-namespace,
.hl-constructor,
.hl-attribute,
.hl-emphasis {
font-style: italic;
}
.hl-keyword,
.hl-tag {
font-weight: bold;
color: #555555;
}
.hl-string {
color: #555555;
}
.hl-number,
.hl-constant,
.hl-label {
font-style: italic;
color: #555555;
}
.hl-punctuation,
.hl-operator {
color: #555555;
}
.hl-comment {
font-style: italic;
color: #aaaaaa;
}
table {
border-collapse: collapse;
font-size: 0.85em;
+42 -6
View File
@@ -221,29 +221,65 @@ hr.rule {
font-size: 0.85em;
}
/* CREngine readers (KOReader, PocketBook) default `code` to a block. */
.article-body code {
display: inline;
}
/* Left even inside a `<figure>`, which centres for image captions. */
.article-body pre {
margin: 0.8em 0;
text-align: left;
white-space: pre-wrap;
word-wrap: break-word;
border-left: 2px solid #cccccc;
padding-left: 0.5em;
}
/* Syntax tokens (src/highlight.rs), monochrome for e-ink: weight and slant
carry the structure, one grey for comments. Flat selectors on purpose. */
.hl-keyword,
.hl-tag,
/* Syntax tokens (src/highlight.rs), monochrome for e-ink, after zenbones and
lackluster: what the code names stays in ink (functions bold, types italic),
the language's own furniture steps back to grey (keywords bold grey,
literals and punctuation grey), comments lightest. Flat selectors. */
.hl-function,
.hl-macro,
.hl-title,
.hl-strong {
font-weight: bold;
}
.hl-comment {
.hl-type,
.hl-namespace,
.hl-constructor,
.hl-attribute,
.hl-emphasis {
font-style: italic;
}
.hl-keyword,
.hl-tag {
font-weight: bold;
color: #555555;
}
.hl-string {
color: #555555;
}
.hl-number,
.hl-constant,
.hl-label {
font-style: italic;
color: #555555;
}
.hl-emphasis {
.hl-punctuation,
.hl-operator {
color: #666666;
}
.hl-comment {
font-style: italic;
color: #888888;
}
.article-body table {
+188 -7
View File
@@ -124,10 +124,161 @@ fn is_dropped_declaration(decl: &str) -> bool {
false
}
/// Class of every paragraph [`paragraph_code_lines`] makes from a listing
/// line; the first and last also get [`CODE_FIRST_CLASS`] / [`CODE_LAST_CLASS`].
pub const CODE_LINE_CLASS: &str = "code-line";
pub const CODE_FIRST_CLASS: &str = "code-first";
pub const CODE_LAST_CLASS: &str = "code-last";
/// Marks a line followed by blank lines in the source; it gets a short gap.
pub const CODE_GAP_CLASS: &str = "code-gap";
/// Strip layout constructs the X4 renderer handles poorly from chapter markup,
/// then soft-hyphenate anything too long for it to wrap (§3.10).
pub fn simplify_xhtml(xhtml: &str) -> String {
break_long_words(&strip_attributes(xhtml, DROPPED_ATTRIBUTES))
paragraph_code_lines(&break_long_words(&strip_attributes(
xhtml,
DROPPED_ATTRIBUTES,
)))
}
/// Turn each line of every `<pre>` into its own `<p class="code-line">`.
///
/// The X4 firmwares (CrossPoint and forks such as CrossInk) ignore
/// `white-space` and lay out only `p`, `div`, `li`, `blockquote`, lists and
/// `br` as blocks: a `<pre>` is inline, so its margins never apply and its
/// `<br/>` lines are handled by each firmware's own rules. CrossInk glues the
/// first line onto the paragraph before, and empty `<br/>` lines count as scene
/// breaks. A paragraph per line takes the one path every firmware lays out the
/// same way. The first and last lines carry the block's margins; blank lines
/// become a half-line gap (see [`code_paragraphs`]).
///
/// Inline elements open across a line break (a `<code>`, a multi-line
/// comment's span) are closed at the end of the line and reopened at the start
/// of the next, so every paragraph is well formed.
fn paragraph_code_lines(html: &str) -> String {
let mut out = String::with_capacity(html.len() + 256);
let mut cursor = 0usize;
while let Some(rel) = html[cursor..].find('<') {
let start = cursor + rel;
out.push_str(&html[cursor..start]);
let Some(end) = tag_end(html, start) else {
out.push_str(&html[start..]);
return out;
};
let tag = &html[start..end];
out.push_str(tag);
cursor = end;
if !is_element_tag(tag, "pre", false) {
continue;
}
let Some(close) = find_close_tag(html, cursor, "pre") else {
continue;
};
out.push_str(&code_paragraphs(&html[cursor..close]));
cursor = close;
}
out.push_str(&html[cursor..]);
out
}
/// The paragraphs for one `<pre>` body whose line breaks are `<br/>`.
fn code_paragraphs(body: &str) -> String {
// Each line's markup, plus whether it shows any text.
let mut lines: Vec<(String, bool)> = Vec::new();
// Inline elements open at this point: (name, opening tag).
let mut open: Vec<(String, &str)> = Vec::new();
let mut line = String::new();
let mut visible = false;
let mut cursor = 0usize;
let finish = |line: &mut String, visible: &mut bool, open: &[(String, &str)]| {
let mut done = std::mem::take(line);
if !*visible {
done.push(NO_BREAK_SPACE);
}
for (name, _) in open.iter().rev() {
done.push_str(&format!("</{name}>"));
}
let shown = std::mem::replace(visible, false);
for (_, tag) in open {
line.push_str(tag);
}
(done, shown)
};
while cursor < body.len() {
let Some(rel) = body[cursor..].find('<') else {
let text = &body[cursor..];
visible |= shows_text(text);
line.push_str(text);
break;
};
let start = cursor + rel;
let text = &body[cursor..start];
visible |= shows_text(text);
line.push_str(text);
let Some(end) = tag_end(body, start) else {
line.push_str(&body[start..]);
break;
};
let tag = &body[start..end];
cursor = end;
let inner = tag.trim_start_matches('<').trim_end_matches('>');
let name = crate::html::tag_name(inner);
if name == "br" {
lines.push(finish(&mut line, &mut visible, &open));
} else if inner.starts_with('/') {
if let Some(at) = open.iter().rposition(|(open_name, _)| *open_name == name) {
open.remove(at);
}
line.push_str(tag);
} else {
if !inner.trim_end().ends_with('/') {
open.push((name, tag));
}
line.push_str(tag);
}
}
lines.push(finish(&mut line, &mut visible, &open));
// Blank lines become a short gap after the line before them rather than
// paragraphs of their own: with the firmware's paragraph spacing on, a
// blank paragraph adds a full line plus two half-line gaps and reads as a
// double newline. Leading, trailing and repeated blanks collapse.
let mut kept: Vec<(String, bool)> = Vec::with_capacity(lines.len());
for (line, shown) in lines {
if shown {
kept.push((line, false));
} else if let Some(previous) = kept.last_mut() {
previous.1 = true;
}
}
if kept.is_empty() {
// An all-blank listing still occupies one line.
kept.push((NO_BREAK_SPACE.to_string(), false));
}
let last = kept.len() - 1;
let mut out = String::with_capacity(body.len() + kept.len() * 48);
for (index, (line, gap_after)) in kept.into_iter().enumerate() {
let mut class = String::from(CODE_LINE_CLASS);
if index == 0 {
class.push(' ');
class.push_str(CODE_FIRST_CLASS);
}
if index == last {
class.push(' ');
class.push_str(CODE_LAST_CLASS);
} else if gap_after {
class.push(' ');
class.push_str(CODE_GAP_CLASS);
}
out.push_str(&format!("<p class=\"{class}\">{line}</p>"));
}
out
}
/// Whether a run of character data shows anything but spaces.
fn shows_text(text: &str) -> bool {
text.chars()
.any(|c| !c.is_whitespace() && c != NO_BREAK_SPACE)
}
/// Insert soft hyphens into over-long words and make `<pre>` whitespace
@@ -609,7 +760,7 @@ mod tests {
#[test]
fn pre_whitespace_is_made_explicit_inside_nested_tags() {
let input = "<pre><code>fn main() {\n if x &lt; 2 {\n <em>return</em>;\n }\n <strong><a href=\"#\">// done</a></strong>\n}</code></pre>";
let out = simplify_xhtml(input);
let out = break_long_words(input);
assert_eq!(
out,
concat!(
@@ -622,10 +773,40 @@ mod tests {
);
}
#[test]
fn each_code_line_becomes_a_paragraph() {
let input = "<p>Before.</p><figure><pre><code>let a = 1;\n\n let b = 2;</code></pre></figure><p>After.</p>";
assert_eq!(
simplify_xhtml(input),
concat!(
"<p>Before.</p><figure><pre>",
"<p class=\"code-line code-first code-gap\"><code>let a = 1;</code></p>",
"<p class=\"code-line code-last\"><code>\u{00a0}\u{00a0}let b = 2;</code></p>",
"</pre></figure><p>After.</p>"
)
);
// `<pre` as a prefix of another name is not a code block.
assert_eq!(simplify_xhtml("<prefix>x</prefix>"), "<prefix>x</prefix>");
}
#[test]
fn spans_open_across_a_line_break_are_closed_and_reopened() {
let input = "<pre><code><span class=\"hl-comment\">/* one\ntwo */</span> x\n</code></pre>";
assert_eq!(
simplify_xhtml(input),
concat!(
"<pre>",
"<p class=\"code-line code-first\"><code><span class=\"hl-comment\">/* one</span></code></p>",
"<p class=\"code-line code-last\"><code><span class=\"hl-comment\">two */</span> x</code></p>",
"</pre>"
)
);
}
#[test]
fn tabs_in_pre_become_four_no_break_spaces() {
assert_eq!(
simplify_xhtml("<pre><code>one\ttwo</code></pre>"),
break_long_words("<pre><code>one\ttwo</code></pre>"),
"<pre><code>one\u{00a0}\u{00a0}\u{00a0}\u{00a0}two</code></pre>"
);
}
@@ -634,7 +815,7 @@ mod tests {
fn whitespace_outside_pre_is_untouched() {
let input = "<p>outside text\n next\tword</p><pre>x y</pre><p>tail text</p>";
assert_eq!(
simplify_xhtml(input),
break_long_words(input),
"<p>outside text\n next\tword</p><pre>x\u{00a0}\u{00a0}y</pre><p>tail text</p>"
);
}
@@ -642,11 +823,11 @@ mod tests {
#[test]
fn newline_right_after_pre_open_is_not_a_blank_line() {
assert_eq!(
simplify_xhtml("<pre><code>\nlet a = 1;\n\nlet b = 2;</code></pre>"),
break_long_words("<pre><code>\nlet a = 1;\n\nlet b = 2;</code></pre>"),
"<pre><code>let a = 1;<br/><br/>let b = 2;</code></pre>"
);
assert_eq!(
simplify_xhtml("<pre>\n x</pre>"),
break_long_words("<pre>\n x</pre>"),
"<pre>\u{00a0}\u{00a0}x</pre>"
);
}
@@ -654,7 +835,7 @@ mod tests {
#[test]
fn pre_without_a_trailing_newline_gets_no_extra_break() {
let input = "<pre><code>let answer = 42;</code></pre>";
assert_eq!(simplify_xhtml(input), input);
assert_eq!(break_long_words(input), input);
}
/// The shipped X4 stylesheet must already satisfy the X4 rules, so the
+39 -6
View File
@@ -32,6 +32,8 @@ pub const EXITED_BEFORE_START: &str = "unit exited before the job started; see t
pub enum Job {
/// `generate` (today) or `generate-YYYY-MM-DD`.
Generate { date: Option<Date> },
/// `rebuild-YYYY-MM-DD` → `rebuild --date`.
Rebuild { date: Date },
/// `dry-run` → `generate --dry-run`.
DryRun,
/// `profile-rebuild` → `profile rebuild`.
@@ -70,8 +72,9 @@ impl Job {
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-')
}
/// Parse a catalogue name or the dated `generate-YYYY-MM-DD` form; anything
/// else (including `../x`, uppercase, unknown names) is `None`.
/// Parse a catalogue name or a dated `generate-YYYY-MM-DD` or
/// `rebuild-YYYY-MM-DD` form; anything else (including `../x`, uppercase,
/// unknown names) is `None`.
pub fn parse(name: &str) -> Option<Job> {
if !Self::valid_name(name) {
return None;
@@ -86,10 +89,17 @@ impl Job {
"import-ratings" => Some(Job::ImportRatings),
"interests-categorize" => Some(Job::InterestsCategorize),
_ => {
let date = name.strip_prefix("generate-")?;
let (rebuild, date) = match name.strip_prefix("generate-") {
Some(date) => (false, date),
None => (true, name.strip_prefix("rebuild-")?),
};
// Exactly `YYYY-MM-DD`; the round trip rejects `2026-9-3`.
let parsed: Date = date.parse().ok()?;
(parsed.to_string() == date).then_some(Job::Generate { date: Some(parsed) })
(parsed.to_string() == date).then_some(if rebuild {
Job::Rebuild { date: parsed }
} else {
Job::Generate { date: Some(parsed) }
})
}
}
}
@@ -98,6 +108,7 @@ impl Job {
match self {
Job::Generate { date: None } => "generate".into(),
Job::Generate { date: Some(date) } => format!("generate-{date}"),
Job::Rebuild { date } => format!("rebuild-{date}"),
Job::DryRun => "dry-run".into(),
Job::ProfileRebuild => "profile-rebuild".into(),
Job::FeaturesBackfill => "features-backfill".into(),
@@ -121,6 +132,9 @@ impl Job {
Job::Generate { date: Some(_) } => {
"Build and publish the issue of a given date, republishing it if it exists."
}
Job::Rebuild { .. } => {
"Re-render and republish a published issue's EPUBs from its stored content, without curating again."
}
Job::DryRun => "Run the whole pipeline but publish nothing and record no issue.",
Job::ProfileRebuild => {
"Regenerate the learned taste adjustments from ratings with the editor model."
@@ -144,6 +158,7 @@ impl Job {
pub fn takes_lock(&self) -> Option<&'static str> {
match self {
Job::Generate { .. } | Job::DryRun => Some("generate"),
Job::Rebuild { .. } => Some("rebuild"),
Job::ProfileRebuild => Some("profile rebuild"),
Job::FeaturesBackfill => Some("features backfill"),
Job::BackfillSocial => Some("backfill-social"),
@@ -151,9 +166,9 @@ impl Job {
}
}
/// Needs a confirmation dialog: `generate` republishes an issue.
/// Needs a confirmation dialog: `generate` and `rebuild` republish an issue.
pub fn dangerous(&self) -> bool {
matches!(self, Job::Generate { .. })
matches!(self, Job::Generate { .. } | Job::Rebuild { .. })
}
}
@@ -481,6 +496,24 @@ mod tests {
assert!(!Job::parse("dry-run").unwrap().dangerous());
}
#[test]
fn parse_accepts_the_dated_rebuild_form() {
let job = Job::parse("rebuild-2026-10-01").unwrap();
assert_eq!(
job,
Job::Rebuild {
date: "2026-10-01".parse().unwrap()
}
);
assert_eq!(job.name(), "rebuild-2026-10-01");
assert_eq!(job.unit(), "daily-epub-job@rebuild-2026-10-01.service");
assert_eq!(job.takes_lock(), Some("rebuild"));
assert!(job.dangerous());
for bad in ["rebuild", "rebuild-", "rebuild-2026-10-1", "rebuild-today"] {
assert_eq!(Job::parse(bad), None, "{bad}");
}
}
#[test]
fn parse_rejects_traversal_uppercase_and_unknown_names() {
for name in [
+45 -1
View File
@@ -36,6 +36,8 @@ struct Cli {
enum Command {
/// Build (and publish) one issue.
Generate(GenerateArgs),
/// Rebuild and republish a published issue's EPUBs from its stored content.
Rebuild(RebuildArgs),
/// Run the rating endpoints, the OPDS catalog and downloads.
Serve,
/// Taste-profile maintenance.
@@ -159,6 +161,13 @@ enum ConfigCommand {
Check,
}
#[derive(Debug, clap::Args)]
struct RebuildArgs {
/// Date of the published issue to rebuild.
#[arg(long, value_name = "YYYY-MM-DD")]
date: jiff::civil::Date,
}
#[derive(Debug, clap::Args)]
struct GenerateArgs {
/// Issue date in the configured timezone (defaults to today).
@@ -382,6 +391,17 @@ async fn main() -> Result<()> {
let outcome = pipeline::generate(&config, &db, &GenerateOptions::from(&args)).await?;
print_outcome(&outcome);
}
Command::Rebuild(args) => {
// No migration: this only re-renders what an existing schema holds.
let db = Db::open(&config.database_path).await?;
let published = pipeline::rebuild(&config, &db, args.date).await?;
for artifact in &published.epubs {
println!("{:?}: {}", artifact.edition, artifact.path.display());
}
if let Some(xtc) = &published.xtc {
println!("XTC: {}", xtc.display());
}
}
Command::Serve => {
let db = Db::open_and_migrate(&config.database_path).await?;
let config_path = Config::resolve_path(cli.config.as_deref());
@@ -451,6 +471,14 @@ async fn main() -> Result<()> {
}
.description()
);
eprintln!(
" {:<20} {}",
"rebuild-YYYY-MM-DD",
jobs::Job::Rebuild {
date: jiff::civil::Date::default()
}
.description()
);
std::process::exit(2);
};
let db = Db::open_and_migrate(&config.database_path).await?;
@@ -461,12 +489,13 @@ async fn main() -> Result<()> {
}
/// The commands that write the database and provider budgets and so hold the
/// run lock (§5): `generate`, `profile rebuild`, `features backfill`,
/// run lock (§5): `generate`, `rebuild`, `profile rebuild`, `features backfill`,
/// `backfill-social`, `interests backfill`, and a `job run` of any of them.
/// Everything else is read-only or its own writer.
fn lock_holder(command: &Command) -> Option<&'static str> {
match command {
Command::Generate(_) => Some("generate"),
Command::Rebuild(_) => Some("rebuild"),
// Writes the same tables the discovery stage of a run writes.
Command::Feeds(FeedsCommand::Discover(_)) => Some("feeds discover"),
Command::Profile(ProfileCommand::Rebuild) => Some("profile rebuild"),
@@ -1161,6 +1190,21 @@ async fn run_job(config: &Config, db: &Db, job: &jobs::Job) -> Result<(String, O
jobs::Job::Generate { date } => {
generate_job(config, db, date.map(|date| date.to_string()), false).await
}
jobs::Job::Rebuild { date } => {
let published = pipeline::rebuild(config, db, *date).await?;
let xtc = if published.xtc.is_some() {
" and the XTC"
} else {
""
};
Ok((
format!(
"rebuilt the {date} issue: {} EPUBs{xtc}",
published.epubs.len()
),
None,
))
}
jobs::Job::DryRun => generate_job(config, db, None, true).await,
jobs::Job::ProfileRebuild => Ok((cmd_profile_rebuild(config, db).await?, None)),
jobs::Job::FeaturesBackfill => {
+130
View File
@@ -1095,6 +1095,51 @@ async fn persist_articles(db: &Db, articles: &mut [Article]) -> Result<()> {
Ok(())
}
/// Rebuild and republish a published issue's EPUBs (and XTC) from its stored
/// snapshot, for when rendering changed after the issue went out.
///
/// The content is exactly the issue's: the `issues.issue_json` snapshot with
/// each article's body restored from its row (the snapshot drops bodies).
/// Unlike the web reader it does not refresh anything else from the live rows,
/// so social lines and metadata stay as published. Nothing is curated or
/// fetched except the images, which are downloaded again as in a run.
pub async fn rebuild(config: &Config, db: &Db, date: Date) -> Result<Published> {
let row = db
.issue_by_date(date)
.await?
.with_context(|| format!("no issue for {date}"))?;
let raw = row
.issue_json
.with_context(|| format!("the {date} issue predates issue snapshots"))?;
let mut issue: Issue = serde_json::from_str(&raw).context("decoding issues.issue_json")?;
let ids: Vec<ArticleId> = issue.lineup.picks.iter().map(|p| p.article.id).collect();
let mut articles = db.get_articles(&ids).await?;
for pick in &mut issue.lineup.picks {
let article = articles
.remove(&pick.article.id)
.with_context(|| format!("article {} is gone", pick.article.id))?;
pick.article.content_html = article.content_html;
}
let (artifacts, _) = epub::build_all(&issue, config, &config.out_dir)
.await
.context("building the EPUB editions")?;
let x4 = artifacts.iter().find(|a| a.edition == Edition::X4);
let xtc = match x4 {
Some(artifact) if config.xtc.enabled => {
Some(epub::x4::convert(&config.xtc, &artifact.path, &config.out_dir).await?)
}
_ => None,
};
let published = publish::publish_issue(config, &issue, &artifacts, xtc.as_deref())
.await
.context("publishing the issue")?;
record_issue(db, &issue, &published)
.await
.context("recording the issue")?;
Ok(published)
}
/// Write the `issues` row and replace `issue_articles` for the date (notes §12).
async fn record_issue(db: &Db, issue: &Issue, published: &Published) -> Result<()> {
let path_for = |edition: Edition| {
@@ -1346,6 +1391,91 @@ fn elapsed_ms(since: Timestamp) -> i64 {
mod tests {
use super::*;
/// Read one entry of a zip archive as text.
fn zip_entry(path: &std::path::Path, name: &str) -> String {
use std::io::Read;
let mut archive = zip::ZipArchive::new(std::fs::File::open(path).unwrap()).unwrap();
let mut text = String::new();
archive
.by_name(name)
.unwrap()
.read_to_string(&mut text)
.unwrap();
text
}
#[tokio::test]
async fn rebuild_republishes_from_the_snapshot_without_touching_it() {
let dir = tempfile::tempdir().unwrap();
let db = Db::open_and_migrate(&dir.path().join("db.sqlite"))
.await
.unwrap();
let mut issue = crate::epub::fixtures::issue();
for pick in &mut issue.lineup.picks {
// No images: the rebuild must not need the network here.
pick.article.content_html =
"<p>Before.</p><pre data-lang=\"rust\"><code>fn main() {}</code></pre>".into();
pick.article.image_count = 0;
let article = &pick.article;
db.upsert_entry(&crate::types::Entry {
id: article.best_entry_id,
feed_id: article.feed_id,
feed_title: Some(article.feed_title.clone()),
category: article.category.clone(),
title: article.title.clone(),
url: article.url.clone(),
canonical_url: Some(article.canonical_url.clone()),
author: article.author.clone(),
published_at: article.published_at,
comments_url: article.comments_url.clone(),
raw_content: article.content_html.clone(),
fetched_at: article.first_seen,
})
.await
.unwrap();
pick.article.id = db.upsert_article(article).await.unwrap();
}
let mut snapshot = issue.clone();
for pick in &mut snapshot.lineup.picks {
pick.article.content_html.clear();
}
let issue_json = serde_json::to_string(&snapshot).unwrap();
db.upsert_issue(
issue.meta.date,
issue.meta.issue_number,
issue.meta.generated_at,
None,
None,
None,
Some(&issue.editorial.front_page_html),
None,
Some(&issue_json),
)
.await
.unwrap();
let mut config = Config {
out_dir: dir.path().join("out"),
..Config::default()
};
config.publish.epub_dir = dir.path().join("epubs");
config.publish.xtc_dir = dir.path().join("xtc");
config.xtc.enabled = false;
let published = rebuild(&config, &db, issue.meta.date).await.unwrap();
assert_eq!(published.epubs.len(), 2);
let row = db.issue_by_date(issue.meta.date).await.unwrap().unwrap();
assert_eq!(row.issue_json.as_deref(), Some(issue_json.as_str()));
let standard = &published.epubs[0];
assert_eq!(
row.epub_path.as_deref(),
Some(standard.path.to_str().unwrap())
);
let chapter = zip_entry(&standard.path, "OEBPS/art-1001.xhtml");
assert!(chapter.contains("<p>Before.</p>"), "body restored");
assert!(chapter.contains("<span class=\"hl-keyword\">fn</span>"));
}
#[test]
fn display_date_matches_the_masthead_format() {
let date: Date = "2026-08-15".parse().unwrap();