thallada/the-daily-epub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Break up the image and EPUB god files

Browse Source 
The image fixes left two problems of navigation. Image handling was
spread across `extract.rs` (300 lines of normalization) and
`epub/images.rs` (download, re-encode, markup rewriting, plus generic
HTML helpers that comments, world and x4 were all reaching into a
module named "images" to borrow). And `epub/build.rs` had grown to
1,148 lines of code covering cover rendering, ten askama templates,
every chapter renderer and the zip assembly.

New homes:

- `src/html.rs` — markup helpers that do not care what the markup is
  about: tag scanning, attribute parsing, entity decoding, escaping,
  XHTML fixups, reading a fragment as text. Previously scattered between
  `epub/images.rs` and `extract.rs`.
- `src/images/` — one module per stage of an article's images, in the
  order they run: `normalize` (make `<img>` usable, pre-readability),
  `refs` (what an article references), `fetch` + `encode` (download and
  re-encode per edition), `embed` (point the markup at what shipped).
- `src/epub/{cover,chapters,build}.rs` — the cover, the chapter
  renderers, and the ordering plus assembly that puts them together.
  `epub/fixtures.rs` takes the shared test issue, which was a public
  module wedged inside `build.rs`.
- `src/curate/profile/themes.rs` — a 260-line keyword table that sat in
  the middle of the profile logic.

`curate::html_to_text` is renamed `prompt_text`: it is a different
function from `html::html_to_text` (collapses whitespace, no DOM, sized
for prompt budgets) and sharing a name with it was a trap.

Largest module drops from 1,148 code lines to 828, and no file mixes
two subjects. Behaviour is unchanged: 231 lib tests plus 25 integration
tests green, and the real-world audit over issues 1–3 still reports 214
images referenced, 214 shown, 0 placeholders, 0 orphaned assets.

`image_audit` gains `--epub-out DIR`, which writes a readable EPUB of
the audited articles so images can be checked on a device.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
thallada
2026-08-18 03:35:21 +00:00
co-authored by Claude Opus 5
parent 254eaeb713
commit db19d08257
27 changed files with 3513 additions and 3132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/epub/fixtures.rs
+182
View File
@@ -0,0 +1,182 @@
//! A synthetic, fully offline [`crate::types::Issue`] for tests.
//!
//! Lives here rather than inside a `#[cfg(test)]` block because the integration
//! tests in `tests/` can only see the public API.
use std::collections::BTreeMap;
use jiff::Timestamp;
use crate::types::*;
pub fn timestamp() -> Timestamp {
"2026-08-15T05:30:00Z".parse().expect("fixed timestamp")
}
pub fn article(id: ArticleId, entry_id: EntryId, title: &str) -> Article {
Article {
id,
canonical_url: format!("https://example.com/{entry_id}"),
title: title.to_string(),
best_entry_id: entry_id,
content_html: format!(
"<p>Body of <em>{title}</em> with an image.</p><img src=\"https://img.example/{entry_id}.png\" alt=\"A chart of the daily figures\"><p>More words &amp; things.</p>"
),
word_count: 1200,
excerpt_only: false,
image_count: 1,
sources: vec![SourceRef {
entry_id,
feed_id: 7,
feed_title: "Example Feed".into(),
category: Some("Tech".into()),
kind: SourceKind::Feed,
}],
first_seen: timestamp(),
url: format!("https://example.com/{entry_id}"),
author: Some("A. Writer".into()),
feed_id: 7,
feed_title: "Example Feed".into(),
category: Some("Tech".into()),
published_at: Some(timestamp()),
comments_url: None,
image_urls: vec![format!("https://img.example/{entry_id}.png")],
social: vec![SocialRef {
article_id: id,
source: SocialSource::Hn,
item_id: Some("40100000".into()),
score: 342,
num_comments: 210,
item_url: Some("https://news.ycombinator.com/item?id=40100000".into()),
fetched_at: timestamp(),
}],
extract_method: ExtractMethod::Readability,
}
}
pub fn discussion(article_id: ArticleId, entry_id: EntryId) -> Discussion {
Discussion {
article_id,
chapter_id: format!("disc-{entry_id}"),
threads: vec![CommentThread {
source: SocialSource::Hn,
item_url: "https://news.ycombinator.com/item?id=40100000".into(),
total_comments: 210,
comments: vec![Comment {
author: "alice".into(),
points: Some(61),
text_html: "<p>The write path is the interesting part.</p>".into(),
depth: 0,
children: vec![Comment {
author: "bob".into(),
points: Some(24),
text_html: "<p>Agreed.</p>".into(),
depth: 1,
children: vec![],
}],
}],
}],
}
}
/// A synthetic two-article issue, one of them carrying a discussion.
pub fn issue() -> Issue {
let lead = Pick {
article: article(1, 1001, "The Lead Story"),
section: "Top Stories".into(),
position: 0,
is_lead: true,
summary: Some("What it argues, and why it is worth the time.".into()),
llm: None,
discussion: Some(discussion(1, 1001)),
};
let second = Pick {
article: article(2, 1002, "A Niche Delight & Other Tales"),
section: "Niche Corner".into(),
position: 0,
is_lead: false,
summary: None,
llm: None,
discussion: None,
};
let mut section_intros = BTreeMap::new();
section_intros.insert("Top Stories".to_string(), "The day in brief.".to_string());
let mut summaries = BTreeMap::new();
summaries.insert(2, "A short abstract for the second piece.".to_string());
Issue {
meta: IssueMeta {
date: "2026-08-15".parse().expect("fixed date"),
issue_number: 42,
generated_at: timestamp(),
display_date: "Friday, August 15, 2026".into(),
article_count: 2,
section_count: 2,
total_words: 2400,
reading_minutes: 11,
},
lineup: Lineup {
date: "2026-08-15".parse().expect("fixed date"),
picks: vec![lead, second],
section_order: vec!["Top Stories".into(), "Niche Corner".into()],
},
editorial: Editorial {
front_page_html: "<p>Two stories today, both worth your coffee.</p>".into(),
section_intros,
summaries,
},
world_briefing: Some(WorldBriefing {
date: "2026-08-15".parse().expect("fixed date"),
source_url: "https://en.wikipedia.org/wiki/Portal:Current_events/2026_August_15".into(),
overview: Some("A concise view of the day.".into()),
sections: vec![WorldBriefingSection {
title: "Top Stories".into(),
events: vec![WorldEvent {
id: "s1-e1".into(),
source_text: "Something happened somewhere.".into(),
links: vec![],
children: vec![],
summary: Some("The event in context.".into()),
}],
}],
}),
colophon: Colophon {
model: "deepseek-v4-flash".into(),
entries_fetched: 431,
feeds_seen: 92,
candidates: 120,
cost_usd: 0.0731,
generator_version: "daily-epub 0.1.0".into(),
},
}
}
/// A crude XHTML well-formedness check for rendered chapters.
///
/// The real proof is `epubcheck`, which cannot run in a unit test; this catches
/// the mistakes that actually happen — an unclosed void element or a raw
/// `&nbsp;`, both of which make an EPUB3 content document unparseable.
pub fn assert_xml_ok(xhtml: &str) {
// A crude well-formedness check: the document parses as XML only if every
// tag is closed, so compare open/close counts for the elements we emit.
assert!(xhtml.starts_with("<?xml version=\"1.0\" encoding=\"utf-8\"?>"));
assert!(xhtml.contains("xmlns=\"http://www.w3.org/1999/xhtml\""));
assert!(xhtml.trim_end().ends_with("</html>"));
for tag in ["html", "head", "body", "div", "p"] {
let opens = xhtml.matches(&format!("<{tag}")).count();
let closes = xhtml.matches(&format!("</{tag}>")).count();
assert_eq!(opens, closes, "unbalanced <{tag}> in\n{xhtml}");
}
assert!(!xhtml.contains("&nbsp;"));
for void in ["<br>", "<hr>", "<img "] {
if void == "<img " {
for (i, _) in xhtml.match_indices("<img ") {
let tail = &xhtml[i..];
let end = tail.find('>').unwrap_or(0);
assert!(tail[..end].ends_with('/'), "unclosed <img> in\n{xhtml}");
}
} else {
assert!(!xhtml.contains(void), "unclosed {void} in\n{xhtml}");
}
}
}