Break up the image and EPUB god files
The image fixes left two problems of navigation. Image handling was
spread across `extract.rs` (300 lines of normalization) and
`epub/images.rs` (download, re-encode, markup rewriting, plus generic
HTML helpers that comments, world and x4 were all reaching into a
module named "images" to borrow). And `epub/build.rs` had grown to
1,148 lines of code covering cover rendering, ten askama templates,
every chapter renderer and the zip assembly.
New homes:
- `src/html.rs` — markup helpers that do not care what the markup is
about: tag scanning, attribute parsing, entity decoding, escaping,
XHTML fixups, reading a fragment as text. Previously scattered between
`epub/images.rs` and `extract.rs`.
- `src/images/` — one module per stage of an article's images, in the
order they run: `normalize` (make `<img>` usable, pre-readability),
`refs` (what an article references), `fetch` + `encode` (download and
re-encode per edition), `embed` (point the markup at what shipped).
- `src/epub/{cover,chapters,build}.rs` — the cover, the chapter
renderers, and the ordering plus assembly that puts them together.
`epub/fixtures.rs` takes the shared test issue, which was a public
module wedged inside `build.rs`.
- `src/curate/profile/themes.rs` — a 260-line keyword table that sat in
the middle of the profile logic.
`curate::html_to_text` is renamed `prompt_text`: it is a different
function from `html::html_to_text` (collapses whitespace, no DOM, sized
for prompt budgets) and sharing a name with it was a trap.
Largest module drops from 1,148 code lines to 828, and no file mixes
two subjects. Behaviour is unchanged: 231 lib tests plus 25 integration
tests green, and the real-world audit over issues 1–3 still reports 214
images referenced, 214 shown, 0 placeholders, 0 orphaned assets.
`image_audit` gains `--epub-out DIR`, which writes a readable EPUB of
the audited articles so images can be checked on a device.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
254eaeb713
commit
db19d08257
27 files changed
+3513
-3132
No files matched your search
+9
-5
@@ -164,7 +164,11 @@ pub fn approx_tokens(text: &str) -> usize {
|
||||
|
||||
/// Strip markup and collapse whitespace, so article bodies can go into prompts
|
||||
/// as plain text (cheaper and less confusing for the model than raw HTML).
|
||||
pub fn html_to_text(html: &str) -> String {
|
||||
///
|
||||
/// Deliberately not [`crate::html::html_to_text`]: this one collapses runs of
|
||||
/// whitespace and never builds a DOM, because it runs over every candidate
|
||||
/// body on every run and only has to be good enough to size a prompt.
|
||||
pub fn prompt_text(html: &str) -> String {
|
||||
/// Does `tail` open the named element, i.e. `<name` or `</name`?
|
||||
fn opens(tail: &str, name: &str) -> bool {
|
||||
let bytes = tail.as_bytes();
|
||||
@@ -272,11 +276,11 @@ mod tests {
|
||||
fn html_becomes_readable_text() {
|
||||
let html = "<h1>Title</h1><p>First & best.</p><script>alert('x')</script>\
|
||||
<p>Second<br/>line</p><style>p{color:red}</style>";
|
||||
assert_eq!(html_to_text(html), "Title First & best. Second line");
|
||||
assert_eq!(html_to_text(""), "");
|
||||
assert_eq!(html_to_text("no markup at all"), "no markup at all");
|
||||
assert_eq!(prompt_text(html), "Title First & best. Second line");
|
||||
assert_eq!(prompt_text(""), "");
|
||||
assert_eq!(prompt_text("no markup at all"), "no markup at all");
|
||||
// Unicode survives byte-wise walking.
|
||||
assert_eq!(html_to_text("<p>café — naïve</p>"), "café — naïve");
|
||||
assert_eq!(prompt_text("<p>café — naïve</p>"), "café — naïve");
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
Reference in new issue
Block a user