Break up the image and EPUB god files
The image fixes left two problems of navigation. Image handling was
spread across `extract.rs` (300 lines of normalization) and
`epub/images.rs` (download, re-encode, markup rewriting, plus generic
HTML helpers that comments, world and x4 were all reaching into a
module named "images" to borrow). And `epub/build.rs` had grown to
1,148 lines of code covering cover rendering, ten askama templates,
every chapter renderer and the zip assembly.
New homes:
- `src/html.rs` — markup helpers that do not care what the markup is
about: tag scanning, attribute parsing, entity decoding, escaping,
XHTML fixups, reading a fragment as text. Previously scattered between
`epub/images.rs` and `extract.rs`.
- `src/images/` — one module per stage of an article's images, in the
order they run: `normalize` (make `<img>` usable, pre-readability),
`refs` (what an article references), `fetch` + `encode` (download and
re-encode per edition), `embed` (point the markup at what shipped).
- `src/epub/{cover,chapters,build}.rs` — the cover, the chapter
renderers, and the ordering plus assembly that puts them together.
`epub/fixtures.rs` takes the shared test issue, which was a public
module wedged inside `build.rs`.
- `src/curate/profile/themes.rs` — a 260-line keyword table that sat in
the middle of the profile logic.
`curate::html_to_text` is renamed `prompt_text`: it is a different
function from `html::html_to_text` (collapses whitespace, no DOM, sized
for prompt budgets) and sharing a name with it was a trap.
Largest module drops from 1,148 code lines to 828, and no file mixes
two subjects. Behaviour is unchanged: 231 lib tests plus 25 integration
tests green, and the real-world audit over issues 1–3 still reports 214
images referenced, 214 shown, 0 placeholders, 0 orphaned assets.
`image_audit` gains `--epub-out DIR`, which writes a readable EPUB of
the audited articles so images can be checked on a device.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -399,10 +399,32 @@ extraction → prefilter → selection (both the `--skip-llm` route and a
|
||||
`MockBackend` DeepSeek route) → editorial → both EPUB editions → publish → OPDS
|
||||
and database rows, with no network access anywhere.
|
||||
|
||||
Layout: `src/pipeline.rs` wires the stages; `src/{dedupe,extract,social,curate,
|
||||
comments,world,epub,publish,server}` implement them; `src/auth.rs` owns the rating
|
||||
token formula used by both the EPUB writer and the server; `src/types.rs` is the
|
||||
contract between stages.
|
||||
### Layout
|
||||
|
||||
`src/pipeline.rs` wires the stages; `src/types.rs` is the contract between them;
|
||||
`src/auth.rs` owns the rating token formula, shared by the EPUB writer and the
|
||||
server. The stages themselves:
|
||||
|
||||
```text
|
||||
miniflux.rs ingest curate/ scoring and selection
|
||||
dedupe.rs clustering prefilter, llm, score, select, editorial
|
||||
extract.rs body text profile/ the reader's taste profile
|
||||
images/ article images comments.rs discussion chapters
|
||||
normalize usable <img> world.rs the world briefing
|
||||
refs what's there epub/ the two editions
|
||||
fetch download chapters, cover, build, x4
|
||||
encode re-encode publish.rs BookOrbit + XTC
|
||||
embed into the page server.rs ratings, OPDS
|
||||
html.rs markup helpers db.rs SQLite
|
||||
```
|
||||
|
||||
Two modules are worth knowing about before you go looking for their contents.
|
||||
`src/html.rs` holds the generic markup helpers — tag scanning, escaping, entity
|
||||
decoding, XHTML fixups — that extraction, images, comments and the world briefing
|
||||
all need; put anything that works on markup without caring what the markup is
|
||||
*about* there. `src/images/` owns every stage of an article's images, which is
|
||||
otherwise the kind of concern that smears itself across extraction and EPUB
|
||||
building; see its module docs for the order the stages run in.
|
||||
|
||||
### Auditing images against real articles
|
||||
|
||||
@@ -419,7 +441,9 @@ cargo run --release --example image_audit -- \
|
||||
It prints per-article `refs / embedded / shown / placeholders`, a tally of loss
|
||||
reasons, and totals. Pages are cached on first run, so a change can be measured
|
||||
against byte-identical input; `--dump <title substring>` lists the URLs one
|
||||
article resolved to. It needs the network and is not part of `cargo test`.
|
||||
article resolved to, and `--epub-out DIR` writes a readable EPUB of the audited
|
||||
articles so the images can be looked at on a device rather than counted in a
|
||||
table. It needs the network and is not part of `cargo test`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user