Fix the six ways article images went missing
Across the first three issues, 145 of 208 referenced images reached the
page; 38 that had been downloaded, re-encoded and written into the EPUB
were never referenced by any chapter, and 69 became `[image: …]` lines.
Six independent causes, each verified against the real articles:
- **Entity-encoded URLs never matched their asset.** `prepare_body`
re-runs `ammonia::clean`, which writes `&` as `&`, and the tag
scanner that looks the asset up compared raw attribute text against a
URL a real parser had decoded. Every image with a query string lost.
`parse_attrs` now decodes entities.
- **Readability clobbers a working `src`.** Its lazy-image heuristic
copies any attribute containing `.jpg` over `src`, which on NPR meant
a `{width}` template (the CDN answers those with a grey square reading
"Image"), on Substack a JSON blob, on dfarq an entire `srcset` string.
Pages are now normalized before readability sees them: every `<img>`
is reduced to `src`/`alt`/`title` with the best candidate from the
lazy attributes, `src`, `srcset` and `<picture><source>`, so there is
nothing left for the heuristic to substitute. Candidates that cannot
resolve — braces, whitespace, quotes, `data:` — are rejected by shape
rather than by publisher.
- **Readability deletes `<button>` and its subtree**, taking lightbox
images with it and leaving the captions behind. Image-only wrappers
are unwrapped first.
- **SVG was undecodable**, so vector charts became placeholders. They
are rasterized with resvg, which the cover already depends on.
- **Relative URLs resolved against the pre-redirect URL**, 404ing every
image on an article reached through a shortener. `fetch_readable` now
reports where it landed.
- **The 12-image cap** silently truncated photo essays, and the images
past it were advertised as failures. Removed; the issue-wide byte
budget is the real backstop.
An image that still cannot be embedded is now dropped rather than
announced, unless its alt text is a real description — decorative rules,
spacers and dead links were generating most of the placeholder noise.
`examples/image_audit.rs` replays the pipeline over the articles of
published issues and reports what reaches the page. On issues 1–3:
208→214 images referenced, 145→214 shown, 69→0 placeholders, 38→0
orphaned assets. The three images still not embedded are a 14×14 favicon
and a 650×2 divider — correctly declined, and no longer announced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -404,6 +404,23 @@ comments,world,epub,publish,server}` implement them; `src/auth.rs` owns the rati
|
||||
token formula used by both the EPUB writer and the server; `src/types.rs` is the
|
||||
contract between stages.
|
||||
|
||||
### Auditing images against real articles
|
||||
|
||||
Image handling fails in ways no synthetic fixture predicts, because every
|
||||
publisher invents its own lazy-loading scheme. `examples/image_audit.rs` replays
|
||||
the extraction and image pipeline over the articles of issues already published
|
||||
and counts what actually reaches the page:
|
||||
|
||||
```sh
|
||||
cargo run --release --example image_audit -- \
|
||||
--cache /tmp/pagecache ~/bookorbit/books/daily-epub/*.epub
|
||||
```
|
||||
|
||||
It prints per-article `refs / embedded / shown / placeholders`, a tally of loss
|
||||
reasons, and totals. Pages are cached on first run, so a change can be measured
|
||||
against byte-identical input; `--dump <title substring>` lists the URLs one
|
||||
article resolved to. It needs the network and is not part of `cargo test`.
|
||||
|
||||
---
|
||||
|
||||
## Known limitations
|
||||
@@ -425,6 +442,10 @@ From spec §7, plus what implementation turned up:
|
||||
falls back to matching the feed title.
|
||||
- **Images are downloaded once per edition** (the two editions need different
|
||||
resolutions and colour profiles), so an image-heavy issue makes two passes.
|
||||
- **An image that cannot be embedded is dropped, not announced,** unless its alt
|
||||
text is a real description — decorative rules, spacers and dead links would
|
||||
otherwise litter the page with `[image: …]` lines. Verify image handling with
|
||||
`examples/image_audit.rs` after touching extraction.
|
||||
- **`dc:date` rides inside a `dcterms:date` metadata fragment** because
|
||||
`epub-builder` neither exposes `dc:date` nor accepts a non-`chrono` date. The
|
||||
OPF output is correct; the mechanism is a workaround.
|
||||
|
||||
Reference in New Issue
Block a user