Behind-the-paper chapter (behind.xhtml, both editions) built from the run's
StageCounts and candidate_runs near misses; daily-epub stats [--days N];
StageCounts gains knn/feed gates and verdicts_in_prompt, timings split
into summaries + brief, the four-line §15.4 info block logged once per
run and printed by print_report; src/lock.rs flock guard on
<database_path>.lock for generate, profile rebuild, features backfill and
backfill-social; README updated for the new CLI, env vars and costs.
Implemented by a Claude agent from docs/plans/curation-v2-briefs/step6.md;
reviewed against plan §5, §15.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
- AnthropicBackend (Messages API, cached system block, output_config.effort,
server-side fallbacks, refusal surfaced as an error); Llms { bulk, editor }
with editor_or_bulk(); PriceTable-based UsageMeter per provider.
- [anthropic], [editorial], deepseek.max_concurrent_requests and
curation.max_article_count config; startup logs resolved providers.
- Budget day is the UTC date of started_at, preloaded from
runs.provider_costs_json; finish_run writes provider_costs_json and
config_json. Stage A batches run concurrently with per-batch budget checks.
- Editor prompt with one-line "why" per pick; no minimum lineup size;
--max-articles is a ceiling; top-up branch deleted; why stored on picks and
issue_articles.why and rendered in chapters and In this issue.
- Summaries on the editor client (3k-token input, concurrency 4, bulk then
excerpt fallback); "The Brief" replaces From the Editor; section intros gone.
- Colophon carries per-provider costs and models.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
The image fixes left two problems of navigation. Image handling was
spread across `extract.rs` (300 lines of normalization) and
`epub/images.rs` (download, re-encode, markup rewriting, plus generic
HTML helpers that comments, world and x4 were all reaching into a
module named "images" to borrow). And `epub/build.rs` had grown to
1,148 lines of code covering cover rendering, ten askama templates,
every chapter renderer and the zip assembly.
New homes:
- `src/html.rs` — markup helpers that do not care what the markup is
about: tag scanning, attribute parsing, entity decoding, escaping,
XHTML fixups, reading a fragment as text. Previously scattered between
`epub/images.rs` and `extract.rs`.
- `src/images/` — one module per stage of an article's images, in the
order they run: `normalize` (make `<img>` usable, pre-readability),
`refs` (what an article references), `fetch` + `encode` (download and
re-encode per edition), `embed` (point the markup at what shipped).
- `src/epub/{cover,chapters,build}.rs` — the cover, the chapter
renderers, and the ordering plus assembly that puts them together.
`epub/fixtures.rs` takes the shared test issue, which was a public
module wedged inside `build.rs`.
- `src/curate/profile/themes.rs` — a 260-line keyword table that sat in
the middle of the profile logic.
`curate::html_to_text` is renamed `prompt_text`: it is a different
function from `html::html_to_text` (collapses whitespace, no DOM, sized
for prompt budgets) and sharing a name with it was a trap.
Largest module drops from 1,148 code lines to 828, and no file mixes
two subjects. Behaviour is unchanged: 231 lib tests plus 25 integration
tests green, and the real-world audit over issues 1–3 still reports 214
images referenced, 214 shown, 0 placeholders, 0 orphaned assets.
`image_audit` gains `--epub-out DIR`, which writes a readable EPUB of
the audited articles so images can be checked on a device.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Across the first three issues, 145 of 208 referenced images reached the
page; 38 that had been downloaded, re-encoded and written into the EPUB
were never referenced by any chapter, and 69 became `[image: …]` lines.
Six independent causes, each verified against the real articles:
- **Entity-encoded URLs never matched their asset.** `prepare_body`
re-runs `ammonia::clean`, which writes `&` as `&`, and the tag
scanner that looks the asset up compared raw attribute text against a
URL a real parser had decoded. Every image with a query string lost.
`parse_attrs` now decodes entities.
- **Readability clobbers a working `src`.** Its lazy-image heuristic
copies any attribute containing `.jpg` over `src`, which on NPR meant
a `{width}` template (the CDN answers those with a grey square reading
"Image"), on Substack a JSON blob, on dfarq an entire `srcset` string.
Pages are now normalized before readability sees them: every `<img>`
is reduced to `src`/`alt`/`title` with the best candidate from the
lazy attributes, `src`, `srcset` and `<picture><source>`, so there is
nothing left for the heuristic to substitute. Candidates that cannot
resolve — braces, whitespace, quotes, `data:` — are rejected by shape
rather than by publisher.
- **Readability deletes `<button>` and its subtree**, taking lightbox
images with it and leaving the captions behind. Image-only wrappers
are unwrapped first.
- **SVG was undecodable**, so vector charts became placeholders. They
are rasterized with resvg, which the cover already depends on.
- **Relative URLs resolved against the pre-redirect URL**, 404ing every
image on an article reached through a shortener. `fetch_readable` now
reports where it landed.
- **The 12-image cap** silently truncated photo essays, and the images
past it were advertised as failures. Removed; the issue-wide byte
budget is the real backstop.
An image that still cannot be embedded is now dropped rather than
announced, unless its alt text is a real description — decorative rules,
spacers and dead links were generating most of the placeholder noise.
`examples/image_audit.rs` replays the pipeline over the articles of
published issues and reports what reaches the page. On issues 1–3:
208→214 images referenced, 145→214 shown, 69→0 placeholders, 38→0
orphaned assets. The three images still not embedded are a 14×14 favicon
and a 650×2 divider — correctly declined, and no longer announced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
thalladaandClaude Fable 5.1