Commit Graph
14 Commits
Author SHA1 Message Date
thalladaandClaude Fable 5.1 f2a44507d4 Show the publication after the feed name, and link feeds to Miniflux for admins
Readers see "Feed · Publication" on the issue, article, public and RSS
pages and in the EPUB index and chapter headers; the publication is the
page's site name, else its domain, and is dropped when it would repeat
the feed's title. Admins get the feed name linked to its Miniflux page.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YWmCpUojfHXhSZ2129Z7Nv
2026-09-09 06:01:29 +00:00
thalladaandClaude Fable 5.1 f71a4c2799 Keep the publication an article's page declares
Readability already reads og:site_name and JSON-LD publisher.name; carry
that through Page and Extracted and store it in articles.publication, the
same way the page byline is kept. `publication_label` is what the readers
will see after the feed name: the site name, else the domain, and nothing
when it would only repeat the feed's own title.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YWmCpUojfHXhSZ2129Z7Nv
2026-09-09 05:48:46 +00:00
thalladaandClaude Fable 5.1 b02e1c7b8c Add an AI slop verdict that downrates the author's future articles
A fourth explicit verdict, "AI slop" (Vote::Slop, label `slop`), joins the
rating widget on the web, the EPUB footer links, the /r/ confirmation page,
the CLI, imports and the dashboard filters. It counts as a full negative
(curation.feedback.slop_value, -1.0) in the neighbour and affinity signals.

Beyond that, each run loads the authors whose current verdict is slop, with
no lookback, and multiplies the preliminary blend and the utility of every
candidate by that author by 1 - curation.ranking.slop_author_penalty (0.75),
so they sink before triage. The flag is recorded in signals_json, shown by
`explain` and the dashboard signals table, and the confirmation names the
author (or says no author is known, in which case only the rating applies).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VXYGBPoHZSDSfE5WcJ9bvj
2026-09-09 03:36:50 +00:00
thalladaandClaude Fable 5.1 bedbcb218f Show what the pipeline understood about each article
A muted one-line note now follows every article: the deep assessment's
topic group, format, depth and technicality, the specific topics it
extracted, and the reader interests the article matched best, e.g.

  Software engineering · analysis essay · in depth · highly technical ·
  Topics: copy-on-write, ZFS · Interests: Filesystems, Rust

It renders on the web article page footer, on the public and private
issue-page entries, in the EPUB article footer and in the EPUB "In This
Issue" index (both editions), always styled like the existing meta lines
so it stays secondary to the content.

`Pick` gains `top_interests` (serde default, no migration), filled from
the signals stage when the editor assembles the lineup; the facets were
already on the issue snapshot via `Pick.llm`, so earlier issues show the
facet and topic parts and only future issues carry interests. One shared
builder, `understanding_line` in epub::chapters, feeds every surface.

The dev seed gives four picks facets and interests so the line can be
eyeballed locally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4
2026-09-08 19:44:00 +00:00
thalladaandClaude Fable 5.1 0212e407e5 Make code-block whitespace explicit in the X4 edition
The X4 firmware collapses whitespace inside <pre> regardless of
white-space: pre-wrap, so code blocks came out as one run-on line. Turn
newlines into <br/>, tabs and indentation into no-break spaces, and drop the
newline right after <pre> as browsers do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QVPagF6jfDv78CC5Jv2wp4
2026-09-07 17:34:40 +00:00
thalladaandClaude Fable 5.1 d403c51edf Curation v2 step 7: cleanup, prune paths, implementation notes
Dead code and stale v1 comments removed (clippy -W dead_code clean, the
three world.rs warnings fixed), the Brief chapter's TOC title renamed from
"From the Editor", features prune now also sweeps article_assessments and
generate runs the sweep once after publishing, the example config is
tested key-for-key against Config::default(), README commands match
--help, and docs/plans/2026-08-15-implementation-notes.md records the
Anthropic and Voyage facts, the new tables, the budget-day rule and the
lock.

Implemented by a Claude agent from docs/plans/curation-v2-briefs/step7.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 16:32:18 +00:00
thalladaandClaude Fable 5.1 d261cd485d Curation v2 step 6: Behind the paper, stats, run report block, lock
Behind-the-paper chapter (behind.xhtml, both editions) built from the run's
StageCounts and candidate_runs near misses; daily-epub stats [--days N];
StageCounts gains knn/feed gates and verdicts_in_prompt, timings split
into summaries + brief, the four-line §15.4 info block logged once per
run and printed by print_report; src/lock.rs flock guard on
<database_path>.lock for generate, profile rebuild, features backfill and
backfill-social; README updated for the new CLI, env vars and costs.

Implemented by a Claude agent from docs/plans/curation-v2-briefs/step6.md;
reviewed against plan §5, §15.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 16:21:06 +00:00
thalladaandClaude Fable 5.1 57efbb49b4 Curation v2 step 2: Claude Opus 5 editor, the Brief, per-provider budgets
- AnthropicBackend (Messages API, cached system block, output_config.effort,
  server-side fallbacks, refusal surfaced as an error); Llms { bulk, editor }
  with editor_or_bulk(); PriceTable-based UsageMeter per provider.
- [anthropic], [editorial], deepseek.max_concurrent_requests and
  curation.max_article_count config; startup logs resolved providers.
- Budget day is the UTC date of started_at, preloaded from
  runs.provider_costs_json; finish_run writes provider_costs_json and
  config_json. Stage A batches run concurrently with per-batch budget checks.
- Editor prompt with one-line "why" per pick; no minimum lineup size;
  --max-articles is a ceiling; top-up branch deleted; why stored on picks and
  issue_articles.why and rendered in chapters and In this issue.
- Summaries on the editor client (3k-token input, concurrency 4, bulk then
  excerpt fallback); "The Brief" replaces From the Editor; section intros gone.
- Colophon carries per-provider costs and models.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 03:56:14 +00:00
thalladaandClaude Fable 5.1 3a9f4b99e0 Curation v2 step 1: three-way feedback, rating events, reader profile
- Migration 0002: rating_events, article_embeddings, interest_embeddings,
  article_assessments, candidate_runs, runs.config_json/provider_costs_json,
  issue_articles.why; copies ratings into rating_events and drops ratings and
  feed_priors (scores stays until step 4).
- Vote is Loved | Good | NotForMe; legacy `up` links still verify as Loved.
- Footer offers Loved it / Good / Not for me; the confirmation page offers the
  other two so a mis-tap can be corrected. handle_rating appends one event.
- db::current_ratings implements the latest-explicit-event rule with summaries
  and facets; `ratings list|set|clear` CLI appends source='cli' events.
- data/profile.md replaces the hard-coded reader prose; the system prompt is
  rebuilt every run in the §8.4 order with a recent-verdicts block.
- Weekly rebuild reads summaries, facets and notes; feed priors removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 03:23:39 +00:00
thalladaandClaude Opus 5 db19d08257 Break up the image and EPUB god files
The image fixes left two problems of navigation. Image handling was
spread across `extract.rs` (300 lines of normalization) and
`epub/images.rs` (download, re-encode, markup rewriting, plus generic
HTML helpers that comments, world and x4 were all reaching into a
module named "images" to borrow). And `epub/build.rs` had grown to
1,148 lines of code covering cover rendering, ten askama templates,
every chapter renderer and the zip assembly.

New homes:

- `src/html.rs` — markup helpers that do not care what the markup is
  about: tag scanning, attribute parsing, entity decoding, escaping,
  XHTML fixups, reading a fragment as text. Previously scattered between
  `epub/images.rs` and `extract.rs`.
- `src/images/` — one module per stage of an article's images, in the
  order they run: `normalize` (make `<img>` usable, pre-readability),
  `refs` (what an article references), `fetch` + `encode` (download and
  re-encode per edition), `embed` (point the markup at what shipped).
- `src/epub/{cover,chapters,build}.rs` — the cover, the chapter
  renderers, and the ordering plus assembly that puts them together.
  `epub/fixtures.rs` takes the shared test issue, which was a public
  module wedged inside `build.rs`.
- `src/curate/profile/themes.rs` — a 260-line keyword table that sat in
  the middle of the profile logic.

`curate::html_to_text` is renamed `prompt_text`: it is a different
function from `html::html_to_text` (collapses whitespace, no DOM, sized
for prompt budgets) and sharing a name with it was a trap.

Largest module drops from 1,148 code lines to 828, and no file mixes
two subjects. Behaviour is unchanged: 231 lib tests plus 25 integration
tests green, and the real-world audit over issues 1–3 still reports 214
images referenced, 214 shown, 0 placeholders, 0 orphaned assets.

`image_audit` gains `--epub-out DIR`, which writes a readable EPUB of
the audited articles so images can be checked on a device.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 03:35:21 +00:00
thalladaandClaude Opus 5 254eaeb713 Fix the six ways article images went missing
Across the first three issues, 145 of 208 referenced images reached the
page; 38 that had been downloaded, re-encoded and written into the EPUB
were never referenced by any chapter, and 69 became `[image: …]` lines.
Six independent causes, each verified against the real articles:

- **Entity-encoded URLs never matched their asset.** `prepare_body`
  re-runs `ammonia::clean`, which writes `&` as `&amp;`, and the tag
  scanner that looks the asset up compared raw attribute text against a
  URL a real parser had decoded. Every image with a query string lost.
  `parse_attrs` now decodes entities.
- **Readability clobbers a working `src`.** Its lazy-image heuristic
  copies any attribute containing `.jpg` over `src`, which on NPR meant
  a `{width}` template (the CDN answers those with a grey square reading
  "Image"), on Substack a JSON blob, on dfarq an entire `srcset` string.
  Pages are now normalized before readability sees them: every `<img>`
  is reduced to `src`/`alt`/`title` with the best candidate from the
  lazy attributes, `src`, `srcset` and `<picture><source>`, so there is
  nothing left for the heuristic to substitute. Candidates that cannot
  resolve — braces, whitespace, quotes, `data:` — are rejected by shape
  rather than by publisher.
- **Readability deletes `<button>` and its subtree**, taking lightbox
  images with it and leaving the captions behind. Image-only wrappers
  are unwrapped first.
- **SVG was undecodable**, so vector charts became placeholders. They
  are rasterized with resvg, which the cover already depends on.
- **Relative URLs resolved against the pre-redirect URL**, 404ing every
  image on an article reached through a shortener. `fetch_readable` now
  reports where it landed.
- **The 12-image cap** silently truncated photo essays, and the images
  past it were advertised as failures. Removed; the issue-wide byte
  budget is the real backstop.

An image that still cannot be embedded is now dropped rather than
announced, unless its alt text is a real description — decorative rules,
spacers and dead links were generating most of the placeholder noise.

`examples/image_audit.rs` replays the pipeline over the articles of
published issues and reports what reaches the page. On issues 1–3:
208→214 images referenced, 145→214 shown, 69→0 placeholders, 38→0
orphaned assets. The three images still not embedded are a 14×14 favicon
and a 650×2 divider — correctly declined, and no longer announced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 02:51:28 +00:00
thallada 6ba3c3d525 Halve page margins in standard edition EPUBs 2026-08-18 01:36:40 +00:00
thallada 38fb493989 Fix world briefing & add LLM enrichment, colophon fix, x4 cover fix
- World Briefing now starts with yesterday, preserves complete nested
event hierarchies and Wikipedia links, fetches deduplicated article
leads concurrently, and adds stable-ID summaries plus a daily overview.
- Enrichment runs after normal editorial work and degrades safely with
report warnings.
- Colophon facts now use X4-safe block paragraphs.
- Standard cover remains 1200×1600 RGB PNG.
- X4 cover is a 480×800 baseline RGB JPEG with consistent XHTML and OPF
declarations.
- Added CrossInk cache-clearing guidance to README.md.
- Worked around an epub-builder duplicate XML-ID defect discovered by
validation.
2026-08-15 21:13:56 +00:00
thallada 9e30c1dcdf Initial commit: The Daily EPUB full implementation
Full implementation of a personalized daily newspaper delivered as an
EPUB.

Articles are pulled from a local self-hosted Miniflux instance, enriched
with comments, summarized and filtered by DeepSeek AI, and then
assembled into two EPUB editions: standard and optimized for the Xteink
X4 e-ink reader. Both are served by the local self-hosted BookOrbit OPDS
server in a separate library. Then the X4 edition is futher converted to
XTC format and served over a separate OPDS server hosted by the Rust
binary. Runs are tracked in a local SQLite database so runs are
idempotent per date.

Full documentation of the plan is in docs/plans and setup and install
instructions are in the README.md file.
2026-08-15 17:46:19 +00:00