diff --git a/Cargo.toml b/Cargo.toml index e611b8e..9440834 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -59,10 +59,10 @@ tower_governor = { version = "0.8.0", features = ["axum"] } tracing = "0.1.44" tracing-subscriber = { version = "0.3.23", features = ["env-filter"] } url = { version = "2.5.8", features = ["serde"] } +zip = { version = "6", default-features = false, features = ["deflate"] } [dev-dependencies] figment = { version = "0.10.19", features = ["test", "toml", "env"] } roxmltree = "0.21.1" tempfile = "3.27.0" tower = { version = "0.5.3", features = ["util"] } -zip = { version = "6", default-features = false, features = ["deflate"] } diff --git a/README.md b/README.md index a7607e8..34fe907 100644 --- a/README.md +++ b/README.md @@ -189,8 +189,9 @@ the database or takes the lock, so it is safe to run next to a live `generate`. ## Web site and dashboard The server is both the public newspaper index and the private operator UI. An -anonymous visitor sees only titles, authors, sources, metadata and outbound -comment links; generated and scraped text stays private. A signed-in `user` +anonymous visitor sees titles, authors, sources, metadata, AI summaries, why +lines and outbound comment links; article bodies, the Brief, the World Briefing +and comments stay private. A signed-in `user` sees complete issues and article chapters and can download artifacts. An `admin` can additionally rate articles and use every `/dashboard/*` page, including settings and jobs. Personalization is shared across accounts for now. diff --git a/docs/plans/2026-09-03-web-dashboard.md b/docs/plans/2026-09-03-web-dashboard.md index 9e33a59..2be083a 100644 --- a/docs/plans/2026-09-03-web-dashboard.md +++ b/docs/plans/2026-09-03-web-dashboard.md @@ -19,7 +19,7 @@ These are settled. Do not reopen them during implementation. | Frontend | Server-rendered **askama** templates plus one hand-written CSS file and a few hundred lines of plain JavaScript. No node build, no SPA, no framework; assets are embedded in the binary with `include_str!`. | | Auth | Username + password, sessions in a SQLite table, one `HttpOnly; Secure; SameSite=Lax` cookie. Users are created from the CLI (`daily-epub users add`). No third-party identity, no passkeys. **Use the ecosystem, not hand-rolled code**: `axum-login` over `tower-sessions` for login/logout/session lifecycle and route guards, `password-auth` for argon2 hashing, `tower_governor` for the login throttle. axum-login is taken from **git at the pinned rev `151c72d7a1b4646830f86b4332e6bd6e34d719a7`** (`main`, 2026-05-07: tower-sessions 0.15 and the finalized `Require` API), not from the 0.18.0 crates.io release; §2 records why and what was verified. The only auth code we write is a ~60-line `SessionStore` over our own sqlx pool (the official store pins sqlx 0.8; we are on 0.9) and the `AuthnBackend` glue (§6). | | Roles | `user` and `admin`. **Signed-in users of either role** see every issue in full HTML (Brief, summaries, article bodies, discussions, World Briefing, Behind the paper) and can download the EPUB/XTC files. **Admins** additionally rate, browse the dashboard, edit settings and the profile, and start jobs. | -| Copyright boundary | Anonymous visitors never see generated or scraped text: no Brief, no summaries, no `why` lines, no article bodies, no comments, no World Briefing. Sections, reading time and word count are fine. The public renderer takes a dedicated `PublicIssue` type that cannot carry the private fields. | +| Copyright boundary | Anonymous visitors see titles, metadata, AI summaries and `why` lines, but never the Brief, article bodies, comments or World Briefing. The public renderer takes a dedicated `PublicIssue` type that cannot carry those private fields. | | Personalization | One shared algorithm for now. `rating_events` gains a nullable `user_id` so a per-user algorithm is possible later; nothing else is per user. HMAC links from the EPUB stay unattributed (`user_id NULL`, `source = 'epub'`). | | Settings | Every key of `config.toml`, grouped by table, editable from the UI and written back **in place with comments preserved** (`toml_edit`). A key overridden by a `DAILY_EPUB_*` environment variable is shown locked. API keys and the HMAC secret are shown as present/absent only. The page re-reads the file on every view, so hand edits show up. `data/profile.md` gets an editor with version history. | | Actions | A Jobs page starts pipeline work through **systemd**: `systemctl start daily-epub-job@.service`, permitted by a polkit rule for the `daily-epub` user. The server never runs the pipeline in-process (its unit has `MemoryDenyWriteExecute=yes`, which the XTC converter's Node JIT cannot live with). | @@ -438,12 +438,15 @@ pub struct PublicEntry { pub source: String, // feed_title pub domain: String, // host of `url`, "www." stripped pub reading_minutes: i64, pub word_count: i64, + pub summary: Option, pub why: Option, pub comment_links: Vec, // {label, url, meta} pub is_lead: bool, } ``` -`PublicIssue::from(&Issue)` is the only constructor, and it copies exactly these fields. There is deliberately no `summary`, `why`, `body`, `brief`, or `world` field, so the template cannot render them. Test `public_issue_carries_no_generated_text` (§17) renders the fixture issue publicly and asserts that the Brief, every summary, every `why`, every article body sentence and every comment string are absent from the HTML. +`PublicIssue::from(&Issue)` is the only constructor, and it copies exactly these fields. There is deliberately no `body`, `brief`, or `world` field, so the template cannot render them. Test `public_issue_shows_summaries_and_why_but_no_bodies` (§17) renders the fixture issue publicly and asserts that summaries and `why` lines are present while the Brief, every article body sentence and every comment string are absent from the HTML. + +**2026-09-03:** the operator chose to publish summaries and why lines; bodies/Brief/World/comments remain private. `comment_links`: one per `SocialRef` with an `item_url` — label `Hacker News` / `Lobsters` / `Reddit`, meta `"342 points · 210 comments"` — plus `Comments` for `entries.comments_url` when it is set and differs from the article URL. Links carry `rel="noopener"` and `target="_blank"`; article title links are plain `` (the aggregator's purpose is to send readers to the source, so no `nofollow`). @@ -461,12 +464,12 @@ Public pages send `Cache-Control: public, max-age=300` only when no session cook ## 8. Full issue views for signed-in users -`web::issue::load(state, date) -> Result>` (§5.2). `IssueView { issue: Issue, downloads: Vec, from_json: bool }` where `Download { label, href, size_bytes }` lists the standard EPUB, the X4 EPUB and the XTC file **only when the file at `issues.*_path` (or `publish::issue_filename` in `epub_dir`) exists**; `href` is the existing `/files/epub/{name}` or `/files/xtc/{name}`. +`web::issue::load(state, date) -> Result>` (§5.2). `IssueView` carries the reconstructed `Issue`, downloads, snapshot provenance, and an optional trusted World Briefing HTML fragment recovered from a legacy EPUB. `Download { label, href, size_bytes }` lists the standard EPUB, the X4 EPUB and the XTC file **only when the file at `issues.*_path` (or `publish::issue_filename` in `epub_dir`) exists**; `href` is the existing `/files/epub/{name}` or `/files/xtc/{name}`. - `GET /issues/{date}` (signed in) — `issue_full.html`: masthead and dateline; **The Brief** (`editorial.front_page_html`, already sanitized XHTML, rendered `|safe`); download buttons; the index exactly like the EPUB's In-this-issue page: per section, title (→ `/issues/{date}/articles/{id}`), `source · N min read`, summary, `Why it's here`, the rating widget for admins; then links to World Briefing and Behind the paper; the colophon facts (models, cost, counts) in a footer block. - `GET /issues/{date}/articles/{article_id}` — `article.html`: `article-header` (title linking to the source, byline, meta line `feed · 1,850 words · ~8 min`, `Why it's here`, social line via `chapters::social_line`, summary), the body (`ammonia::clean(articles.content_html)` — reuse the same ammonia configuration the EPUB uses; do **not** run `to_xhtml`; `` tags keep their remote `src` and get `loading="lazy" referrerpolicy="no-referrer"`), the discussion (`comments::render_xhtml(discussion, title)` when `pick.discussion` is present; it is XHTML and renders fine as HTML), prev/next links in issue order, "Read online ↗", and the rating widget for admins. 404 when the article is not in that issue. -- `GET /issues/{date}/world` — `world::render_xhtml` in the layout; 404 when the issue has no briefing (fallback issues). -- `GET /issues/{date}/behind` — the same lines as the EPUB chapter (`chapters::behind_*_line`, near misses linking to `/dashboard/articles/{id}` for admins). +- `GET /issues/{date}/world` — `world::render_xhtml` in the layout for snapshots, or trusted `OEBPS/world.xhtml` body content recovered from our own sanitized legacy EPUB; 404 when neither source has a briefing. +- `GET /issues/{date}/behind` — the same lines as the EPUB chapter (`chapters::behind_*_line`, near misses linking to `/dashboard/articles/{id}` for admins), reconstructed from the stored report/run for legacy issues. `/files/epub/{name}` and `/files/xtc/{name}`: `serve_file` accepts either a valid session (any role) **or** the existing Basic auth; when neither is present and Basic auth is configured, it challenges as today; when Basic auth is not configured it redirects HTML clients to `/login?next=` and returns 401 to others. OPDS clients are unaffected. @@ -716,7 +719,7 @@ New dependencies: `axum-login = { git = "https://github.com/maxcountryman/axum-l - Login: `tower_governor` per-IP limit on `POST /login` (§6.5); equal-timing dummy verification for unknown users; no username enumeration in messages ("invalid username or password"). - Authorization: `login_required!`/`permission_required!` route layers on whole sub-routers (§6.3), so a forgotten check in one handler still redirects or 403s; `POST /rate` sits under the admin layer. - Headers on every app response (via a `tower-http` `SetResponseHeader` layer or a small middleware): `Content-Security-Policy: default-src 'self'; img-src * data:; style-src 'self'; script-src 'self'; frame-ancestors 'none'; form-action 'self'`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin`. (`img-src *` because article pages show remote images for signed-in users.) nginx keeps adding its own. -- All user text is escaped by askama; the only `|safe` inputs are `front_page_html` (already sanitized by the pipeline), `ammonia::clean` output, `comments::render_xhtml` and `world::render_xhtml` (built from sanitized comment/portal HTML), and the server-rendered SVG sparklines. +- All user text is escaped by askama; the only `|safe` inputs are `front_page_html` (already sanitized by the pipeline), `ammonia::clean` output, `comments::render_xhtml`, `world::render_xhtml` (built from sanitized comment/portal HTML), a recovered World Briefing body read from our own sanitized EPUB XHTML, and the server-rendered SVG sparklines. - Settings: secrets never rendered, never written; validation through `Config::load` before rename; permissions preserved; every change attributed. `Path` fields are written as given — the operator is the admin, and the config validates paths on load. - Jobs: unit names come only from the fixed catalogue regex; the polkit rule allows `start` only, on that regex only, for that user only. - Files: `safe_join` unchanged. `robots.txt` disallows private paths. Public caching only without a cookie. @@ -735,8 +738,8 @@ No test touches the network or systemd. Router-level tests use `tower::ServiceEx - **Guards**: anonymous `/dashboard` → 302 `/login?next=/dashboard`; signed-in `user` role → 403 site page; admin → 200; `POST /rate` follows the same three outcomes. - **Origin check**: POST with `Sec-Fetch-Site: cross-site` → 403; absent fetch metadata and a foreign `Origin` → 403; same-origin passes; the HMAC rating route (GET) is untouched. - **Throttle**: with a test governor config of burst 3, the fourth `POST /login` from one address → 429; another address still passes. -- **Public rendering**: `public_issue_carries_no_generated_text` (§7.1); comment links built from social refs and `comments_url`; the World Briefing is absent publicly; the feed validates as XML, has one entry per issue, and its content is the public list; `robots.txt` content; `Cache-Control` public without a cookie, `no-store` with one. -- **Full rendering**: signed-in `/issues/{date}` contains the Brief, summaries and `why`; article page contains the body and the discussion; the fallback loader (no `issue_json`) renders an issue from rows with no world/discussion links; downloads listed only for files that exist; `/files/epub` accepts a session, still challenges Basic auth when configured and no session. +- **Public rendering**: `public_issue_shows_summaries_and_why_but_no_bodies` (§7.1); comment links built from social refs and `comments_url`; the Brief, article bodies, comments and World Briefing are absent publicly; the feed validates as XML, has one entry per issue, and its content is the public list with summaries and why lines; `robots.txt` content; `Cache-Control` public without a cookie, `no-store` with one. +- **Full rendering**: signed-in `/issues/{date}` contains the Brief, summaries and `why`; article page contains the body and the discussion; the fallback loader (no `issue_json`) reconstructs colophon/Behind facts and recovers the World chapter from an available EPUB while discussion links remain unavailable; downloads listed only for files that exist; `/files/epub` accepts a session, still challenges Basic auth when configured and no session. - **issue_json**: `record_issue` writes it with empty bodies; the loader rehydrates bodies; `runs.report_json` and `issues.report_json` written; `/issues.json` returns real reports. - **Rating**: `POST /rate` as admin appends a `dashboard` event with `user_id`; as user → 403; anonymous → redirect/401; `cleared` writes value 0; JSON and form variants; `next` validated. - **Dashboard queries**: funnel counts match a seeded `candidate_runs` set; candidate filters and sorts are allow-listed (an unknown sort falls back, never errors); config diff finds changed dotted keys and ignores unchanged; articles list filters; article detail shows assessments, run history and rating events. @@ -806,6 +809,7 @@ Each step is a shippable commit or small series; `cargo fmt --check`, `cargo cli | Search over article bodies (FTS5) | Title/URL `LIKE` feels slow or insufficient. | | Live log streaming (SSE) on the job page | The 5-second refresh feels slow. | | Passkeys / WebAuthn | A second admin or a phishing concern. | -| Public per-article "why" lines | The operator decides the second-person tone reads fine publicly (one field added to `PublicEntry`, one template line, one test change). | | Backfilling `issue_json` for old issues from the EPUB files on disk | Only the last `retention_days` of EPUBs exist; the fallback renderer covers the rest. | | Charts beyond sparklines | `stats` grows a question the tables cannot answer. | + +**2026-09-03:** the operator chose to publish summaries and why lines; bodies/Brief/World/comments remain private. Public per-article why lines are therefore no longer deferred. diff --git a/docs/plans/briefs/web-dashboard-v2/handoff-step1.md b/docs/plans/briefs/web-dashboard-v2/handoff-step1.md new file mode 100644 index 0000000..72c02b6 --- /dev/null +++ b/docs/plans/briefs/web-dashboard-v2/handoff-step1.md @@ -0,0 +1,64 @@ +# Web dashboard v2 — step 1 handoff + +## What landed + +- Public issue pages and Atom entries now show each pick's trimmed summary and + its “Why it's here” line. The public view model still has no article body, + Brief, World Briefing, or discussion/comment content, and the public tests + assert those fields do not leak. +- Legacy `issues.issue_json IS NULL` loads now recover their colophon and + Behind the paper data from `issues.report_json`, matching the report to its + run by `started_at` for near misses. If the issue report is absent, the loader + uses the latest finished run for that issue date. Missing historical values + render as `n/a` in the web colophon rather than misleading zeroes. +- Historical model names come from the resolved run config. Summary-provider + resolution follows `editorial.summary_model`; missing model metadata falls + back to the current config and is called out in the generator note. +- Legacy World Briefings are recovered from `OEBPS/world.xhtml` in the stored + standard EPUB (or its canonical publish path). The EPUB body is used after + removing its own leading heading; missing or corrupt files are debug-logged + and treated as “no World chapter,” not as request errors. +- The development seed's 2026-09-01 issue now has a real report, a real EPUB + path with fixture World Briefing content, `issue_json = NULL`, and three + `candidate_runs` near misses. The 2026-09-02 issue retains a full snapshot. +- README and the dashboard plan now document the operator's 2026-09-03 public + summary/why decision and list recovered EPUB XHTML among the trusted `|safe` + inputs. `zip` moved from dev-only to runtime dependencies. + +## Tests + +- Focused web issue tests: **10 passed**, including report and run-only + colophon fallback, recovered World/Behind routes, missing EPUB behavior, and + public/feed privacy assertions. +- Public view-model test: **1 passed**. +- Development seed smoke test: successfully generated the database and EPUB; + SQL verification showed the legacy row has a null snapshot, present report + and EPUB path, plus **3** near-miss rows. +- `cargo fmt`: clean. +- `cargo clippy --all-targets -- -D warnings`: clean. +- Unfiltered `cargo test`: library result **427 passed, 13 failed**. Every + failure was a pre-existing loopback-listener test rejected by the sandbox. +- Filtered sandbox run: **461 passed, 0 failed, 15 filtered** across library, + binary, integration, and doc-test targets. The filters were the 13 loopback + unit tests and both `tests/m7_server.rs` TCP tests. + +## Deviations and follow-up + +- `00-shared.md` lists four Anthropic listener tests, one extraction listener + test, five server listener tests, and `m7_server` as sandbox-only failures. + The code also has three pre-existing OpenAI mock-server tests that bind the + same forbidden loopback listener; they failed with the identical + `PermissionDenied` error. `docs/plans/2026-08-15-implementation-notes.md` + already lists those three. They were therefore filtered for the complete + sandbox verification and should run normally in the orchestrator. +- No migrations, routes, auth behavior, or security headers changed. No + database backfill is performed. +- This sandbox mounts the worktree's shared Git metadata under + `/home/thallada/workspace/the-daily-epub/.git/worktrees/the-daily-epub-v2a` + read-only. The required `git add` failed while creating `index.lock`, so the + verified changes remain unstaged and the single requested commit still needs + to be created by the orchestrator with message `Web dashboard v2 step 1: + public summaries, legacy issue fallback`. No source or data in the forbidden + main checkout was changed. +- Beyond that commit and the orchestrator's unsandboxed full-suite run, nothing + remains for step 1. This was not a design step, so no screenshots were taken. diff --git a/examples/seed_dev_db.rs b/examples/seed_dev_db.rs index ae4e5c6..69c9960 100644 --- a/examples/seed_dev_db.rs +++ b/examples/seed_dev_db.rs @@ -16,10 +16,12 @@ use std::collections::BTreeMap; use std::path::{Path, PathBuf}; +use daily_epub::config::Config; +use daily_epub::curate::telemetry::{self, CandidateRun}; use daily_epub::db::Db; use daily_epub::epub::fixtures; -use daily_epub::report::{RunReport, RunStatus}; -use daily_epub::types::{Article, Entry, Issue, Pick}; +use daily_epub::report::{ProviderUsage, RunReport}; +use daily_epub::types::{Article, Edition, Entry, Issue, Pick, TokenUsage}; use daily_epub::web::users; use jiff::Timestamp; use jiff::civil::Date; @@ -116,7 +118,7 @@ fn story_article(index: usize, story: &(&str, &str, &str, &str, &str, i64)) -> A source.feed_id = article.feed_id; } article.content_html = body_html(title, words); - if index % 3 != 0 { + if !index.is_multiple_of(3) { article.social.clear(); } if index == 4 { @@ -198,7 +200,12 @@ fn dev_issue(date: Date, issue_number: i64, generated_at: Timestamp) -> Issue { issue } -async fn seed_issue(db: &Db, issue: &mut Issue, with_snapshot: bool) -> anyhow::Result<()> { +async fn seed_issue( + db: &Db, + issue: &mut Issue, + with_snapshot: bool, + epub_path: Option<&Path>, +) -> anyhow::Result { for pick in &mut issue.lineup.picks { let article = &pick.article; db.upsert_entry(&Entry { @@ -232,12 +239,13 @@ async fn seed_issue(db: &Db, issue: &mut Issue, with_snapshot: bool) -> anyhow:: serde_json::to_string(&snapshot) }) .transpose()?; - let run_id = db - .start_run(issue.meta.date, issue.meta.generated_at) - .await?; - let mut report = RunReport::new(issue.meta.date, issue.meta.generated_at); - report.finished_at = Some(issue.meta.generated_at); - report.status = RunStatus::Ok; + let started_at = issue + .meta + .generated_at + .checked_sub(jiff::Span::new().minutes(22)) + .unwrap_or(issue.meta.generated_at); + let run_id = db.start_run(issue.meta.date, started_at).await?; + let mut report = RunReport::new(issue.meta.date, started_at); report.counts.entries_fetched = issue.colophon.entries_fetched; report.counts.feeds_seen = issue.colophon.feeds_seen; report.counts.articles = issue.behind.considered; @@ -252,17 +260,37 @@ async fn seed_issue(db: &Db, issue: &mut Issue, with_snapshot: bool) -> anyhow:: report.counts.knn_gate = issue.behind.knn_gate; report.counts.feed_gate = issue.behind.feed_gate; report.counts.embedded = 300; - report.cost_usd = issue.colophon.cost_usd; + report.provider_costs = issue + .colophon + .provider_costs + .iter() + .map(|(provider, cost_usd)| { + ( + provider.clone(), + ProviderUsage { + usage: TokenUsage::default(), + cost_usd: *cost_usd, + }, + ) + }) + .collect(); report.config_json = serde_json::json!({ - "llm": {"bulk": {"model": issue.colophon.models.bulk}, "editor": {"model": issue.colophon.models.editor}}, - "voyage": {"model": "voyage-4"}, + "models": { + "bulk": issue.colophon.models.bulk, + "editor": issue.colophon.models.editor, + "embedding": issue.behind.embedding_model, + }, + "editorial": {"summary_model": "editor"}, + "voyage": {"model": issue.behind.embedding_model}, }); + report.finish(issue.meta.generated_at); db.finish_run(run_id, &report).await?; + let epub_path = epub_path.map(|path| path.to_string_lossy().into_owned()); db.upsert_issue( issue.meta.date, issue.meta.issue_number, issue.meta.generated_at, - None, + epub_path.as_deref(), None, None, Some(&issue.editorial.front_page_html), @@ -272,6 +300,71 @@ async fn seed_issue(db: &Db, issue: &mut Issue, with_snapshot: bool) -> anyhow:: .await?; db.replace_issue_articles(issue.meta.date, &issue.lineup.picks) .await?; + Ok(run_id) +} + +async fn seed_near_misses(db: &Db, run_id: i64) -> anyhow::Result<()> { + for (index, title, quality, fit, utility) in [ + ( + 0, + "The Database Migration That Almost Made the Cut", + 8.4, + 7.2, + 81.5, + ), + ( + 1, + "An Oral History of the First E-Ink Hackers", + 7.7, + 8.1, + 79.8, + ), + (2, "A Very Good Essay About Municipal Trees", 8.0, 6.8, 76.2), + ] { + let mut article = fixtures::article(0, 9001 + index, title); + article.feed_title = "The Near-Miss Review".into(); + let entry = Entry { + id: article.best_entry_id, + feed_id: article.feed_id, + feed_title: Some(article.feed_title.clone()), + category: article.category.clone(), + title: article.title.clone(), + url: article.url.clone(), + canonical_url: Some(article.canonical_url.clone()), + author: article.author.clone(), + published_at: article.published_at, + comments_url: article.comments_url.clone(), + raw_content: article.content_html.clone(), + fetched_at: article.first_seen, + }; + db.upsert_entry(&entry).await?; + let article_id = db.upsert_article(&article).await?; + let signals = serde_json::json!({ + "v": 1, + "raw": {"quality": quality, "fit": fit}, + "norm": {"quality": quality / 10.0, "fit": fit / 10.0}, + "present": {"quality": true, "fit": true}, + "weights": {"quality": 0.6, "fit": 0.4}, + }) + .to_string(); + telemetry::write( + db, + &CandidateRun { + run_id, + article_id, + stage: "shortlisted", + excluded_reason: Some("not_selected"), + admitted_by: Some("[\"blend\"]"), + signals_json: &signals, + utility: Some(utility), + rank_utility: Some(index + 10), + cluster_id: Some(index), + cluster_rank: Some(1), + editor_why: None, + }, + ) + .await?; + } Ok(()) } @@ -310,10 +403,20 @@ async fn main() -> anyhow::Result<()> { // Legacy issue: published before `issue_json` existed, so the site must // rebuild the colophon and back matter from the run instead. let mut legacy = dev_issue("2026-09-01".parse()?, 18, "2026-09-01T05:31:00Z".parse()?); - seed_issue(&db, &mut legacy, false).await?; + let mut epub_config = Config::default(); + epub_config.publish.epub_dir = dir.join("epubs"); + let legacy_epub = daily_epub::epub::build_edition_with_images( + &legacy, + Edition::Standard, + &epub_config, + &epub_config.publish.epub_dir, + &[], + )?; + let legacy_run = seed_issue(&db, &mut legacy, false, Some(&legacy_epub.path)).await?; + seed_near_misses(&db, legacy_run).await?; let mut latest = dev_issue("2026-09-02".parse()?, 19, "2026-09-02T05:29:00Z".parse()?); - seed_issue(&db, &mut latest, true).await?; + seed_issue(&db, &mut latest, true, None).await?; println!("seeded {}", db_path.display()); println!( diff --git a/src/web/issue.rs b/src/web/issue.rs index 0efa303..d11af68 100644 --- a/src/web/issue.rs +++ b/src/web/issue.rs @@ -1,4 +1,5 @@ use std::collections::{BTreeMap, HashMap, HashSet}; +use std::io::Read; use std::path::{Path as FsPath, PathBuf}; use anyhow::Context; @@ -14,7 +15,7 @@ use crate::epub::chapters; use crate::pipeline::display_date; use crate::server::AppState; use crate::types::{ - ArticleId, BehindThePaper, Colophon, Edition, Editorial, Issue, IssueMeta, Lineup, Pick, + ArticleId, BehindThePaper, Colophon, Edition, Editorial, Issue, IssueMeta, Lineup, Models, Pick, }; use crate::web::rate::{self, RatingWidget}; use crate::web::session::{AuthSession, Viewer}; @@ -33,6 +34,42 @@ pub struct IssueView { pub issue: Issue, pub downloads: Vec, pub from_json: bool, + pub world_html: Option, + has_behind: bool, + legacy_counts: Option, +} + +#[derive(Debug, Clone, Default)] +struct LegacyColophonCounts { + entries_fetched: Option, + feeds_seen: Option, + candidates: Option, + cost_usd: Option, +} + +#[derive(Debug, Default)] +struct LegacyFacts { + has_source: bool, + run_id: Option, + entries_fetched: Option, + feeds_seen: Option, + considered: Option, + eligible: Option, + triaged: Option, + assessed: Option, + shortlisted: Option, + candidates: Option, + selected: Option, + admitted_by: BTreeMap, + rated_with_embeddings: Option, + knn_gate: Option, + feed_gate: Option, + embedded: Option, + provider_costs: BTreeMap, + cost_usd: Option, + config_json: Option, + started_at: Option, + finished_at: Option, } pub async fn load( @@ -43,6 +80,11 @@ pub async fn load( let Some(row) = db.issue_by_date(date).await? else { return Ok(None); }; + let legacy = if row.issue_json.is_none() { + Some(load_legacy_facts(db, date, row.report_json.as_deref()).await?) + } else { + None + }; let (mut issue, from_json) = if let Some(raw) = row.issue_json.as_deref() { let mut issue: Issue = serde_json::from_str(raw).context("decoding issues.issue_json")?; for pick in &mut issue.lineup.picks { @@ -114,6 +156,33 @@ pub async fn load( let total_words = picks.iter().map(|pick| pick.article.word_count).sum(); let article_count = picks.len() as i64; let section_count = section_order.len() as i64; + let Some(facts) = legacy.as_ref() else { + return Err(anyhow::anyhow!("legacy issue facts were not loaded")); + }; + let (models, embedding_model, models_from_current_config) = + legacy_models(facts.config_json.as_ref(), facts.embedded, config); + let near_misses = if let Some(run_id) = facts.run_id { + match crate::curate::telemetry::paper_near_misses(db, run_id, 10).await { + Ok(near_misses) => near_misses, + Err(error) => { + tracing::debug!(%error, %date, run_id, "could not recover legacy near misses"); + Vec::new() + } + } + } else { + Vec::new() + }; + let generation_secs = facts + .started_at + .zip(facts.finished_at) + .map(|(started, finished)| (finished.as_second() - started.as_second()).max(0)) + .unwrap_or(0); + let generator_version = if models_from_current_config { + "not recorded (issue predates snapshots; model names from the current configuration)" + .to_string() + } else { + "not recorded (issue predates snapshots)".to_string() + }; ( Issue { meta: IssueMeta { @@ -132,17 +201,47 @@ pub async fn load( section_order, }, editorial: Editorial { - front_page_html: row.front_page_html.unwrap_or_default(), + front_page_html: row.front_page_html.clone().unwrap_or_default(), summaries, }, world_briefing: None, - colophon: Colophon::default(), - behind: BehindThePaper::default(), + colophon: Colophon { + provider_costs: facts.provider_costs.clone(), + models: models.clone(), + entries_fetched: facts.entries_fetched.unwrap_or(0), + feeds_seen: facts.feeds_seen.unwrap_or(0), + candidates: facts.candidates.unwrap_or(0), + cost_usd: facts.cost_usd.unwrap_or(0.0), + generator_version, + }, + behind: BehindThePaper { + considered: facts.considered.unwrap_or(0), + feeds_seen: facts.feeds_seen.unwrap_or(0), + eligible: facts.eligible.unwrap_or(0), + triaged: facts.triaged.unwrap_or(0), + read_closely: facts.assessed.unwrap_or(0), + shortlisted: facts.shortlisted.unwrap_or(0), + selected: facts.selected.unwrap_or(article_count), + admitted_by: facts.admitted_by.clone(), + rated_with_embeddings: facts.rated_with_embeddings.unwrap_or(0), + knn_gate: facts.knn_gate.unwrap_or(0.0), + feed_gate: facts.feed_gate.unwrap_or(0.0), + near_misses, + models, + embedding_model, + cost_usd: facts.cost_usd.unwrap_or(0.0), + generation_secs, + }, }, false, ) }; issue.meta.article_count = issue.lineup.picks.len() as i64; + let world_html = if from_json || issue.world_briefing.is_some() { + None + } else { + recover_world_html(&row, config) + }; let downloads = [ ( "EPUB", @@ -173,9 +272,330 @@ pub async fn load( issue, downloads, from_json, + world_html, + has_behind: from_json || legacy.as_ref().is_some_and(|facts| facts.has_source), + legacy_counts: legacy.map(|facts| LegacyColophonCounts { + entries_fetched: facts.entries_fetched, + feeds_seen: facts.feeds_seen, + candidates: facts.candidates, + cost_usd: facts.cost_usd, + }), })) } +async fn load_legacy_facts( + db: &Db, + date: Date, + issue_report_json: Option<&str>, +) -> anyhow::Result { + let issue_report = + issue_report_json.and_then(|raw| parse_legacy_json(raw, "issues.report_json")); + let report_started_at = issue_report + .as_ref() + .and_then(|report| report.get("started_at")) + .and_then(serde_json::Value::as_str); + let run = if let Some(started_at) = report_started_at { + sqlx::query( + "SELECT id, started_at, finished_at, entries_fetched, candidates, selected, + cost_usd, provider_costs_json, config_json, report_json + FROM runs + WHERE date = ? AND started_at = ? AND finished_at IS NOT NULL + ORDER BY id DESC LIMIT 1", + ) + .bind(date.to_string()) + .bind(started_at) + .fetch_optional(db.pool()) + .await? + } else { + sqlx::query( + "SELECT id, started_at, finished_at, entries_fetched, candidates, selected, + cost_usd, provider_costs_json, config_json, report_json + FROM runs + WHERE date = ? AND finished_at IS NOT NULL + ORDER BY finished_at DESC, id DESC LIMIT 1", + ) + .bind(date.to_string()) + .fetch_optional(db.pool()) + .await? + }; + let run_report = run + .as_ref() + .and_then(|row| row.get::, _>("report_json")) + .as_deref() + .and_then(|raw| parse_legacy_json(raw, "runs.report_json")); + let report = issue_report.as_ref().or(run_report.as_ref()); + + let run_config = run + .as_ref() + .and_then(|row| row.get::, _>("config_json")) + .as_deref() + .and_then(|raw| parse_legacy_json(raw, "runs.config_json")); + let config_json = report + .and_then(|value| value.get("config_json")) + .filter(|value| !value.is_null()) + .cloned() + .or(run_config); + + let run_provider_costs = run + .as_ref() + .and_then(|row| row.get::, _>("provider_costs_json")) + .as_deref() + .and_then(|raw| parse_legacy_json(raw, "runs.provider_costs_json")) + .as_ref() + .map(provider_costs) + .unwrap_or_default(); + let report_provider_costs = report + .and_then(|value| value.get("provider_costs")) + .map(provider_costs) + .unwrap_or_default(); + + let report_timestamp = |name: &str| { + report + .and_then(|value| value.get(name)) + .and_then(serde_json::Value::as_str) + .and_then(|raw| raw.parse().ok()) + }; + let run_timestamp = |name: &str| { + run.as_ref() + .and_then(|row| row.get::, _>(name)) + .and_then(|raw| raw.parse().ok()) + }; + let run_i64 = |name: &str| run.as_ref().map(|row| row.get::(name)); + let run_f64 = |name: &str| run.as_ref().map(|row| row.get::(name)); + + Ok(LegacyFacts { + has_source: issue_report_json.is_some() || run.is_some(), + run_id: run.as_ref().map(|row| row.get("id")), + entries_fetched: report_count(report, "entries_fetched") + .or_else(|| run_i64("entries_fetched")), + feeds_seen: report_count(report, "feeds_seen"), + considered: report_count(report, "articles"), + eligible: report_count(report, "eligible"), + triaged: report_count(report, "triaged"), + assessed: report_count(report, "assessed"), + shortlisted: report_count(report, "shortlisted"), + candidates: report_count(report, "candidates").or_else(|| run_i64("candidates")), + selected: report_count(report, "selected").or_else(|| run_i64("selected")), + admitted_by: report + .and_then(|value| value.pointer("/counts/admitted_by")) + .and_then(serde_json::Value::as_object) + .map(|values| { + values + .iter() + .filter_map(|(name, count)| count.as_i64().map(|count| (name.clone(), count))) + .collect() + }) + .unwrap_or_default(), + rated_with_embeddings: report_count(report, "rated_with_embeddings"), + knn_gate: report_count_f64(report, "knn_gate"), + feed_gate: report_count_f64(report, "feed_gate"), + embedded: report_count(report, "embedded"), + provider_costs: if report_provider_costs.is_empty() { + run_provider_costs + } else { + report_provider_costs + }, + cost_usd: report + .and_then(|value| value.get("cost_usd")) + .and_then(serde_json::Value::as_f64) + .or_else(|| run_f64("cost_usd")), + config_json, + started_at: report_timestamp("started_at").or_else(|| run_timestamp("started_at")), + finished_at: report_timestamp("finished_at").or_else(|| run_timestamp("finished_at")), + }) +} + +fn parse_legacy_json(raw: &str, column: &str) -> Option { + match serde_json::from_str(raw) { + Ok(value) => Some(value), + Err(error) => { + tracing::debug!(%error, column, "could not decode legacy issue metadata"); + None + } + } +} + +fn report_count(report: Option<&serde_json::Value>, name: &str) -> Option { + report + .and_then(|value| value.get("counts")) + .and_then(|counts| counts.get(name)) + .and_then(serde_json::Value::as_i64) +} + +fn report_count_f64(report: Option<&serde_json::Value>, name: &str) -> Option { + report + .and_then(|value| value.get("counts")) + .and_then(|counts| counts.get(name)) + .and_then(serde_json::Value::as_f64) +} + +fn provider_costs(value: &serde_json::Value) -> BTreeMap { + value + .as_object() + .into_iter() + .flatten() + .filter_map(|(provider, usage)| { + usage + .as_f64() + .or_else(|| usage.get("cost_usd").and_then(serde_json::Value::as_f64)) + .map(|cost| (provider.clone(), cost)) + }) + .collect() +} + +fn legacy_models( + stored: Option<&serde_json::Value>, + embedded: Option, + config: &crate::config::Config, +) -> (Models, String, bool) { + let stored_model = |role: &str| { + stored + .and_then(|value| value.pointer(&format!("/models/{role}"))) + .and_then(serde_json::Value::as_str) + .map(str::to_string) + .or_else(|| { + stored + .and_then(|value| value.pointer(&format!("/llm/{role}/model"))) + .and_then(serde_json::Value::as_str) + .map(str::to_string) + }) + .or_else(|| { + let provider = stored + .and_then(|value| value.pointer(&format!("/llm/{role}"))) + .and_then(serde_json::Value::as_str)?; + stored + .and_then(|value| value.pointer(&format!("/providers/{provider}/model"))) + .and_then(serde_json::Value::as_str) + .map(str::to_string) + }) + }; + + let current_bulk = || { + config + .bulk_provider() + .map(|(_, provider)| provider.model.clone()) + .unwrap_or_else(|| "none".into()) + }; + let mut used_current = false; + let bulk = stored_model("bulk").unwrap_or_else(|| { + used_current = true; + current_bulk() + }); + let stored_editor = stored_model("editor"); + let editor = match stored_editor.as_deref() { + Some("disabled") | Some("none") => format!("{bulk} (bulk fallback)"), + Some(editor) => editor.to_string(), + None => { + used_current = true; + config + .editor_provider() + .map(|(_, provider)| provider.model.clone()) + .unwrap_or_else(|| format!("{bulk} (bulk fallback)")) + } + }; + let summary_role = stored + .and_then(|value| value.pointer("/editorial/summary_model")) + .and_then(serde_json::Value::as_str); + let summaries = match summary_role { + Some("bulk") => bulk.clone(), + Some("editor") => match stored_editor.as_deref() { + Some("disabled") | Some("none") => bulk.clone(), + _ => editor.clone(), + }, + _ => { + used_current = true; + match config.editorial.summary_model { + crate::config::SummaryModel::Editor if config.editor_provider().is_some() => config + .editor_provider() + .map(|(_, provider)| provider.model.clone()) + .unwrap_or_else(|| bulk.clone()), + _ => bulk.clone(), + } + } + }; + let configured_embedding = stored + .and_then(|value| value.pointer("/models/embedding")) + .and_then(serde_json::Value::as_str) + .or_else(|| { + stored + .and_then(|value| value.pointer("/voyage/model")) + .and_then(serde_json::Value::as_str) + }) + .map(str::to_string) + .unwrap_or_else(|| { + used_current = true; + if config.voyage.enabled { + config.voyage.model.clone() + } else { + "disabled".into() + } + }); + let embedding = if embedded == Some(0) { + "none".to_string() + } else { + configured_embedding + }; + ( + Models { + bulk, + editor, + summaries, + }, + embedding, + used_current, + ) +} + +fn recover_world_html(row: &crate::db::IssueRow, config: &crate::config::Config) -> Option { + let fallback = config.publish.epub_dir.join(crate::publish::issue_filename( + row.date, + Edition::Standard, + "epub", + )); + let path = row + .epub_path + .as_deref() + .map(FsPath::new) + .filter(|path| path.is_file()) + .map(FsPath::to_path_buf) + .or_else(|| fallback.is_file().then_some(fallback))?; + match read_world_html(&path) { + Ok(html) => Some(html), + Err(error) => { + tracing::debug!(%error, path = %path.display(), "could not recover legacy World Briefing"); + None + } + } +} + +fn read_world_html(path: &FsPath) -> anyhow::Result { + let file = std::fs::File::open(path).context("opening legacy EPUB")?; + let mut archive = zip::ZipArchive::new(file).context("opening legacy EPUB archive")?; + let mut chapter = archive + .by_name("OEBPS/world.xhtml") + .context("reading OEBPS/world.xhtml")?; + let mut xhtml = String::new(); + chapter + .read_to_string(&mut xhtml) + .context("decoding OEBPS/world.xhtml")?; + world_body_without_heading(&xhtml).context("finding the World Briefing body") +} + +fn world_body_without_heading(xhtml: &str) -> Option { + let body_start = xhtml.find("")? + content_start; + let mut body = xhtml[content_start..content_end].to_string(); + if let Some(heading_start) = body.find("") + { + let heading_end = heading_open_end + relative_end + "".len(); + body.replace_range(heading_start..heading_end, ""); + } + Some(body.trim().to_string()).filter(|body| !body.is_empty()) +} + fn download( label: &str, raw: Option<&str>, @@ -243,14 +663,14 @@ struct ColophonView { editor_model: String, summaries_model: String, provider_costs: Vec, - entries_fetched: i64, - feeds_seen: i64, - candidates: i64, + entries_fetched: Option, + feeds_seen: Option, + candidates: Option, article_count: i64, section_count: i64, total_words: String, reading_minutes: i64, - cost_usd: String, + cost_usd: Option, generator_version: String, } @@ -301,7 +721,7 @@ struct ArticleTemplate { #[template(path = "world.html")] struct WorldTemplate { page: Page, - display_date: String, + display_date: Option, body_html: String, issue_href: String, } @@ -368,7 +788,7 @@ pub async fn render_full( name, }) .collect(); - let colophon = colophon_view(&view.issue); + let colophon = colophon_view(&view.issue, view.legacy_counts.as_ref()); let mut page = Page::new(format!("Issue {date}"), Some(viewer), "latest"); page.flash = take_flash(session).await?; Ok(Html(IssueFullTemplate { @@ -379,8 +799,8 @@ pub async fn render_full( front_page_html: view.issue.editorial.front_page_html.clone(), downloads: view.downloads, sections, - has_world: view.issue.world_briefing.is_some(), - has_behind: view.from_json, + has_world: view.issue.world_briefing.is_some() || view.world_html.is_some(), + has_behind: view.has_behind, date, colophon, }) @@ -489,15 +909,22 @@ pub async fn world( let Some(view) = load(&state.db, &state.config(), date).await? else { return Err(WebError::NotFound); }; - let Some(briefing) = view.issue.world_briefing else { + let (display_date, body_html) = if let Some(briefing) = view.issue.world_briefing { + ( + Some(display_date(briefing.date)), + crate::world::render_xhtml(&briefing), + ) + } else if let Some(body_html) = view.world_html { + (None, body_html) + } else { return Err(WebError::NotFound); }; let mut page = Page::new("World Briefing", Some(viewer), "latest"); page.flash = take_flash(&session).await?; Ok(Html(WorldTemplate { page, - display_date: display_date(briefing.date), - body_html: crate::world::render_xhtml(&briefing), + display_date, + body_html, issue_href: format!("/issues/{date}"), }) .into_response()) @@ -519,7 +946,7 @@ pub async fn behind( let Some(view) = load(&state.db, &state.config(), date).await? else { return Err(WebError::NotFound); }; - if !view.from_json { + if !view.has_behind { return Err(WebError::NotFound); } let behind = &view.issue.behind; @@ -561,8 +988,21 @@ fn summary_for<'a>(issue: &'a Issue, pick: &'a Pick) -> Option<&'a str> { .filter(|summary| !summary.trim().is_empty()) } -fn colophon_view(issue: &Issue) -> ColophonView { +fn colophon_view(issue: &Issue, legacy_counts: Option<&LegacyColophonCounts>) -> ColophonView { let colophon = &issue.colophon; + let entries_fetched = legacy_counts + .map(|counts| counts.entries_fetched) + .unwrap_or(Some(colophon.entries_fetched)); + let feeds_seen = legacy_counts + .map(|counts| counts.feeds_seen) + .unwrap_or(Some(colophon.feeds_seen)); + let candidates = legacy_counts + .map(|counts| counts.candidates) + .unwrap_or(Some(colophon.candidates)); + let cost_usd = legacy_counts + .map(|counts| counts.cost_usd) + .unwrap_or(Some(colophon.cost_usd)) + .map(|cost| format!("${cost:.4}")); ColophonView { generated_at: issue.meta.generated_at.to_string(), bulk_model: colophon.models.bulk.clone(), @@ -576,14 +1016,14 @@ fn colophon_view(issue: &Issue) -> ColophonView { cost: format!("${cost:.4}"), }) .collect(), - entries_fetched: colophon.entries_fetched, - feeds_seen: colophon.feeds_seen, - candidates: colophon.candidates, + entries_fetched, + feeds_seen, + candidates, article_count: issue.meta.article_count, section_count: issue.meta.section_count, total_words: thousands(issue.meta.total_words), reading_minutes: issue.meta.reading_minutes, - cost_usd: format!("${:.4}", colophon.cost_usd), + cost_usd, generator_version: if colophon.generator_version.is_empty() { format!("daily-epub {}", env!("CARGO_PKG_VERSION")) } else { @@ -732,15 +1172,120 @@ mod tests { } serde_json::to_string(&snapshot).unwrap() }); + let epub_path = if with_json { + None + } else { + Some( + crate::epub::build_edition_with_images( + &issue, + Edition::Standard, + &crate::config::Config::default(), + dir.path(), + &[], + ) + .unwrap() + .path, + ) + }; + let started_at = issue + .meta + .generated_at + .checked_sub(jiff::Span::new().minutes(23)) + .unwrap(); + let run_id = db.start_run(issue.meta.date, started_at).await.unwrap(); + let mut report = crate::report::RunReport::new(issue.meta.date, started_at); + report.counts.entries_fetched = issue.colophon.entries_fetched; + report.counts.feeds_seen = issue.colophon.feeds_seen; + report.counts.articles = issue.behind.considered; + report.counts.eligible = issue.behind.eligible; + report.counts.triaged = issue.behind.triaged; + report.counts.assessed = issue.behind.read_closely; + report.counts.shortlisted = issue.behind.shortlisted; + report.counts.candidates = issue.colophon.candidates; + report.counts.selected = issue.meta.article_count; + report.counts.admitted_by = issue.behind.admitted_by.clone(); + report.counts.rated_with_embeddings = issue.behind.rated_with_embeddings; + report.counts.knn_gate = issue.behind.knn_gate; + report.counts.feed_gate = issue.behind.feed_gate; + report.counts.embedded = 300; + report.provider_costs = issue + .colophon + .provider_costs + .iter() + .map(|(provider, cost_usd)| { + ( + provider.clone(), + crate::report::ProviderUsage { + usage: crate::types::TokenUsage::default(), + cost_usd: *cost_usd, + }, + ) + }) + .collect(); + report.config_json = json!({ + "models": { + "bulk": issue.colophon.models.bulk, + "editor": issue.colophon.models.editor, + "embedding": issue.behind.embedding_model, + }, + "editorial": {"summary_model": "editor"}, + "voyage": {"model": issue.behind.embedding_model}, + }); + report.finish(issue.meta.generated_at); + db.finish_run(run_id, &report).await.unwrap(); + + if !with_json { + let mut article = crate::epub::fixtures::article(0, 3001, "Legacy near miss"); + article.feed_title = "Near Misses Weekly".into(); + db.upsert_entry(&Entry { + id: article.best_entry_id, + feed_id: article.feed_id, + feed_title: Some(article.feed_title.clone()), + category: article.category.clone(), + title: article.title.clone(), + url: article.url.clone(), + canonical_url: Some(article.canonical_url.clone()), + author: article.author.clone(), + published_at: article.published_at, + comments_url: article.comments_url.clone(), + raw_content: article.content_html.clone(), + fetched_at: article.first_seen, + }) + .await + .unwrap(); + let article_id = db.upsert_article(&article).await.unwrap(); + crate::curate::telemetry::write( + &db, + &crate::curate::telemetry::CandidateRun { + run_id, + article_id, + stage: "shortlisted", + excluded_reason: Some("not_selected"), + admitted_by: Some("[\"blend\"]"), + signals_json: r#"{"v":1,"raw":{"quality":8.2,"fit":7.4}}"#, + utility: Some(81.0), + rank_utility: Some(3), + cluster_id: Some(1), + cluster_rank: Some(2), + editor_why: None, + }, + ) + .await + .unwrap(); + } + let report_json = serde_json::to_string(&report).unwrap(); + let epub_path = epub_path + .as_ref() + .map(|path| path.to_string_lossy().into_owned()); db.upsert_issue( issue.meta.date, issue.meta.issue_number, issue.meta.generated_at, - None, + epub_path.as_deref(), None, None, Some(&issue.editorial.front_page_html), - Some("{\"status\":\"ok\"}"), + Some(&report_json), issue_json.as_deref(), ) .await @@ -793,6 +1338,16 @@ mod tests { ["Top Stories", "Niche Corner"] ); assert!(loaded.issue.world_briefing.is_none()); + assert!(loaded.world_html.as_deref().is_some_and(|html| { + html.contains("Something happened somewhere") && !html.contains("

World Briefing") + })); + assert_eq!(loaded.issue.colophon.entries_fetched, 431); + assert_eq!(loaded.issue.colophon.feeds_seen, 92); + assert_eq!(loaded.issue.colophon.candidates, 120); + assert_eq!(loaded.issue.colophon.models.bulk, "deepseek-v4-flash"); + assert_eq!(loaded.issue.colophon.models.editor, "claude-opus-5"); + assert_eq!(loaded.issue.colophon.models.summaries, "claude-opus-5"); + assert_eq!(loaded.issue.behind.near_misses[0].title, "Legacy near miss"); assert!( loaded .issue @@ -836,8 +1391,12 @@ mod tests { .unwrap(); assert!(html.contains("The Lead Story")); assert!(html.contains("Hacker News")); + assert!(html.contains("What it argues, and why it is worth the time.")); + assert!(html.contains("Why it's here")); assert!(!html.contains("Two stories today")); assert!(!html.contains("Something happened")); + assert!(!html.contains("Body of")); + assert!(!html.contains("write path")); let archive = app .clone() @@ -880,7 +1439,12 @@ mod tests { .count(), 1 ); + assert!(feed.contains("What it argues, and why it is worth the time.")); + assert!(feed.contains("Why it's here")); assert!(!feed.contains("Something happened")); + assert!(!feed.contains("Two stories today")); + assert!(!feed.contains("Body of")); + assert!(!feed.contains("write path")); let robots = app .clone() @@ -1025,7 +1589,7 @@ mod tests { .oneshot( Request::builder() .uri(format!("/issues/{}/articles/999999", source.meta.date)) - .header(header::COOKIE, cookie) + .header(header::COOKIE, &cookie) .body(Body::empty()) .unwrap(), ) @@ -1036,7 +1600,7 @@ mod tests { } #[tokio::test] - async fn fallback_full_issue_omits_ephemeral_chapter_links() { + async fn fallback_full_issue_recovers_colophon_world_and_behind() { let (_dir, db, source) = seeded_issue(false).await; crate::web::users::add(&db, "reader", "correct horse battery", false) .await @@ -1061,20 +1625,101 @@ mod tests { assert_eq!(issue.status(), StatusCode::OK); let issue = response_text(issue).await; assert!(issue.contains("Two stories today")); - assert!(!issue.contains(&format!("/issues/{}/world", source.meta.date))); - assert!(!issue.contains(&format!("/issues/{}/behind", source.meta.date))); + assert!(issue.contains(&format!("/issues/{}/world", source.meta.date))); + assert!(issue.contains(&format!("/issues/{}/behind", source.meta.date))); + assert!(issue.contains("431 from 92 feeds")); + assert!(issue.contains("deepseek-v4-flash")); + assert!(issue.contains("claude-opus-5")); + assert!(!issue.contains("0 from 0 feeds")); let world = app + .clone() .oneshot( Request::builder() .uri(format!("/issues/{}/world", source.meta.date)) + .header(header::COOKIE, &cookie) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(world.status(), StatusCode::OK); + let world = response_text(world).await; + assert!(world.contains("Something happened somewhere")); + assert_eq!(world.matches("World Briefing").count(), 2); // page title + chapter heading + + let behind = app + .oneshot( + Request::builder() + .uri(format!("/issues/{}/behind", source.meta.date)) .header(header::COOKIE, cookie) .body(Body::empty()) .unwrap(), ) .await .unwrap(); - assert_eq!(world.status(), StatusCode::NOT_FOUND); + assert_eq!(behind.status(), StatusCode::OK); + let behind = response_text(behind).await; + assert!(behind.contains("Considered 412 articles")); + assert!(behind.contains("Legacy near miss")); + } + + #[tokio::test] + async fn fallback_uses_finished_run_columns_and_marks_unknown_counts_na() { + let (_dir, db, source) = seeded_issue(false).await; + sqlx::query( + "UPDATE issues SET report_json = NULL, epub_path = '/missing/legacy.epub' + WHERE date = ?", + ) + .bind(source.meta.date.to_string()) + .execute(db.pool()) + .await + .unwrap(); + sqlx::query("UPDATE runs SET report_json = NULL WHERE date = ?") + .bind(source.meta.date.to_string()) + .execute(db.pool()) + .await + .unwrap(); + crate::web::users::add(&db, "reader", "correct horse battery", false) + .await + .unwrap(); + let app = crate::server::router(crate::server::AppState::new( + db, + crate::config::Config::default(), + None, + )); + let cookie = login_cookie(&app, "reader", "correct horse battery").await; + let issue = app + .clone() + .oneshot( + Request::builder() + .uri(format!("/issues/{}", source.meta.date)) + .header(header::COOKIE, &cookie) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(issue.status(), StatusCode::OK); + let issue = response_text(issue).await; + assert!(issue.contains("431 from n/a feeds")); + assert!(issue.contains("120")); + assert!(!issue.contains("0 from 0 feeds")); + assert!(issue.contains(&format!("/issues/{}/behind", source.meta.date))); + assert!(!issue.contains(&format!("/issues/{}/world", source.meta.date))); + + let behind = app + .oneshot( + Request::builder() + .uri(format!("/issues/{}/behind", source.meta.date)) + .header(header::COOKIE, cookie) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(behind.status(), StatusCode::OK); + assert!(response_text(behind).await.contains("Legacy near miss")); } #[tokio::test] diff --git a/src/web/public.rs b/src/web/public.rs index 8dbb58a..237a70a 100644 --- a/src/web/public.rs +++ b/src/web/public.rs @@ -38,6 +38,8 @@ pub struct PublicEntry { pub domain: String, pub reading_minutes: i64, pub word_count: i64, + pub summary: Option, + pub why: Option, pub comment_links: Vec, pub is_lead: bool, } @@ -102,6 +104,20 @@ impl From<&Issue> for PublicIssue { domain: domain(&article.canonical_url), reading_minutes: article.reading_minutes(), word_count: article.word_count, + summary: pick + .summary + .as_deref() + .or_else(|| { + issue + .editorial + .summaries + .get(&article.id) + .map(String::as_str) + }) + .map(str::trim) + .filter(|summary| !summary.is_empty()) + .map(str::to_string), + why: pick.why.clone(), comment_links, is_lead: pick.is_lead, } @@ -356,16 +372,18 @@ mod tests { use super::*; #[test] - fn public_issue_carries_no_generated_text() { + fn public_issue_shows_summaries_and_why_but_no_bodies() { let source = crate::epub::fixtures::issue(); let public = PublicIssue::from(&source); let html = FeedEntryTemplate { issue: &public }.render().unwrap(); assert!(html.contains("The Lead Story")); assert!(html.contains("Hacker News")); + assert!(html.contains("What it argues, and why it is worth the time.")); + assert!(html.contains("A short abstract for the second piece.")); + assert!(html.contains("The systems story with enough operational detail to matter")); + assert!(html.contains("A small-scene delight outside the usual technical orbit")); for private in [ "Two stories today", - "What it argues", - "systems story", "Body of", "write path", "Agreed", diff --git a/src/web/templates/feed_entry.html b/src/web/templates/feed_entry.html index fdd4aa5..7472231 100644 --- a/src/web/templates/feed_entry.html +++ b/src/web/templates/feed_entry.html @@ -1 +1 @@ -{% for section in issue.sections %}

{{ section.name }}

{% endfor %} +{% for section in issue.sections %}

{{ section.name }}

    {% for entry in section.entries %}
  • {{ entry.title }} — {{ entry.source }} ({{ entry.domain }}){% match entry.summary %}{% when Some with (summary) %}

    {{ summary }}

    {% when None %}{% endmatch %}{% match entry.why %}{% when Some with (why) %}

    Why it's here: {{ why }}

    {% when None %}{% endmatch %}{% if !entry.comment_links.is_empty() %} · {% for link in entry.comment_links %}{{ link.label }}{% endfor %}{% endif %}
    • {% endfor %}
{% endfor %} diff --git a/src/web/templates/issue_full.html b/src/web/templates/issue_full.html index 1bd3124..792df86 100644 --- a/src/web/templates/issue_full.html +++ b/src/web/templates/issue_full.html @@ -22,12 +22,12 @@
Bulk model
{{ colophon.bulk_model }}
Editor model
{{ colophon.editor_model }}
Summaries model
{{ colophon.summaries_model }}
-
Entries considered
{{ colophon.entries_fetched }} from {{ colophon.feeds_seen }} feeds
-
Candidates scored
{{ colophon.candidates }}
+
Entries considered
{% match colophon.entries_fetched %}{% when Some with (entries) %}{{ entries }}{% match colophon.feeds_seen %}{% when Some with (feeds) %} from {{ feeds }} feeds{% when None %} from n/a feeds{% endmatch %}{% when None %}n/a{% endmatch %}
+
Candidates scored
{% match colophon.candidates %}{% when Some with (candidates) %}{{ candidates }}{% when None %}n/a{% endmatch %}
Articles selected
{{ colophon.article_count }} across {{ colophon.section_count }} sections
Words
{{ colophon.total_words }} · ~{{ colophon.reading_minutes }} min read
{% for cost in colophon.provider_costs %}
{{ cost.provider }} cost
{{ cost.cost }}
{% endfor %} -
Total token cost
{{ colophon.cost_usd }}
+
Total token cost
{% match colophon.cost_usd %}{% when Some with (cost) %}{{ cost }}{% when None %}n/a{% endmatch %}
Generator
{{ colophon.generator_version }}
{% endblock %} diff --git a/src/web/templates/issue_public.html b/src/web/templates/issue_public.html index aff3194..c8b8b12 100644 --- a/src/web/templates/issue_public.html +++ b/src/web/templates/issue_public.html @@ -3,5 +3,5 @@

{{ issue.display_date }} · No. {{ issue.issue_number }}

{{ issue.stats_line }}

A personal morning paper, assembled daily; the selection is the reader's, the words are the authors'.

{% if !downloads.is_empty() %}

{% for download in downloads %}{{ download.label }} ({{ download.size }}){% endfor %}

{% endif %} -{% for section in issue.sections %}

{{ section.name }}

{% for entry in section.entries %}

{{ entry.title }}

{% match entry.author %}{% when Some with (author) %}{{ author }} · {% when None %}{% endmatch %}{{ entry.source }} ({{ entry.domain }}) · {{ entry.reading_minutes }} min · {{ entry.word_count }} words

{% if !entry.comment_links.is_empty() %}

{% for link in entry.comment_links %}{{ link.label }}{% if !link.meta.is_empty() %}: {{ link.meta }}{% endif %}{% endfor %}

{% endif %}{% endfor %}
{% endfor %} +{% for section in issue.sections %}

{{ section.name }}

{% for entry in section.entries %}

{{ entry.title }}

{% match entry.author %}{% when Some with (author) %}{{ author }} · {% when None %}{% endmatch %}{{ entry.source }} ({{ entry.domain }}) · {{ entry.reading_minutes }} min · {{ entry.word_count }} words

{% match entry.summary %}{% when Some with (summary) %}

{{ summary }}

{% when None %}{% endmatch %}{% match entry.why %}{% when Some with (why) %}

Why it's here: {{ why }}

{% when None %}{% endmatch %}{% if !entry.comment_links.is_empty() %}

{% for link in entry.comment_links %}{{ link.label }}{% if !link.meta.is_empty() %}: {{ link.meta }}{% endif %}{% endfor %}

{% endif %}{% endfor %}
{% endfor %}

Browse the archive

{% endif %}{% endblock %} diff --git a/src/web/templates/world.html b/src/web/templates/world.html index c67282d..6501147 100644 --- a/src/web/templates/world.html +++ b/src/web/templates/world.html @@ -1,5 +1,5 @@ {% extends "layout.html" %}{% block content %}
-

World Briefing

{{ display_date }}

+

World Briefing

{% match display_date %}{% when Some with (date) %}

{{ date }}

{% when None %}{% endmatch %}
{{ body_html|safe }}

← Back to this issue

{% endblock %}