Compare commits

..
12 Commits
Author SHA1 Message Date
thalladaandClaude Fable 5.1 af2a3ceb49 Bisect provider-rejected batches in triage and deep assessment
DeepSeek's content filter rejects a whole request (400 "Content Exists
Risk") when one article trips it, which cost the other articles in the
batch their assessment and retried them every run. New curate/batch.rs
runs both stages through a bisecting runner: a rejected batch is split
until the offending article is isolated, that article is retried once on
the editor provider when it is a different one, and a still-rejected
article is recorded as a provider_rejected assessment row so it is not
retried for assessment_reuse_days. Cache reuse accepts rows from either
configured model. Each stage logs reused/requested/rejected counts, the
curation: line shows rejections when non-zero, explain prints them, and
the llm_assess span reports the deep-set size.

Implemented by a Claude agent from an orchestrator brief; verified
fmt/clippy(-W dead_code)/test green (354 lib tests).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 20:44:28 +00:00
thallada 7a842b61b6 Bump version to 0.2.0 2026-09-02 19:12:21 +00:00
thalladaandClaude Fable 5.1 99d1338890 Provider-agnostic LLM registry, config check, migration runbook
[llm] assigns the bulk and editor roles by name over a [providers.*]
registry (kind = openai | anthropic, per-provider model, effort, daily
ceiling and price table); DeepseekBackend becomes OpenAiCompatibleBackend
(reasoning_effort passthrough), AnthropicBackend builds from the same
ProviderConfig, meters and provider_costs are keyed by provider name.
Gemini 3.8 Flash is declared via Google's OpenAI-compatible endpoint so
switching the editor is one line (or DAILY_EPUB_LLM__EDITOR=gemini for an
A/B dry run). Stale [deepseek]/[anthropic] tables, the top-level
max_daily_usd and the old key env vars fail loudly.

daily-epub config check validates and prints the resolved roles, models,
key presence and paths without opening the database.

docs/runbooks/curation-v2-migration.md walks the server upgrade from v1.

Registry implemented by a Claude agent from an orchestrator brief;
verified fmt/clippy(-W dead_code)/test green.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 18:44:57 +00:00
thalladaandClaude Fable 5.1 5edbeb509f Curation v2: session 2 handoff (steps 4-7 done, briefs preamble updated)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 16:32:18 +00:00
thalladaandClaude Fable 5.1 d403c51edf Curation v2 step 7: cleanup, prune paths, implementation notes
Dead code and stale v1 comments removed (clippy -W dead_code clean, the
three world.rs warnings fixed), the Brief chapter's TOC title renamed from
"From the Editor", features prune now also sweeps article_assessments and
generate runs the sweep once after publishing, the example config is
tested key-for-key against Config::default(), README commands match
--help, and docs/plans/2026-08-15-implementation-notes.md records the
Anthropic and Voyage facts, the new tables, the budget-day rule and the
lock.

Implemented by a Claude agent from docs/plans/curation-v2-briefs/step7.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 16:32:18 +00:00
thalladaandClaude Fable 5.1 d261cd485d Curation v2 step 6: Behind the paper, stats, run report block, lock
Behind-the-paper chapter (behind.xhtml, both editions) built from the run's
StageCounts and candidate_runs near misses; daily-epub stats [--days N];
StageCounts gains knn/feed gates and verdicts_in_prompt, timings split
into summaries + brief, the four-line §15.4 info block logged once per
run and printed by print_report; src/lock.rs flock guard on
<database_path>.lock for generate, profile rebuild, features backfill and
backfill-social; README updated for the new CLI, env vars and costs.

Implemented by a Claude agent from docs/plans/curation-v2-briefs/step6.md;
reviewed against plan §5, §15.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 16:21:06 +00:00
thalladaandClaude Fable 5.1 05a74a0dcf Curation v2 step 5: deep assessment, utility, diversified shortlist
assess.rs replaces score.rs (DEEP_INSTRUCTIONS, representative sample
with [BEGINNING]/[MIDDLE]/[END], facets, cached deep rows with --rescore
bypass), rank.rs adds the utility blend over present signals with gate
ramps and the leader-clustered shortlist (cap 2 → 3 → uncapped, protected
top-N, exploration reserve), editor.rs replaces select.rs with the §13
rendering and utility-ordered fallbacks. ScoredArticle is gone; Candidate
is the only flow type. deep_batch_size replaces score_batch_size.

Started by Codex (cut off by its usage limit mid-verification) and
finished by a Claude agent from docs/plans/curation-v2-briefs/step5.md;
reviewed against plan §12–§13.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 16:04:43 +00:00
thalladaandClaude Fable 5.1 10f4afd091 Curation v2 step 4: triage replaces the gate
DeepSeek triage over every eligible article (plan §10) with cached
assessments in article_assessments, the union admission with quotas and
exploration slots (§11), hygiene moved to admit.rs with the churn rule
reading assessments, prefilter.rs reduced to hygiene and text heuristic,
prefilter_keep removed in favour of curation.ranking.deep_keep, the
scores table dropped (migration 0003), and --rescore on generate.

Implemented by Codex (gpt-5.4, high effort) from
docs/plans/curation-v2-briefs/step4.md; reviewed against plan §10–§11.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 15:25:04 +00:00
thalladaandClaude Fable 5.1 76527e3821 Curation v2: progress handoff and per-step agent briefs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 04:16:59 +00:00
thalladaandClaude Fable 5.1 8b51454c20 Merge curation-v2-step3 into curation-v2 (steps 2 and 3 united)
Union merge of the Claude editor (step 2) and the embeddings/signals/telemetry
work (step 3): both config trees, per-provider LLM budgets plus the Voyage
meter, one startup provider log, runs.config_json with ranking and voyage
settings, and selected picks' why lines landing in candidate_runs.editor_why.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 04:16:12 +00:00
thalladaandClaude Fable 5.1 ea3b141373 Curation v2 step 3: Voyage embeddings, cheap signals, candidate telemetry
- embedding.rs: EmbeddingBackend + VoyageBackend, batched bounded-concurrency
  client with its own UsageMeter, f32 BLOB codec, article/interest embedding
  cache keyed by model, dimension and sha256 of the embedded text.
- signals.rs: z-scored interest match, decayed rated-neighbour preference
  with the knn gate, feed affinity with the feed gate, social, text heuristic
  without social terms, mid-rank percentile normalizer, preliminary blend.
- telemetry.rs: candidate_runs writer with §7.5 signals_json, explain and
  near-misses renderers, prune.
- [voyage] and the full [curation.ranking] config with validation.
- CLI: explain, features backfill|prune, generate --skip-embeddings.
- Pipeline: hygiene rows, embed and signals stages before the old prefilter;
  same-date regeneration no longer excludes its own picks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 04:08:45 +00:00
thalladaandClaude Fable 5.1 57efbb49b4 Curation v2 step 2: Claude Opus 5 editor, the Brief, per-provider budgets
- AnthropicBackend (Messages API, cached system block, output_config.effort,
  server-side fallbacks, refusal surfaced as an error); Llms { bulk, editor }
  with editor_or_bulk(); PriceTable-based UsageMeter per provider.
- [anthropic], [editorial], deepseek.max_concurrent_requests and
  curation.max_article_count config; startup logs resolved providers.
- Budget day is the UTC date of started_at, preloaded from
  runs.provider_costs_json; finish_run writes provider_costs_json and
  config_json. Stage A batches run concurrently with per-batch budget checks.
- Editor prompt with one-line "why" per pick; no minimum lineup size;
  --max-articles is a ceiling; top-up branch deleted; why stored on picks and
  issue_articles.why and rendered in chapters and In this issue.
- Summaries on the editor client (3k-token input, concurrency 4, bulk then
  excerpt fallback); "The Brief" replaces From the Editor; section intros gone.
- Colophon carries per-provider costs and models.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
2026-09-02 03:56:14 +00:00
68 changed files with 17629 additions and 3258 deletions

No files matched your search

Generated
+2 -1
View File
@@ -815,7 +815,7 @@ dependencies = [
[[package]]
name = "daily-epub"
version = "0.1.0"
version = "0.2.0"
dependencies = [
"ammonia",
"anyhow",
@@ -831,6 +831,7 @@ dependencies = [
"hmac",
"image",
"jiff",
"libc",
"rand 0.10.2",
"reqwest",
"resvg",
+11 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "daily-epub"
version = "0.1.0"
version = "0.2.0"
edition = "2024"
[dependencies]
@@ -18,8 +18,17 @@ hex = "0.4.3"
hmac = "0.13.0"
image = "0.25.10"
jiff = { version = "0.2.35", features = ["serde"] }
libc = "0.2.189"
rand = "0.10.2"
reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "gzip", "json", "charset", "http2", "query", "system-proxy"] }
reqwest = { version = "0.13.4", default-features = false, features = [
"rustls",
"gzip",
"json",
"charset",
"http2",
"query",
"system-proxy",
] }
resvg = "0.48.1"
scraper = "0.27.0"
serde = { version = "1.0.229", features = ["derive"] }
+257 -47
View File
@@ -4,16 +4,29 @@ A personalized daily newspaper, delivered as an EPUB.
Every morning a systemd timer wakes one Rust binary. It pulls the last ~26 hours
from a self-hosted [Miniflux](https://miniflux.app), deduplicates and extracts
the articles, enriches them with HackerNews/Lobsters/Reddit social proof, filters
300–500 candidates down to ~120 with cheap heuristics, and asks DeepSeek to score,
select and introduce 15–25 of them. It assembles two EPUB editions (a standard one
the articles, enriches them with HackerNews/Lobsters/Reddit social proof, has
DeepSeek triage every eligible opening and closely assess a 120-article union,
then utility-ranks and diversity-caps a 60-item shortlist for Claude Opus 5 — the editor —
which assembles the issue
(no minimum size, a hard ceiling), writes a one-line *why* under every headline,
the summaries and *The Brief*. It assembles two EPUB editions (a standard one
and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and
publishes the lot over its own OPDS catalog — which doubles as a
[BookOrbit](https://github.com/thallada/bookorbit) watched folder if you run one.
Each article chapter ends with Loved it / Good / Not for me links that feed back into tomorrow's curation.
Each article chapter ends with Loved it / Good / Not for me links that feed back
into tomorrow's curation, and a short *Behind the paper* chapter before the
colophon says what the run considered, how the deep set was admitted, whether
the learned signals were active, the ten highest-utility near misses, and what
it all cost.
Steady-state cost is roughly **$0.05–0.30/day** in DeepSeek tokens, hard-capped by
`max_daily_usd`.
Steady-state cost is roughly **$1/day**: $0.05–0.30 in DeepSeek tokens plus
~$0.50–0.80 for the Claude editor and a few cents of Voyage AI embeddings, each
provider with its own per-UTC-day ceiling (`providers.<name>.max_daily_usd` and
`voyage.max_daily_usd`). Those ceilings are runaway guards, not accounting — set
hard spend limits in the providers' dashboards as the real backstop. The two
LLM roles — *bulk* (triage, assessment, fallbacks) and *editor* — are assigned
by name to entries of a provider registry, so swapping DeepSeek or Claude for
Gemini (or anything OpenAI-compatible) is a config line plus an API key.
- Full design: [`docs/plans/2026-08-15-the-daily-epub.md`](docs/plans/2026-08-15-the-daily-epub.md)
- Implementation decisions: [`docs/plans/2026-08-15-implementation-notes.md`](docs/plans/2026-08-15-implementation-notes.md)
@@ -24,20 +37,48 @@ Steady-state cost is roughly **$0.05–0.30/day** in DeepSeek tokens, hard-cappe
```
Miniflux ingest ─▶ dedupe ─▶ extraction ─▶ persist ─▶ social enrichment
─▶ pre-filter ─▶ LLM scoring ─▶ selection ─▶ comments ─▶ editorial
─▶ hygiene ─▶ embeddings (Voyage) + cheap signals ─▶ triage (bulk LLM)
─▶ union admission ─▶ deep assessment (bulk LLM) ─▶ utility + diversity
─▶ editor (editor LLM) ─▶ comments ─▶ editorial (editor LLM)
─▶ world briefing ─▶ EPUB (standard + X4) ─▶ XTC ─▶ publish ─▶ report
```
Every stage writes to SQLite, so a run is idempotent per date: re-running
`generate --date 2026-08-15` replaces that issue rather than duplicating it.
Only one writer runs at a time: `generate`, `profile rebuild`, `features
backfill` and `backfill-social` take an advisory `flock` on
`<database_path>.lock`, and a second invocation exits with `generate is already
running` (naming whichever command holds it). `serve`, `explain`, `stats`,
`ratings`, `features prune` and `db migrate` never wait on it, and the kernel
releases the lock when the holder exits, however it exits.
Every run ends with a four-line summary in the log and on stdout:
```
curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected
admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0
preference: 14 rated w/ embeddings → knn 0.35 · feed off · 41 verdicts in prompt
providers: anthropic $0.62 · deepseek $0.11 · voyage $0.02 · total $0.75 · 23m12s
```
The full report (per-stage counts and timings — `embed`, `signals`, `triage`,
`admit`, `assess`, `rank`, `editor`, `summaries`, `brief` among them — and
per-provider usage) is stored on the `runs` row and in `issues.report_json`.
**Failure policy.** Miniflux ingest, SQLite writes, EPUB assembly and publishing
are fatal — without them there is no issue, and the `runs` row records why.
Social lookups, comment fetching, the world briefing, images and the XTC
conversion are best-effort: they log, add a warning (run status `degraded`) and
the run continues. Every DeepSeek stage *degrades*: a missing key, a dead API or
a tripped budget turns the run into the `--skip-llm` shape (prefilter order
selects, feed excerpts stand in for summaries) instead of losing the day's issue.
the run continues. Every LLM stage *degrades*: an editor call that fails, is
refused, or is over its provider's daily ceiling is retried with the same
prompt on the bulk provider; if the bulk provider is missing, dead or over
budget too, the run takes the `--skip-llm` shape (admission uses cheap signals
and feed excerpts stand in for summaries) instead of losing the day's issue.
Which provider plays which role is the `[llm]` table (`bulk = "deepseek"`,
`editor = "anthropic"` by default); a role whose key is absent is simply
unavailable, and `daily-epub config check` shows the resolved assignment before
a run. Anthropic's server-side refusal fallback (`fallbacks = "default"`) is
enabled on every request to an `anthropic`-kind provider.
---
@@ -47,7 +88,9 @@ selects, feed excerpts stand in for summaries) instead of losing the day's issue
|---|---|---|
| Rust (2024 edition toolchain) | building | `cargo build --release` |
| **Miniflux** with an API key | the only content source | Settings → API Keys. The client is read-only and never mutates read state. |
| **DeepSeek API key** | curation + editorial | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. |
| A key for the **bulk** provider (DeepSeek by default) | triage and deep assessment, and the fallback for every editor call | <https://platform.deepseek.com>. `DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY`. Optional: `--skip-llm` runs the whole pipeline without it. |
| A key for the **editor** provider (Anthropic by default) | the editor: selection, summaries, The Brief, the weekly profile rebuild | <https://console.anthropic.com>. `DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY`. Optional: without it every editor call runs on the bulk provider. Set a dashboard spend limit; `providers.anthropic.max_daily_usd` is only a runaway guard. Any other `[providers.*]` entry (Gemini is shipped) can take either role — see *Switching providers*. |
| **Voyage AI API key** | article and interest embeddings behind the learned ranking signals | <https://www.voyageai.com>. Optional: without it (or with `--skip-embeddings`) the run uses cached vectors only and the learned signals are absent, never a penalty. |
| A 32+ byte random secret | signs the article rating links | `openssl rand -hex 32` |
| **BookOrbit** library + watched folder | *optional* — a richer library UI on top of the same folder | Delivery does not need it: `daily-epub serve` has its own OPDS catalog over `publish.epub_dir`. If you do run it, create a dedicated "The Daily EPUB" library, enable *Watch folders*, and point `publish.epub_dir` at it. |
| **Node.js 18+** and a clone of [`epub-to-xtc-converter`](https://github.com/bigbag/epub-to-xtc-converter) | XTC/XTCH output for the Xteink X4 | Optional (`xtc.enabled = false` turns it off). Needs `npm install` **inside `cli/`**, and a settings JSON naming a real TTF/OTF — see below. It has **no global npm bin** — it is invoked as `node <repo>/cli/index.js convert …`, which is why `xtc.command`/`xtc.args` are fully general. |
@@ -70,14 +113,20 @@ sudo install -m0755 target/release/daily-epub /usr/local/bin/
### Commands
```
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm]
daily-epub generate [--date YYYY-MM-DD] [--dry-run] [--out DIR] [--max-articles N] [--skip-llm] [--skip-embeddings] [--rescore]
daily-epub serve # rating endpoints + OPDS catalog + downloads
daily-epub profile rebuild # regenerate learned profile adjustments
daily-epub ratings list --days 90
daily-epub ratings list [--days 90] [--label loved|good|down|cleared]
daily-epub ratings set --article 42 --label loved --note "excellent"
daily-epub ratings clear --url https://example.com/article
daily-epub backfill-social # re-poll social scores for recent articles
daily-epub explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]
daily-epub explain --date YYYY-MM-DD --near-misses [N]
daily-epub stats [--days 14] # the evaluation framework, one fact per line
daily-epub features backfill [--days 30] [--rated-only] [--all] [--yes]
daily-epub features prune # stale embeddings, old telemetry and assessments
daily-epub backfill-social [--days 7] # re-poll social scores for recent articles
daily-epub db migrate # run migrations (also automatic on every start)
daily-epub config check # validate the config, print the resolved roles, keys and paths
```
`--dry-run` does everything except deliver: it still ingests, persists entries and
@@ -85,6 +134,49 @@ articles, curates and **builds both EPUBs into `--out`**, but it does not copy t
BookOrbit, does not run the retention sweep, does not write the `issues` row and
does not advance the ingest watermark. It prints the lineup and the cost report.
`--skip-embeddings` reads the embedding cache but makes zero Voyage calls.
`--rescore` ignores reusable triage/deep assessments for this run.
`--max-articles N` is a ceiling, never a target: the hard maximum becomes
`min(curation.max_article_count, N)` and the soft target is lowered to fit.
`explain` answers "why was this (not) in the paper" from the `candidate_runs`
row the run persisted for every considered article: the stage it reached and the
reason it stopped, every raw and normalized signal with its presence and
effective weight, the top interests, the nearest rated neighbours, the triage
and deep assessments (quality, fit, category, rationale, facets), utility and
rank, the cluster it landed in and what suppressed it, and the editor's reason
for a pick. `--url` canonicalizes the address; an article that is not in the
database at all is reported as never ingested (a feed problem, not a ranking
one). `--near-misses` lists the highest-utility articles that were not selected
(by preliminary blend for articles the ranker never reached).
`stats` is the whole evaluation framework, on purpose: for the last `--days`
(14) it prints the issues and articles published, the mean issue size, explicit
ratings by label and per issue, the up/down ratio of rated picks per admitting
retriever (`admitted_by[0]` — triage, interest, knn, exploration, blend,
auto_include), the exploration yield (admitted, selected, rated positively),
cost per day per provider from `runs.provider_costs_json`, and the mean
generation time. Plain text, one fact per line. Tune from it and from reading
the paper; anything more waits for more ratings.
`features backfill` embeds the rated and published articles first (the learned
set), then the standing interests, then — only with `--all` — every other
article first seen in the window. It prints an estimate and asks before spending
more than 5M tokens unless `--yes`; a warm cache makes zero calls. `features
prune` drops embeddings of articles neither rated nor published that are older
than `curation.ranking.embedding_retention_days`, and `candidate_runs` rows and
`article_assessments` older than `curation.ranking.telemetry_retention_days`.
`generate` runs the same sweep once after publishing, best effort.
`config check` loads and validates the configuration exactly as `generate`
would and prints one fact per line: the config path, the database, profile,
interests and publish paths with `exists`/`MISSING`, each `[llm]` role with its
provider name, kind, model, effort, ceiling and whether its key is present, the
Voyage line likewise, and `editorial.summary_model`. Lines that need attention
start with `!`. It exits non-zero only on a validation error — a missing key or
file is a warning, since the run degrades rather than fails — and never opens
the database or takes the lock, so it is safe to run next to a live `generate`.
---
## Configuration
@@ -96,12 +188,43 @@ Start from [`config.example.toml`](config.example.toml). Load order, later wins:
3. `DAILY_EPUB_*` environment variables
Nested keys use a **double underscore**: `[miniflux] api_key` becomes
`DAILY_EPUB_MINIFLUX__API_KEY`. Top-level keys are just uppercased:
`DAILY_EPUB_LOOKBACK_HOURS=30`. As a convenience, plain **`DAILY_EPUB_SECRET`**
is accepted as an alias for `server.hmac_secret` (the explicit key wins if both
are set).
`DAILY_EPUB_MINIFLUX__API_KEY`, and a provider's key is
`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY` with the `[providers.<name>]` table name
upper-cased — the shipped registry reads `DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY`,
`DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY` and
`DAILY_EPUB_PROVIDERS__GEMINI__API_KEY` (provider names are therefore lowercase
`a-z0-9_`). Top-level keys are just uppercased: `DAILY_EPUB_LOOKBACK_HOURS=30`.
As a convenience, plain **`DAILY_EPUB_SECRET`** is accepted as an alias for
`server.hmac_secret` (the explicit key wins if both are set).
Secrets belong in the environment file, never in the TOML.
Secrets belong in the environment file, never in the TOML. Stale configuration
fails at startup rather than silently curating without a provider: a
`[deepseek]` or `[anthropic]` table, a top-level `max_daily_usd`, a batch-size
or temperature key outside `[llm]`, or a `DAILY_EPUB_DEEPSEEK__*` /
`DAILY_EPUB_ANTHROPIC__*` environment variable is an error naming the new key.
### Switching providers
The roles are names, the providers are tables. To run the editor on Gemini:
```toml
[llm]
editor = "gemini" # [providers.gemini] is already declared in config.example.toml
```
and put `DAILY_EPUB_PROVIDERS__GEMINI__API_KEY=…` in the env file. A one-off
A/B without touching the file, since every key is also an env var:
```sh
DAILY_EPUB_LLM__EDITOR=gemini daily-epub generate --dry-run --date 2026-09-02
```
The same works for `bulk`. Both roles may name one provider (they then share
one client and one `max_daily_usd`), `editor = ""` runs everything on bulk, and
a new endpoint is a new `[providers.<name>]` table: `kind = "openai"` for any
OpenAI-compatible chat-completions API (DeepSeek, Gemini, OpenAI, a local
server), `kind = "anthropic"` for the Messages API. `daily-epub config check`
prints what resolved.
### Reference
@@ -109,11 +232,9 @@ Secrets belong in the environment file, never in the TOML.
|---|---|---|
| `timezone` | `America/New_York` | Day boundaries and `--date` interpretation. |
| `lookback_hours` | `26` | Size of the ingest window ending at the issue day's end (clamped to now). |
| `target_article_count` | `20` | Lineup size the selector aims for. `--max-articles` overrides it. |
| `prefilter_keep` | `120` | Candidates surviving the heuristic pre-filter. Must be ≥ `target_article_count`. |
| `target_article_count` | `20` | Soft target the editor aims for. There is no minimum: a nine-pick issue is published as nine. |
| `retention_days` | `21` | EPUBs older than this are deleted from `publish.epub_dir`. SQLite history is kept forever. |
| `xtc_retention_count` | `5` | How many XTC issues to keep in `publish.xtc_dir`. Counted, not dated: each `.xtch` is ~80–100 MB, so the binding constraint is disk, not age. |
| `max_daily_usd` | `2.0` | Hard ceiling on DeepSeek spend **per day**, not per run — a re-run inherits what earlier runs for that date already spent. Tripping it skips remaining LLM work and degrades to excerpts. |
| `world_briefing` | `true` | Include the Wikipedia Current Events section. |
| `database_path` | `/var/lib/daily-epub/daily-epub.db` | SQLite file; parent dirs are created. |
| `out_dir` | `/var/lib/daily-epub/out` | Where `generate` writes artifacts before publishing. |
@@ -122,15 +243,33 @@ Secrets belong in the environment file, never in the TOML.
| `miniflux.base_url` | `http://127.0.0.1:8082` | Miniflux root (no `/v1`). |
| `miniflux.api_key` | — | **`DAILY_EPUB_MINIFLUX__API_KEY`**. Required. |
| `miniflux.page_limit` | `250` | Entries per page; Miniflux caps this at 250. |
| `deepseek.base_url` | `https://api.deepseek.com/v1` | OpenAI-compatible endpoint. |
| `deepseek.model` | `deepseek-v4-flash` | Verified 2026-08-15 (DeepSeek-V4-Flash-0731). |
| `deepseek.api_key` | — | **`DAILY_EPUB_DEEPSEEK__API_KEY`**. Absent ⇒ the run curates heuristically. |
| `deepseek.score_batch_size` | `12` | Articles per stage-A scoring request. |
| `deepseek.score_temperature` | `0.3` | Scoring/selection temperature. |
| `deepseek.editorial_temperature` | `0.8` | Summaries, intros, front page. |
| `deepseek.price_input_per_mtok` | `0.14` | USD per 1M cache-miss input tokens (cost guardrail arithmetic). |
| `deepseek.price_cached_input_per_mtok` | `0.0028` | USD per 1M prefix-cache-hit input tokens. |
| `deepseek.price_output_per_mtok` | `0.28` | USD per 1M output tokens. |
| `llm.bulk` | `deepseek` | The `[providers.*]` name that runs triage, deep assessment and every fallback. `""` ⇒ no bulk provider (those stages are skipped). |
| `llm.editor` | `anthropic` | The provider that assembles the lineup, writes the summaries and The Brief and rebuilds the profile. `""` or absent ⇒ everything runs on `bulk`. Naming the same provider as `bulk` shares one client and one ceiling. |
| `llm.triage_batch_size` | `25` | Articles per first-pass triage request. |
| `llm.deep_batch_size` | `8` | Articles per close-reading assessment request. The removed `score_batch_size` key is a startup error. |
| `llm.score_temperature` | `0.3` | Scoring temperature, sent only to `openai`-kind providers. |
| `llm.editorial_temperature` | `0.8` | Summaries and The Brief on an `openai`-kind provider. |
| `providers.<name>.kind` | — | `openai` (chat completions at `{base_url}/chat/completions`, bearer key) or `anthropic` (Messages API: `output_config.effort`, a cached system block, `fallbacks = "default"` with the `server-side-fallback-2026-07-01` beta). Shipped entries: `deepseek`, `anthropic`, `gemini`. |
| `providers.<name>.base_url` | — | Endpoint root. DeepSeek `https://api.deepseek.com/v1`; Anthropic `https://api.anthropic.com`; Gemini `https://generativelanguage.googleapis.com/v1beta/openai`. |
| `providers.<name>.model` | — | `deepseek-v4-flash` (verified 2026-08-15), `claude-opus-5`, `gemini-3.8-flash` (verified 2026-09-02). |
| `providers.<name>.api_key` | — | **`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY`**, environment only. Absent ⇒ that role is unavailable and degrades (bulk ⇒ heuristic curation, editor ⇒ bulk). |
| `providers.<name>.effort` | `high` (anthropic, gemini) | `anthropic`: `low`, `medium`, `high`, `xhigh` or `max` → `output_config.effort`. `openai`: passed through as `reasoning_effort` (Gemini takes `minimal`–`high`); omit it for models without one (DeepSeek). |
| `providers.<name>.max_daily_usd` | `2.0` / `3.0` / `3.0` | That provider's ceiling per **UTC day** of the run's start, not per run — a re-run inherits what earlier runs that day already spent on it (`runs.provider_costs_json`). Tripping it skips that provider's remaining calls; in-flight requests finish and the paper still publishes. `0` disables the guard. |
| `providers.<name>.max_concurrent_requests` | `4` | Triage and deep-assessment batches in flight on the bulk provider; summaries in flight on the summary provider. |
| `providers.<name>.price_input_per_mtok` | `0.14` / `5.0` / `0.75` | USD per 1M cache-miss input tokens (cost guardrail arithmetic). |
| `providers.<name>.price_cache_read_per_mtok` | `0.0028` / `0.5` / `0.075` | USD per 1M cache-hit input tokens. |
| `providers.<name>.price_cache_write_per_mtok` | `0.0` / `6.25` / `0.0` | USD per 1M tokens written to the prompt cache (implicit caches charge nothing). |
| `providers.<name>.price_output_per_mtok` | `0.28` / `25.0` / `3.75` | USD per 1M output tokens, thinking tokens included where the provider bills them as output. |
| `voyage.enabled` | `true` | Embed articles and interests with Voyage AI. `false` ⇒ cached vectors only. |
| `voyage.base_url` | `https://api.voyageai.com/v1` | `POST {base_url}/embeddings`. |
| `voyage.model` | `voyage-4-lite` | Embedding model; changing it invalidates the cache. |
| `voyage.api_key` | — | **`DAILY_EPUB_VOYAGE__API_KEY`**. Absent ⇒ cached vectors only. |
| `voyage.output_dimension` | `512` | One of 256, 512, 1024, 2048. |
| `voyage.batch_size` | `32` | Texts per request. |
| `voyage.max_concurrent_requests` | `4` | Requests in flight. |
| `voyage.max_input_chars` | `60000` | Per-article cut, on a char boundary. |
| `voyage.max_daily_usd` | `0.50` | Runaway guard at $0.02/M tokens. |
| `curation.max_article_count` | `28` | Hard ceiling on issue size. `--max-articles N` lowers it to `min(28, N)` and drags the soft target down with it. Must be ≥ `target_article_count`. |
| `curation.always_include_feeds` | `[]` | Miniflux feed ids or URL substrings that can never be dropped. |
| `curation.blocked_domains` | `[]` | Hosts excluded outright. |
| `curation.paywall_domains` | `[]` | Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, economist, …). |
@@ -139,6 +278,11 @@ Secrets belong in the environment file, never in the TOML.
| `curation.feedback.good_value` | `0.35` | Weight for a Good verdict. |
| `curation.feedback.not_for_me_value` | `-1.0` | Weight for a Not for me verdict. |
| `curation.feedback.verdicts_in_prompt` | `60` | Recent explicit verdicts included in the system prompt. |
| `curation.recent_rejection_days` | `7` | Churn window for recent low triage/deep assessments. |
| `curation.recent_rejection_floor` | `3.0` | Scores below this floor are excluded during the churn window (except auto-includes). |
| `curation.ranking.*` | see below | Every weight, quota, gate and threshold of the personalized ranker. |
| `editorial.summary_model` | `editor` | Which `[llm]` role writes the per-article summaries: `editor` (with per-article bulk fallback) or `bulk`. |
| `editorial.summary_input_tokens` | `3000` | Article text offered to the summary prompt. |
| `publish.epub_dir` | `/srv/bookorbit/libraries/daily-epub` | Both EPUB editions land here by atomic copy, and this is the directory the OPDS feed lists. The editions are distinguished by a `(X4)` tag in **both** the filename and `dc:title` — libraries and OPDS clients list books by title, so the filename alone would make them look identical. Point a BookOrbit watched folder at it if you want its UI too. **Renamed from `bookorbit_dir`**; the old key is a hard config error. |
| `publish.xtc_dir` | `/var/lib/daily-epub/xtc` | XTC artifacts. **Not** listed in the OPDS feed — CrossPoint cannot acquire them — but downloadable at `/files/xtc/<name>` for sideloading. |
| `xtc.enabled` | `true` | Set `false` to skip the converter entirely. |
@@ -151,6 +295,29 @@ Secrets belong in the environment file, never in the TOML.
| `server.hmac_secret` | — | **`DAILY_EPUB_SERVER__HMAC_SECRET`** (or `DAILY_EPUB_SECRET`). Without it, generated links are rejected with 403. |
| `server.basic_auth_user` / `_pass` | unset | Optional Basic auth for `/opds/*` and `/files/*`. |
`[curation.ranking]` holds the ranker's tunables. The learned signals are
gated: `knn` (rated-neighbour preference) ramps from `knn_floor` (8) to
`knn_full` (25) rated articles with embeddings, `feed` (feed affinity) from
`feed_floor` (15) to `feed_full` (40) attributable ratings; below the floor the
signal is absent. Ratings decay with `rating_half_life_days` (60) over
`rating_lookback_days` (180); `neighbour_k` (5) neighbours per side and
`negative_coefficient` (0.75) shape the signal. `triage_max` (800),
`deep_keep` (120), `shortlist_keep` (60), `assessment_reuse_days` (3),
`semantic_min_words` (300), `exploration_slots` (5), `[curation.ranking.quotas]`
(`triage` 60 · `interest` 20 · `knn` 20), `[curation.ranking.weights.utility]`
(`quality` 0.40 · `fit` 0.20 · `knn` 0.15 · `interest` 0.10 · `feed` 0.05 ·
`triage` 0.05 · `social` 0.03 · `heuristic` 0.02, over the signals present for
each article of the deep set) and `[curation.ranking.diversity]`
(`cluster_threshold` 0.85, `per_cluster_cap` 2, `utility_protected` 10) drive
the LLM triage, deep assessment, utility ranking and diversification stages.
`[curation.ranking.weights.preliminary]` (`interest` 0.35 · `knn` 0.25 ·
`heuristic` 0.20 · `feed` 0.10 · `social` 0.10) blends the cheap signals; weights
are renormalized over the signals present for each article, so they need not sum
to 1. `embedding_retention_days` (120) and `telemetry_retention_days` (180) are
what `features prune` enforces. Validation: weights non-negative; `deep_keep ≥
shortlist_keep ≥ target_article_count`; `*_full > *_floor`; `0 ≤
cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; Voyage batch size and concurrency ≥ 1.
---
## Deployment (systemd)
@@ -173,10 +340,20 @@ sudo install -m0640 -o daily-epub -g daily-epub config.example.toml /etc/daily-e
sudo -e /etc/daily-epub/config.toml # set publish dirs, xtc args, public_url
sudo tee /etc/daily-epub/env >/dev/null <<EOF
DAILY_EPUB_MINIFLUX__API_KEY=…
DAILY_EPUB_DEEPSEEK__API_KEY=…
DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY=…
DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY=…
# DAILY_EPUB_PROVIDERS__GEMINI__API_KEY=… # only if a role names "gemini"
DAILY_EPUB_VOYAGE__API_KEY=…
DAILY_EPUB_SERVER__HMAC_SECRET=$(openssl rand -hex 32)
EOF
sudo chown daily-epub:daily-epub /etc/daily-epub/env && sudo chmod 0600 /etc/daily-epub/env
# One key per [providers.<name>] entry a role uses, named after the table
# (upper-cased). Any may be left unset: that role is then unavailable and the
# run degrades (editor → bulk, bulk → heuristic curation, Voyage → cached
# vectors). The pre-registry DAILY_EPUB_DEEPSEEK__API_KEY /
# DAILY_EPUB_ANTHROPIC__API_KEY names are a startup error, not a silent no-op.
sudo -u daily-epub bash -c 'set -a; . /etc/daily-epub/env; set +a;
daily-epub --config /etc/daily-epub/config.toml config check' # roles, keys present?, paths
# publish dirs must exist and be writable by the service user
sudo install -d -o daily-epub -g daily-epub /var/lib/daily-epub/xtc
@@ -335,6 +512,7 @@ Condensed from spec §5. Run it in this order the first time.
```sh
# 1. Config and schema
daily-epub --config /etc/daily-epub/config.toml config check # roles → providers, keys present?
daily-epub --config /etc/daily-epub/config.toml db migrate
# 2. Ingest only, no keys spent: does Miniflux answer, and with how much?
@@ -344,12 +522,13 @@ DAILY_EPUB_OUT_DIR=./out daily-epub generate --dry-run --skip-llm --max-articles
# 3. Inspect the artifacts
ls -la ./out # two .epub files
epubcheck "./out/The Daily EPUB - $(date +%F).epub" # expect zero errors
# open the standard edition in Calibre / KOReader: cover, From the Editor,
# In This Issue, sections, discussions, colophon; TOC depth 2
# open the standard edition in Calibre / KOReader: cover, The Brief,
# In This Issue, sections, discussions, Behind the paper, colophon; TOC depth 2
# 4. Now with DeepSeek, still not publishing
# 4. Now with the bulk and editor providers, still not publishing
daily-epub generate --dry-run --out ./out --max-articles 6
# → check the lineup is sane and the printed cost is well under $0.50
# → check the lineup is sane (at most 6 picks, each with a "why" line) and the
# printed per-provider cost is well under $1
# 5. Full live run
sudo systemctl start daily-epub-generate
@@ -368,12 +547,13 @@ curl -s https://daily.hallada.net/issues.json | jq '.[0]'
sqlite3 /var/lib/daily-epub/daily-epub.db 'select * from rating_events order by event_at desc;'
# 8. Watch cost and quality for a week
sqlite3 /var/lib/daily-epub/daily-epub.db \
'select date, status, entries_fetched, candidates, selected, cost_usd from runs order by id desc limit 7;'
daily-epub stats --days 7
daily-epub explain --date $(date +%F) --near-misses
```
Tune `prefilter_keep`, `target_article_count` and `curation.always_include_feeds`
from what you see in step 8.
Tune `curation.ranking.deep_keep`, `target_article_count` and `curation.always_include_feeds`
from what you see in step 8, and read the paper's *Behind the paper* chapter
each morning: it is the same numbers, on the device.
### Troubleshooting
@@ -402,9 +582,12 @@ cargo fmt
The crate is a library plus a thin binary, so tests drive the pipeline directly.
`tests/e2e_pipeline.rs` is the capstone: synthetic entries → dedupe → offline
extraction → prefilter → selection (both the `--skip-llm` route and a
extraction → signals → triage → admission → selection (both the `--skip-llm` route and a
`MockBackend` DeepSeek route) → editorial → both EPUB editions → publish → OPDS
and database rows, with no network access anywhere.
and database rows, with no network access anywhere. `tests/m4_epub.rs` covers
the rendered chapters, including *Behind the paper* in both editions; the
`stats` and lock tests live next to their modules (`curate/telemetry.rs`,
`lock.rs`).
### Layout
@@ -413,8 +596,9 @@ and database rows, with no network access anywhere.
server. The stages themselves:
```text
miniflux.rs ingest curate/ scoring and selection
dedupe.rs clustering prefilter, llm, score, select, editorial
miniflux.rs ingest curate/ triage, assessment and selection
dedupe.rs clustering prefilter, llm, triage, admit, assess, rank,
editor, editorial, embedding, signals, telemetry
extract.rs body text profile/ the reader's taste profile
images/ article images comments.rs discussion chapters
normalize usable <img> world.rs the world briefing
@@ -423,6 +607,7 @@ images/ article images comments.rs discussion chapters
encode re-encode publish.rs BookOrbit + XTC
embed into the page server.rs ratings, OPDS
html.rs markup helpers db.rs SQLite
lock.rs one writer at a time (flock on <database_path>.lock)
```
Two modules are worth knowing about before you go looking for their contents.
@@ -485,8 +670,33 @@ From spec §7, plus what implementation turned up:
stack, so `curate/llm.rs` speaks the OpenAI-compatible wire protocol over the
shared `reqwest` client instead, behind a `ChatBackend` trait. The dependency
was removed.
- **Only DeepSeek is wired.** Another provider means another `ChatBackend` impl.
- **No embedding-based personal ranker yet** (spec §3.9 future work); the schema
is ready for it once ~200 ratings exist.
- **Two wire protocols are implemented**, `openai` (chat completions) and
`anthropic` (Messages API), each a `ChatBackend` impl. Providers are config
entries over those two kinds, each with its own `UsageMeter`, price table and
ceiling; a protocol that is neither means another impl. Voyage AI embeddings
sit behind the analogous `EmbeddingBackend` trait in `curate/embedding.rs`.
- **Triage and union admission replace the heuristic gate.** Every eligible
article gets interest, rated-neighbour, feed-affinity, social and heuristic
signals, then DeepSeek reads its opening (up to `triage_max`). The deep set is
the union of triage, interest, neighbour, exploration, blend and auto-include
retrievers. `explain` shows the assessment and `admitted_by`. Learned signals
stay absent until their gates open (8 and 15 ratings respectively).
- **Deep assessment and diversity are live.** DeepSeek reads a representative
beginning/middle/end sample, separates editorial quality from reader fit, and
records descriptive facets. Utility is normalized over the deep set; embedding
leader clusters cap near-duplicates before the 60-item editor shortlist.
- **DeepSeek's content filter rejects whole batches.** A `400 Content Exists
Risk` refuses the entire request when any one article in it trips the input
filter, without saying which. Triage and deep assessment therefore bisect a
rejected batch (halves, then quarters, down to single articles) so the other
articles keep their assessment. A single article the bulk provider still
refuses is retried once on the editor provider with the same prompt when that
is a different provider; if that also fails, an `article_assessments` row with
`kind = 'provider_rejected'` and a NULL score is written so the article is not
sent again for `assessment_reuse_days`, and it is ranked on its other signals.
See them with `explain` (`triage: rejected by provider — deepseek: …`), the
`triage:` / `assess:` log lines (`… 3 rejected (2 recovered on gemini)`), the
` · N rejected` suffix on the `curation:` line, or
`sqlite3 /var/lib/daily-epub/daily-epub.db "select stage, count(*) from article_assessments where kind = 'provider_rejected' group by stage"`.
- **One reader, one issue per day.** There is no multi-user support and no
weekly/retrospective edition (spec §6).
+131 -10
View File
@@ -3,17 +3,19 @@
# Load order (later wins): built-in defaults ← this file ← `DAILY_EPUB_*` env vars.
# Nested keys use a double underscore in env vars, e.g.
# DAILY_EPUB_MINIFLUX__API_KEY=...
# DAILY_EPUB_DEEPSEEK__API_KEY=...
# DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY=...
# DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY=...
# DAILY_EPUB_PROVIDERS__GEMINI__API_KEY=...
# DAILY_EPUB_VOYAGE__API_KEY=...
# DAILY_EPUB_SERVER__HMAC_SECRET=...
# DAILY_EPUB_LOOKBACK_HOURS=30
# DAILY_EPUB_LLM__EDITOR=gemini # one-off role override, no file edit
timezone = "America/New_York"
lookback_hours = 26
target_article_count = 20
prefilter_keep = 120
retention_days = 21 # EPUBs, by age
xtc_retention_count = 5 # XTC issues, by count (~80-100 MB each)
max_daily_usd = 2.0
world_briefing = true
# SQLite database file. Parent directories are created on demand.
@@ -31,19 +33,85 @@ base_url = "http://127.0.0.1:8082"
# api_key via DAILY_EPUB_MINIFLUX__API_KEY env
page_limit = 250
[deepseek]
# The two LLM roles, each assigned to a provider declared in [providers.*]
# below. Switching the editor to Gemini is `editor = "gemini"` plus its key in
# the env file; nothing else changes. An empty editor ("") runs everything on
# the bulk provider. Every editor call degrades to bulk when the editor's key
# is missing, its daily ceiling is hit, or the API refuses/fails.
[llm]
bulk = "deepseek" # triage, deep assessment, and every fallback
editor = "anthropic" # lineup, summaries, The Brief, the weekly profile rebuild
triage_batch_size = 25 # articles per first-pass triage request
deep_batch_size = 8 # articles per close-reading assessment request
score_temperature = 0.3 # sent only by providers that take a temperature (kind = "openai")
editorial_temperature = 0.8 # summaries and The Brief on an openai-kind provider
# The provider registry. Any number of entries; a role above names one by its
# table name. Keys never live here: DAILY_EPUB_PROVIDERS__<NAME>__API_KEY.
# `max_daily_usd` is a per-provider runaway guard per UTC day (0 = none), not
# accounting — set hard spend limits in each provider's dashboard as well.
# Prices are USD per 1M tokens and only feed the guard's arithmetic.
[providers.deepseek]
kind = "openai" # openai | anthropic
base_url = "https://api.deepseek.com/v1"
model = "deepseek-v4-flash" # DeepSeek-V4-Flash-0731 (confirmed 2026-08-15)
# api_key via DAILY_EPUB_DEEPSEEK__API_KEY env
score_batch_size = 12
score_temperature = 0.3
editorial_temperature = 0.8
# USD per 1M tokens, used for the cost guardrail.
# api_key via DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY env
max_daily_usd = 2.0
max_concurrent_requests = 4 # triage and deep-assessment batches in flight
price_input_per_mtok = 0.14
price_cached_input_per_mtok = 0.0028
price_cache_read_per_mtok = 0.0028 # prefix-cache hits
price_cache_write_per_mtok = 0.0 # DeepSeek caches implicitly, no write charge
price_output_per_mtok = 0.28
# Claude Opus 5 over the Messages API. Requests carry `output_config.effort`,
# a cached system block, and `fallbacks = "default"` so a classifier refusal is
# re-routed server-side; a refusal that still comes back degrades to bulk.
[providers.anthropic]
kind = "anthropic"
base_url = "https://api.anthropic.com"
model = "claude-opus-5"
# api_key via DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY env
effort = "high" # low | medium | high | xhigh | max → output_config.effort
max_daily_usd = 3.0
max_concurrent_requests = 4 # summaries in flight when this is the summary provider
price_input_per_mtok = 5.0
price_cache_read_per_mtok = 0.5
price_cache_write_per_mtok = 6.25
price_output_per_mtok = 25.0
# Gemini 3.8 Flash through Google's OpenAI-compatible endpoint (beta, verified
# 2026-09-02). Declared but unreferenced until a role names it.
[providers.gemini]
kind = "openai"
base_url = "https://generativelanguage.googleapis.com/v1beta/openai"
model = "gemini-3.8-flash"
# api_key via DAILY_EPUB_PROVIDERS__GEMINI__API_KEY env
effort = "high" # minimal | low | medium | high → reasoning_effort
max_daily_usd = 3.0
max_concurrent_requests = 4
price_input_per_mtok = 0.75 # promotional through 2026-12-31; $1.50 from 2027-01-01
price_cache_read_per_mtok = 0.075 # implicit cache hits; $0.15 from 2027-01-01
price_cache_write_per_mtok = 0.0
price_output_per_mtok = 3.75 # includes thinking tokens; $7.50 from 2027-01-01
# Voyage AI embeddings behind the interest and rated-neighbour signals. Set
# `enabled = false` (or leave the key unset) and the paper still builds: the
# learned signals are simply absent, never a penalty.
[voyage]
enabled = true
base_url = "https://api.voyageai.com/v1"
model = "voyage-4-lite"
# api_key via DAILY_EPUB_VOYAGE__API_KEY env
output_dimension = 512 # 256 | 512 | 1024 | 2048
batch_size = 32
max_concurrent_requests = 4
max_input_chars = 60000 # per article, cut on a char boundary
max_daily_usd = 0.50 # runaway guard ($0.02 / M tokens)
[curation]
max_article_count = 28 # hard ceiling; there is no minimum (§13)
recent_rejection_days = 7
recent_rejection_floor = 3.0
always_include_feeds = [] # miniflux feed ids or site urls
blocked_domains = []
# Extra paywalled hosts, merged with the built-in list (nytimes, wsj, ft, …).
@@ -66,6 +134,59 @@ good_value = 0.35
not_for_me_value = -1.0
verdicts_in_prompt = 60
# Every weight, quota, gate and threshold of the personalized ranker. The
# learned signals (`knn`, `feed`) contribute nothing until their gates open:
# the weight ramps linearly from `*_floor` to `*_full` rated articles.
[curation.ranking]
triage_max = 800 # eligible articles the triage LLM reads
deep_keep = 120 # deep-assessment set
shortlist_keep = 60 # what the editor sees
assessment_reuse_days = 3
rating_lookback_days = 180
rating_half_life_days = 60
neighbour_k = 5
negative_coefficient = 0.75
knn_floor = 8
knn_full = 25
feed_floor = 15
feed_full = 40
semantic_min_words = 300
exploration_slots = 5
embedding_retention_days = 120 # `features prune`: unrated, unpublished vectors
telemetry_retention_days = 180 # `features prune`: candidate_runs and article_assessments
[curation.ranking.quotas]
triage = 60
interest = 20
knn = 20
# Weights need not sum to 1; they are renormalized over the present signals.
[curation.ranking.weights.preliminary]
interest = 0.35
knn = 0.25
heuristic = 0.20
feed = 0.10
social = 0.10
[curation.ranking.weights.utility]
quality = 0.40
fit = 0.20
knn = 0.15
interest = 0.10
feed = 0.05
triage = 0.05
social = 0.03
heuristic = 0.02
[curation.ranking.diversity]
cluster_threshold = 0.85
per_cluster_cap = 2
utility_protected = 10
[editorial]
summary_model = "editor" # editor | bulk — which [llm] role writes the summaries
summary_input_tokens = 3000 # article text offered per summary
[publish]
# Where both EPUB editions land, and what the OPDS feed lists. BookOrbit is
# optional — it just watches this folder if you run it.
+127 -5
View File
@@ -1,6 +1,8 @@
# Implementation notes (shared brief for all implementation agents)
Authoritative spec: `docs/plans/2026-08-15-the-daily-epub.md`. Read it fully before writing code.
For curation (§3.5, §3.6 and §3.9 of that spec) the authority is now
`docs/plans/2026-09-02-personalized-curation-v2.md`; see "Curation v2" below.
This file records implementation-time decisions and verified external facts. Follow both.
## Verified external facts (2026-08-15)
@@ -68,6 +70,47 @@ This file records implementation-time decisions and verified external facts. Fol
walks back up to `MAX_LOOKBACK_DAYS` and the section is datelined with the day it actually
covers, not the masthead date.
## Verified external facts (2026-09-02, curation v2)
- **Anthropic Messages API** (verified 2026-09-02 against the bundled Claude API reference):
`POST https://api.anthropic.com/v1/messages` with headers `x-api-key`,
`anthropic-version: 2023-06-01`, `content-type: application/json` and
`anthropic-beta: server-side-fallback-2026-07-01`. Model id `claude-opus-5`; pricing
**$5.00 / M input, $25.00 / M output**, cache reads 0.1× input ($0.50/M), cache writes
1.25× ($6.25/M); the minimum cacheable prefix is 512 tokens. **No sampling parameters**
(`temperature`, `top_p`, `top_k` are a 400) and no `thinking` block — adaptive thinking is on
by default and depth is set with `output_config: {"effort": "high"}` (`low | medium | high |
xhigh | max`). The system prompt goes in `system: [{type: "text", text, cache_control:
{type: "ephemeral"}}]`; no assistant prefill, JSON is asked for in the prompt and parsed
tolerantly. `"fallbacks": "default"` (with the beta header) routes a request the safety
classifiers would refuse to a fallback model server-side; a response can still end with
`stop_reason: "refusal"` on HTTP 200, which the code treats as an error that degrades the
call to DeepSeek. Usage fields: `input_tokens` (uncached remainder),
`cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`. Timeout 300 s;
retry 429/5xx/network, never 400. Key only from `DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY`
(the provider registry below; the pre-registry `DAILY_EPUB_ANTHROPIC__API_KEY` is a startup
error).
- **Gemini 3.8 Flash over the OpenAI-compatible endpoint** (beta, verified 2026-09-02):
`POST https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` with
`Authorization: Bearer <key>`, the standard `messages` / `temperature` /
`response_format: {"type": "json_object"}` body. Model id `gemini-3.8-flash`. Reasoning depth
is the OpenAI `reasoning_effort` field, which Google maps onto Gemini 3.x's `thinking_level`
(`minimal | low | medium | high`; `none` is not accepted by 3.x models). Usage: implicit
cache hits are reported in `prompt_tokens_details.cached_tokens` (the same field DeepSeek
now fills), and `completion_tokens` already includes the thinking tokens that
`completion_tokens_details.reasoning_tokens` breaks out — so output is priced from
`completion_tokens` alone, never the sum. Prices per 1M tokens (promotional through
2026-12-31): **$0.75 input, $0.075 cache read, $3.75 output** (thinking included); from
2027-01-01 **$1.50 / $0.15 / $7.50**. No cache-write charge. Key only from
`DAILY_EPUB_PROVIDERS__GEMINI__API_KEY`. Shipped as `[providers.gemini]`, unreferenced until
a role names it.
- **Voyage AI embeddings** (verified 2026-09-02): `POST https://api.voyageai.com/v1/embeddings`
with `Authorization: Bearer <key>`; body `{input: [...], model: "voyage-4-lite", input_type:
"document" | "query", truncation: true, output_dimension: 512, output_dtype: "float"}`. Up
to 1,000 inputs and 1M tokens per request, 32k tokens per input. Vectors are
unit-normalized, so dot product = cosine. **$0.02 / M tokens** after a 200M-token free
allocation. Key only from `DAILY_EPUB_VOYAGE__API_KEY`.
## Cross-cutting implementation decisions
1. **sqlx usage**: use *runtime* queries (`sqlx::query(...).bind(...)`) and manual row mapping
@@ -80,12 +123,32 @@ This file records implementation-time decisions and verified external facts. Fol
Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
4. **HTTP**: one shared `reqwest::Client` (rustls, gzip, no cookies, 10s timeouts, UA
`the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`), passed by clone.
5. **LLM**: `async-openai` with custom base URL. All LLM calls go through `curate/llm.rs`
`LlmClient` which tracks token usage into a shared `UsageMeter` (input/cached/output tokens,
cost usd) and enforces `max_daily_usd`.
5. **LLM**: a hand-rolled `reqwest` client, not `async-openai` (the published crate exposes
neither `Client` nor `CreateChatCompletionRequest` at the pinned version). Every LLM call
goes through `curate/llm.rs`: `LlmClient { provider, system_prompt, model, effort,
max_concurrent_requests, meter, backend, retry }` over the `ChatBackend` trait, with two
wire protocols — `OpenAiCompatibleBackend` (`{base_url}/chat/completions`, bearer key,
`response_format: json_object`, `reasoning_effort` when the provider has an `effort`) and
`AnthropicBackend` (Messages API, facts above). **Providers are config, not code**: the
`[providers.<name>]` registry (`kind = openai | anthropic`, `base_url`, `model`, `effort`,
`max_daily_usd`, `max_concurrent_requests`, `price_*`) is a `BTreeMap<String,
ProviderConfig>`, and `[llm] bulk = "<name>"` / `editor = "<name>"` assign the two roles by
name (`editor = ""` means everything runs on bulk; both roles on one provider share one
client and one ceiling). `LlmClient::for_provider(name, &cfg, ..)` dispatches on `kind`;
`Llms::from_config(&config, prompt, &meters)` builds the roles; `editor_or_bulk()` degrades
to bulk when the editor client is missing or its meter is tripped. One `UsageMeter` per
*referenced* provider (`llm::provider_meters`), keyed by provider name — the same key used
for `runs.provider_costs_json`, the UTC-day spend preload and the log lines — plus Voyage's
own. `LlmClient.provider` is the config name, never the kind. The system prompt is sent
first and byte-identical within a run so every provider's prefix cache hits. Keys come only
from `DAILY_EPUB_PROVIDERS__<NAME>__API_KEY` (figment lower-cases the path, so provider
names are `[a-z0-9_]+`); `daily-epub config check` prints the resolved roles without
opening the database.
6. **Testing**: unit tests inline per module; integration tests in `tests/` over fixture JSON in
`tests/fixtures/`. Never hit the network in tests. LLM stage mockable via `--skip-llm`
(prefilter order used for selection, feed excerpts as summaries).
`tests/fixtures/`. Never hit the network in tests: `MockBackend` (`ChatBackend`) and the
embedding mock (`EmbeddingBackend`) stand in for all three providers. `--skip-llm` makes
zero LLM calls (admission by cheap signals, `select_without_llm` by utility, excerpt
summaries); `--skip-embeddings` makes zero Voyage calls.
7. **Style**: rustfmt defaults, `cargo clippy` clean-ish, no `unwrap()` outside tests, tracing
spans per pipeline stage.
8. **File ownership**: waves of agents work in parallel on disjoint files. Do not edit files
@@ -99,3 +162,62 @@ This file records implementation-time decisions and verified external facts. Fol
`style-x4.css`). Askama 0.12+ configured via `askama.toml` if needed.
12. **Determinism**: chapter ids `art-{entry_id}`, stable filenames, issue regeneration for the
same date replaces prior rows/files (idempotent upsert everywhere).
## Curation v2 (2026-09-02)
The personalized ranker is specified in `docs/plans/2026-09-02-personalized-curation-v2.md`
(§0 settled decisions, §3 target pipeline, §19 configuration, §21 the seven landed steps);
`docs/plans/2026-09-02-curation-v2-progress.md` records per-step deviations. Facts an
implementer needs that are easy to get wrong:
- **Tables** (`migrations/0002_curation_v2.sql`, `0003_drop_scores.sql`; never edit
`0001_init.sql`): `rating_events` (append-only; the current verdict is the latest
`explicit` event), `article_embeddings` and `interest_embeddings` (f32 little-endian BLOBs,
`input_hash` = sha256 of the embedded text), `article_assessments` (`stage IN ('triage',
'deep')`, reused while `model` and `prompt_version` match and `assessed_at` is within
`assessment_reuse_days`; `--rescore` ignores the cache), `candidate_runs` (one row per
considered article per run, upserted with every column set on each stage transition),
`runs.config_json` / `runs.provider_costs_json`, `issue_articles.why`. `ratings`,
`feed_priors` and `scores` are dropped; `kv` keeps `ingest_watermark`, `taste_profile`,
`taste_profile_learned`, `profile_version`.
- **Feedback**: `Vote` is `loved | good | down` (`NotForMe`); the HMAC message stays
`{issue_date}/{article_id}/{vote}`. `Vote::parse("up")` → `Loved` and `auth::verify_token`
still accepts tokens signed over the literal `up` segment because published issues carry
those links. Keep both.
- **Budget day**: each provider's `UsageMeter` is preloaded with the spend of earlier runs on
the **UTC date of the run's `started_at`**, summed from `runs.provider_costs_json`
(`db::spend_for_date` by nominal issue date is gone). A tripped meter skips that provider's
remaining calls; the paper always publishes.
- **Lock**: `src/lock.rs` takes `libc::flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`
for `generate`, `profile rebuild`, `features backfill` and `backfill-social`; a second
writer exits with "<command> is already running". `serve`, `explain`, `stats`, `ratings`,
`features prune` and `db migrate` never take it.
- **Retention**: `telemetry::prune` removes `article_embeddings` of unrated, unpublished
articles older than `embedding_retention_days` (120) and `candidate_runs` rows plus
`article_assessments` older than `telemetry_retention_days` (180). `features prune` runs it
on demand; `generate` runs it once after publishing, best effort.
- **Keys**: `DAILY_EPUB_PROVIDERS__<NAME>__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` map onto
`ProviderConfig.api_key` / `VoyageConfig.api_key` through figment; the fields exist only
for that mapping and are never documented in TOML, logged, or stored
(`Config::providers_redacted()` is what reaches `runs.config_json`).
- **Provider registry** (2026-09-02, after step 7): the `[deepseek]` and `[anthropic]` tables
and the top-level `max_daily_usd` are gone. `[llm]` holds the role names and the role-level
knobs (`triage_batch_size`, `deep_batch_size`, `score_temperature`,
`editorial_temperature`); `[providers.deepseek]`, `[providers.anthropic]` and
`[providers.gemini]` ship in `config.example.toml` and are `Config::default()` key for key.
Stale shapes fail at load, naming the new key: a `[deepseek]`/`[anthropic]` header, a
top-level `max_daily_usd`, any of the four role keys outside `[llm]`, or a
`DAILY_EPUB_DEEPSEEK__*` / `DAILY_EPUB_ANTHROPIC__*` environment variable. Batching
concurrency (`triage`, `assess`) is the bulk provider's `max_concurrent_requests`; the
summaries fan out at the summary provider's. `Models { bulk, editor, summaries }` in the
colophon and Behind the paper stay model ids taken from the built clients.
- **Provider rejections** (2026-09-02): `curate::batch` bisects a triage or deep batch that
fails with a non-transient error (`Api`, `Refusal`, `EmptyResponse`, or a response that
parses to zero items) down to single articles; a rejected single is retried once on the
editor client when it is another provider. An article both refuse gets an
`article_assessments` row with `kind = 'provider_rejected'`, `score`/`fit` NULL,
`rationale = "<provider>: <message ≤ 200 chars>"`, the bulk `model` and the stage's
`prompt_version`; the cache loaders skip it (leaving the assessment absent) while it is within
`assessment_reuse_days`, `--rescore` ignores it, and `admit::hygiene` never treats it as a low
score. Because a recovered article's row carries the editor's model, reuse accepts rows whose
`model` is either configured model (`triage::reusable_models`).
@@ -0,0 +1,139 @@
# Personalized Curation v2 — implementation progress and handoff
**Updated:** 2026-09-02 (end of session 2)
**Plan:** `docs/plans/2026-09-02-personalized-curation-v2.md` (§21 is the step sequence)
**Branch:** `curation-v2` (branched from `main` at `a599745`; not merged, not pushed)
**Briefs:** `docs/plans/curation-v2-briefs/` — `00-preamble.md` + one `stepN.md` per step. Each
brief was handed to an implementation agent as `cat 00-preamble.md stepN.md`.
## Where things stand
| Step (§21) | Status | Commit |
|---|---|---|
| 1. Feedback and profile | **done**, reviewed (codex review: no actionable issues) | `3a9f4b9` |
| 2. Claude editor and editorial | **done** | `57efbb4` |
| 3. Embeddings, signals, telemetry | **done** (branch `curation-v2-step3`, merged) | `ea3b141` + merge commit |
| 4. Triage replaces the gate | **done** (Codex plugin, high effort), reviewed | `10f4afd` |
| 5. Deep assessment, utility, diversity | **done** (Codex started, cut off by quota; Claude agent finished), reviewed | `05a74a0` |
| 6. Paper telemetry, stats, lock | **done** (Claude agent), reviewed | `d261cd4` |
| 7. Cleanup + implementation notes | **done** (Claude agent), reviewed | `d403c51` |
**Post-plan addition (same day):** a provider-agnostic LLM registry — `[llm]` roles (`bulk`,
`editor`) over named `[providers.*]` entries of `kind = openai | anthropic`, keys from
`DAILY_EPUB_PROVIDERS__<NAME>__API_KEY`, a `gemini` entry (Gemini 3.8 Flash via Google's
OpenAI-compatible endpoint) declared but unreferenced, and a `daily-epub config check` subcommand.
The old `[deepseek]`/`[anthropic]` tables, top-level `max_daily_usd` and the old key env vars
fail loudly. The server upgrade is written up in `docs/runbooks/curation-v2-migration.md`.
All seven plan steps are implemented. `cargo fmt --check`, `cargo clippy --all-targets`
(including `-W dead_code`) and `cargo test` (329 lib tests + 6 bin tests + the 7 integration suites) are green at HEAD.
Nothing has been run against the production database or the real providers. No local
`config.toml`, database, or API keys exist on the dev box, so verification so far is tests only.
## Decisions and deviations made while implementing
Steps 1–3 (session 1):
- **Legacy `up` links.** `Vote::parse("up")` → `Loved`, and `auth::verify_token` also accepts tokens
signed over the literal `up` segment so already-published issues keep working. Keep both.
- **Same-date regeneration** no longer excludes its own picks (`published_before` uses issue dates
strictly before the run date), per plan §8.1.
- **`VoyageConfig.api_key`** exists as a field (figment maps `DAILY_EPUB_VOYAGE__API_KEY` into it;
`deny_unknown_fields` would otherwise reject the env var). Never document it in TOML.
- **`anthropic.max_concurrent_requests`** is validated but not consumed; summary concurrency is
the constant `SUMMARY_CONCURRENCY = 4` in `editorial.rs`.
- `RATINGS_LOOKBACK_DAYS` in `profile/mod.rs` is effectively unbounded (36,500) for the prompt
verdict block and the weekly rebuild; the knn/feed preference state uses
`curation.ranking.rating_lookback_days` (180) as the plan says.
- Footer CSS: `.rating` has no `white-space: nowrap` (it would clip on narrow e-ink screens).
Step 4:
- Hygiene order is blocked → published_before → recently_rejected, all skipped for auto-includes.
- `admitted_by` records every retriever whose own top-N would have taken the article (first entry
is the admitting one); exploration is flagged only when `exploration` admitted it.
- Under the pool cap (`triage_max`) the surplus gets `stage='eligible'`, `excluded_reason='not_admitted'`.
- Migration `0003_drop_scores.sql` drops `scores`; the churn rule reads `article_assessments`.
Step 5:
- `Candidate` is the only flow type; `ScoredArticle`, `combined_score`, `score.rs`, `select.rs` are gone.
- The editor omits the `facets:` line when every facet is unknown (plan renders it unconditionally).
- Articles cut at the shortlist keep the stage they reached (`admitted` if never deep-assessed,
else `assessed`) with `excluded_reason` `cluster_suppressed` (hit the base cap) or `shortlist_cap`.
- `Curator::assess` is a no-op under `--skip-llm`; cached deep rows are still reused when DeepSeek
is merely down.
- `--near-misses` and the Behind-the-paper list order by utility, falling back to the preliminary
blend per row.
- Fixtures: `tests/fixtures/deepseek_deep_batch{,_messy}.json` replace the old score fixtures;
`deepseek_triage_batch.json` added in step 4.
Step 6:
- `BehindThePaper` hangs off `Issue.behind` (filled after `build_issue`), not a `build_issue` argument.
Generation time and cost shown in the chapter are measured at issue assembly (they exclude the
EPUB build and publish, a few seconds); the cost includes Voyage.
- Voyage spend is recorded under `provider_costs["voyage"]`; `finish()` counts it once and keeps its
tokens out of the LLM aggregate. Voyage spend is not preloaded into its meter for the UTC day.
- `src/lock.rs` uses `libc::flock` (`libc` was already in `Cargo.lock`; now a direct dependency).
The holder's command name is written into the lock file for the "X is already running" message.
- The mid-run `admission:` line is `debug`; the four-line §15.4 block is logged once after
`finish()` and printed by `print_report`. Timings `summaries` and `brief` replace `editorial`.
Step 7:
- `features prune` sweeps `article_embeddings`, `candidate_runs` and now `article_assessments`;
`generate` runs it once after a published (non-dry) issue, best effort.
- The Brief chapter's TOC/title string is now "The Brief" (was still "From the Editor").
- Kept on purpose: `UsageMeter::new(&DeepseekConfig, ..)` (a convenience over `with_prices`, 12
call sites); `StageCounts.candidates` (feeds `runs.candidates` and the colophon); the
`prefilter_keep`/`score_batch_size` removed-key startup errors (operator guards); the SQL comment
"filled in by step 4/5" inside the applied migration `0002` (sqlx checksums); six v1-era
unreferenced `pub` items (`enrich_one`, `issues_before`, `today_in_tz`, `to_json_pretty`,
`with_retry_policy`, `db::issues_before`) as pre-v2 API surface.
- `config::tests::shipped_example_config_matches_the_defaults_key_for_key` compares the example
against `Config::default()` in both directions (numbers within 1e-6).
## Operator to-dos before the first real run
1. Set `DAILY_EPUB_ANTHROPIC__API_KEY` and `DAILY_EPUB_VOYAGE__API_KEY` in the systemd env file.
2. Set hard spend limits in the DeepSeek, Anthropic and Voyage dashboards (the meters are runaway
guards, not accounting).
3. Copy `data/profile.md` to wherever `profile_path` points on the server (default is relative to
`WorkingDirectory=/var/lib/daily-epub`, like `data/scour-interests.opml`).
4. Back up the DB, then `daily-epub db migrate` (0002 drops `ratings`/`feed_priors`, 0003 drops `scores`).
5. `daily-epub features backfill --rated-only` then `--days 30` to warm the embedding cache.
6. `daily-epub generate --dry-run`, read the paper (including the new Behind-the-paper chapter),
then `explain --date … --near-misses` and `stats`.
7. Watch the first few real runs' `providers:` log line against the ~$0.80/day estimate (plan §3).
## How the work was run (so the next session can repeat it)
- Orchestrator: Claude Code (this repo), one implementation agent per step, review + commit by the
orchestrator after independent `cargo fmt/clippy/test`.
- **Codex via the `openai-codex` Claude Code plugin (v1.0.6) works on this host as of session 2**
(the operator fixed bubblewrap). Launch:
`node ~/.claude/plugins/cache/openai-codex/codex/1.0.6/scripts/codex-companion.mjs task --background --write --effort high "$(cat brief.md)"`,
poll `… status <job-id> --json` (`.job.status`), read `… result <job-id>`. Inside its sandbox
`bind()` on 127.0.0.1 is forbidden, so exactly ten pre-existing loopback tests fail there
(`curate::llm::tests::anthropic_*` ×4, `extract::tests::relative_urls_resolve_against_the_url_we_landed_on`,
`server::tests::*` ×5, plus the two `m7_server` integration tests); the brief must tell the agent
to ignore those and the orchestrator runs the full suite outside. The old "host quirk" paragraph
in `00-preamble.md` is obsolete.
- Codex usage budget is roughly one large step per ChatGPT usage window: in session 2 it finished
step 4 (~26 min) and was cut off 21 minutes into step 5 ("try again at 7:44 PM"). The job status
becomes `failed` with the partial work left in the tree; a Claude `general-purpose` agent with a
"finish, don't restart" brief (known failures, scope audit checklist) completed it. Steps 6 and 7
ran on Claude agents directly.
- Steps 4 → 5 → 6 → 7 were sequential, each reviewed against the plan sections named in its brief
before committing.
## Next session: what remains
1. `git checkout curation-v2 && cargo test` (expect green). Follow
`docs/runbooks/curation-v2-migration.md` on the server.
2. Optional: a Codex `review --background --scope branch --base main` pass over the whole branch.
3. `git merge --no-ff curation-v2` into `main`, build, deploy, and do the operator to-dos above.
4. After a week of real runs: read `stats`, tune `[curation.ranking]` from what `explain` shows,
and revisit the deferred items in plan §23.
@@ -0,0 +1,58 @@
# Implementation agent brief — The Daily EPUB, "Personalized Curation v2"
You are implementing ONE step of a multi-step plan in the Rust repository at
`/home/thallada/workspace/the-daily-epub` (branch `curation-v2`, already checked out).
The orchestrator hands you `docs/plans/curation-v2-briefs/00-preamble.md` (this file) followed by one
`stepN.md` from the same directory; `docs/plans/2026-09-02-curation-v2-progress.md` records what has
landed so far.
The plan is `docs/plans/2026-09-02-personalized-curation-v2.md`. Read it in full before
writing code: §0 records settled decisions (do not reopen them), §1 lists the files to
read first, §21 is the implementation sequence. Also read
`docs/plans/2026-08-15-implementation-notes.md` §"Cross-cutting implementation decisions"
(note: item 5 is stale — the code uses a hand-rolled `reqwest` client in `src/curate/llm.rs`,
keep that).
## Hard rules
- Implement ONLY the step assigned below. Later steps land separately; do not start them.
Where this step needs a type or table that a later step fills in, create it now exactly as
the plan specifies and leave it empty/unused with a short `// filled in by step N` comment.
- Conventions: sqlx *runtime* queries (`sqlx::query(...).bind(...)`, never the `query!` macros),
`jiff` for time, RFC3339 UTC strings in SQLite, `thiserror`/`anyhow` error style, no `unwrap()`
outside tests, tracing spans per stage, rustfmt defaults, askama templates.
- Tests never touch the network. Use mock backends following `MockBackend` in `src/curate/llm.rs`.
- Follow the plan's names for modules, functions, config keys, table and column names, enum
strings and prompt constants exactly. If the plan is internally contradictory or impossible
for this step, pick the closest behaviour, keep the paper publishable, and describe the
deviation in your final report. Do not invent scope beyond the plan.
- API keys only from env vars; never in config files, logs, tests or the database.
- Prefer existing dependencies in `Cargo.toml`. Add a new crate only if there is no reasonable
way without it, and say so in the report.
- Do NOT commit and do NOT create branches. Leave all changes in the working tree; the
orchestrator reviews and commits. Do not edit anything under `docs/`.
- Keep `config.example.toml` and README in sync with any config keys you add or remove
(the test `shipped_example_config_parses` must pass).
- Before finishing, run and make pass: `cargo fmt`, `cargo clippy --all-targets` (no new
warnings), `cargo test`. Fix what you broke; do not delete or `#[ignore]` tests to get green
unless the plan removes the feature they cover (then move/replace the tests as §18 says).
## Final report
End with a concise report (this is what the orchestrator reads): what you implemented, every
deviation from the plan and why, anything from the step you could not finish, and the exact
`cargo test` summary line(s). Keep it under 60 lines.
## Host notes
- The `openai-codex` Claude Code plugin sandbox works on this host (bubblewrap fixed 2026-09-02):
`apply_patch`, the shell, `cargo build/test/clippy` and the warm `~/.cargo/registry` all work.
- Inside that sandbox `bind()` on 127.0.0.1 is forbidden, so ten pre-existing tests fail there for
environmental reasons: the four `curate::llm::tests::anthropic_*` tests,
`extract::tests::relative_urls_resolve_against_the_url_we_landed_on`, and the five
`server::tests::*` (the two `tests/m7_server.rs` tests likewise). They are out of scope; do not
touch them. Every other test must pass; the orchestrator runs the full suite outside the sandbox.
- Cargo's registry is already warm; do not add dependencies that would need a network fetch.
- Work autonomously to completion. Do not stop to ask questions; make the closest-to-plan choice
and record it in the final report.
+73
View File
@@ -0,0 +1,73 @@
---
# YOUR STEP: 1 — Feedback and profile (plan §21 step 1)
Scope (plan §6, §7, §8; tests in §20 under Vote, Rating events, Migration, and the prompt):
1. **Migration `migrations/0002_curation_v2.sql`** (§7). Create `rating_events`,
`article_embeddings`, `interest_embeddings`, `article_assessments`, `candidate_runs` with
the exact columns/indexes in §7; add `runs.config_json`, `runs.provider_costs_json`,
`issue_articles.why`. Copy `ratings` rows into `rating_events` per §6.2 (`vote=1 → 'loved',1.0`;
`vote=-1 → 'not_for_me',-1.0`; `source='migration'`; `kind='explicit'`; `event_at=rated_at`),
then `DROP TABLE ratings; DROP TABLE feed_priors;`.
**Orchestrator decision:** do NOT drop `scores` in this step. Stage A scoring
(`db::upsert_score`, `recently_low_scored_ids`) still writes/reads it until step 4 replaces
it with `article_assessments`; step 4 will add a `0003` migration dropping it. Do not edit
`0001_init.sql`. Migration is plain SQL, no Rust bootstrap.
2. **Three-way `Vote`** (§6.1) in `src/types.rs`: `Loved | Good | NotForMe`, `as_str` →
`"loved" | "good" | "down"`, `parse` also accepts legacy `"up"` → `Loved`, `value(&FeedbackConfig)`
→ configured 1.0 / 0.35 / −1.0. Add `[curation.feedback]` config
(`loved_value`, `good_value`, `not_for_me_value`, `verdicts_in_prompt = 60`). Remove `Vote::as_i64`,
`Rating`, `FeedPrior`, and everything that depended on `ratings`/`feed_priors`
(`db::upsert_rating`, `ratings_with_feed`, `upsert_feed_prior`, `feed_priors`,
`profile::rebuild_feed_priors`, the feed-prior term in `prefilter.rs` and `ScoredArticle.feed_prior`
— set the prefilter's feed-prior contribution to nothing; the field can go). HMAC links
(`src/auth.rs`) must verify for all three votes; message stays `{issue_date}/{article_id}/{vote}`.
3. **Footer + confirmation page** (§6.1): `RatingLinks` in `src/epub/chapters.rs` and
`src/epub/templates/chapter.xhtml` render three links on one line sized for e-ink:
`Was this a good pick? [ Loved it ] [ Good ] [ Not for me ] Read online ↗`.
X4 edition still renders no links. `server::handle_rating` appends one `rating_events` row
(`kind='explicit'`, `source='epub'`, label `loved|good|not_for_me`, value from config) and
returns "Recorded: Loved it — thanks." (etc.) plus the other two links so a mis-tap can be
corrected. No feed-prior rebuild, no provider call.
4. **`rating_events` access** (§6.2) in `src/db.rs`: `append_rating_event(...)`,
`db::current_ratings(lookback_days) -> Vec<RatedArticle>` implementing the latest-explicit-event
rule (`ORDER BY event_at DESC, id DESC`, `cleared` removes the article from the learned set),
joining `articles`, the best entry's feed title, and the most recent `issue_articles.summary`.
Move `RatedArticle` to `src/types.rs` with `summary: Option<String>`, `facets: Option<Facets>`
(define `Facets` per §12.1 now, all `Option`s, unused until step 5), `note: Option<String>`,
plus `RatingEvent`. Also define `RatedArticle.value: f64` and `label`.
5. **Ratings CLI** (§6.3) in `src/main.rs`: `ratings list [--days 90] [--label loved|good|down|cleared]`,
`ratings set --article ID|--url URL --label loved|good|down [--note "..."]`,
`ratings clear --article ID|--url URL`. `set`/`clear` append rows with `source='cli'` and
`issue_date` from the latest `issue_articles` row for the article if any; `--url` canonicalizes
with `dedupe`'s canonical URL function then `db::article_id_for_url`. Print something useful.
6. **`data/profile.md`** (§8.2): create the file with the initial content shown in §8.2 verbatim;
add `profile_path` config (default `data/profile.md`); loader that parses any `## Interests`
section (one per line, leading `- ` stripped) and unions it case-insensitively with the OPML
interests, passing everything else through verbatim. Remove `STATED_PREFERENCES` and the
reader/wants/does-not-want/how-to-judge parts of `PROFILE_PREAMBLE` from code (they now live
in the file); the editor-in-chief framing paragraph stays in code. If the file is missing,
warn and continue with the OPML interests only.
7. **System prompt** (§8.4): rebuilt every run in this exact order: framing → profile.md verbatim
(minus Interests) → standing interests grouped by `profile::themes::group_into_themes` →
learned adjustments → **recent verdicts**: up to `verdicts_in_prompt` most recent explicit
ratings, newest first, `LOVED | title | feed | one-line summary` (labels `LOVED`/`GOOD`/`NOT FOR ME`),
cleared and duplicate articles removed. Byte-identical within a run. `profile_version` bumps only
on the weekly rebuild, as today. `kv` keeps `ingest_watermark`, `taste_profile`,
`taste_profile_learned`, `profile_version` (§7.8).
8. **Weekly rebuild** (§8.3): still on the DeepSeek client this step (the Claude editor client
arrives in step 2). Each rated line carries
`LOVED | title | feed | summary | facets: format/depth/evidence/technicality/topic_group | note: …`
(omit empty parts), up to 200 most recent explicit ratings, cleared excluded; the softened
"strong prior, not a rule" instruction; the required "Diversity check:" bullet.
9. Pipeline: remove the `rebuild_feed_priors` call and every `feed_priors` reference; the paper
must still build. `handle_rating` and `profile rebuild` must not touch dropped tables.
Tests to add/adapt (§20): Vote parse/serialize incl. `up`→Loved; HMAC for all three; template
renders three links standard / none X4; latest explicit event wins; `cleared` removes from the
learned set; migration on a temp DB through 0001 then 0002 copies old ratings with the right
labels/values and `ratings`/`feed_priors` are gone; CLI `set`/`clear` append `source='cli'` rows
(test the db-level function); profile.md Interests parsing and union; system prompt section order
and verdict block content. Update `tests/m7_server.rs` and `tests/m4_epub.rs` as needed.
+86
View File
@@ -0,0 +1,86 @@
---
# YOUR STEP: 2 — Claude editor and editorial (plan §21 step 2)
Step 1 has landed (three-way votes, `rating_events`, `data/profile.md`, the rebuilt system
prompt). Read `git log -3` and the current `src/curate/llm.rs`, `select.rs`, `editorial.rs`,
`pipeline.rs`, `config.rs`, `report.rs`, `db.rs`, `src/epub/chapters.rs` + templates before coding.
Scope (plan §4.1, §4.2, §5, §7.6, §13, §14, §15.1 chapter/in-this-issue `why`, §18 types, §19 config):
1. **`AnthropicBackend`** in `src/curate/llm.rs` implementing `ChatBackend` exactly per §4.2:
`POST {base_url}/v1/messages`; headers `x-api-key`, `anthropic-version: 2023-06-01`,
`content-type: application/json`, `anthropic-beta: server-side-fallback-2026-07-01`;
body `{model, max_tokens: 16000, system: [{type:"text", text, cache_control:{type:"ephemeral"}}],
messages:[{role:"user", content}], output_config:{effort}, fallbacks:"default"}`.
**Never** send `temperature`, `top_p`, `top_k`, `thinking`, or an assistant prefill. Ask for JSON
in the prompt text and parse tolerantly (reuse `strip_code_fence` + the existing tolerant parsing).
Response: concatenate `content[]` blocks with `type == "text"`. Usage: `input_tokens`,
`cache_creation_input_tokens`, `cache_read_input_tokens`, `output_tokens`; cost = input×price_input
+ cache_creation×price_cache_write + cache_read×price_cache_read + output×price_output (per M).
`stop_reason == "refusal"` (HTTP 200) surfaces as a distinct `LlmError` variant that triggers the
fallback; 429/5xx/network are `Transient` (retried by the existing `RetryPolicy`); 400 is never
retried. Request timeout 300 s. API key only from `DAILY_EPUB_ANTHROPIC__API_KEY`.
`ChatRequest` gains `effort: Option<String>`; the Anthropic backend ignores `temperature`/`json`
and maps `system` to the cached system block. `TokenUsage` needs a cache-write counter (keep the
DeepSeek path's cost identical to today). Generalize `UsageMeter` to take a per-provider price
table instead of `&DeepseekConfig`.
2. **`Llms { bulk: Option<LlmClient>, editor: Option<LlmClient> }`** with `editor_or_bulk()` (the
editor when configured and its meter is not tripped, else bulk). Both clients share the exact same
system prompt string (§8.4). `LlmError::MissingApiKey` and friends must name the provider.
3. **Config** (§4.2, §19): `[anthropic]` with `enabled`, `base_url`, `model = "claude-opus-5"`,
`api_key` (env only), `effort = "high"`, `price_input_per_mtok = 5.0`,
`price_cache_write_per_mtok = 6.25`, `price_cache_read_per_mtok = 0.5`, `price_output_per_mtok = 25.0`,
`max_daily_usd = 3.0`, `max_concurrent_requests = 4`; `[deepseek].max_concurrent_requests = 4`;
`[curation].max_article_count = 28`; `[editorial]` with `summary_model = "editor"` (`editor|bulk`)
and `summary_input_tokens = 3000`. Validate `max_article_count >= target_article_count`, effort ∈
{low, medium, high, xhigh, max}, batch/concurrency ≥ 1. Startup logs the resolved models and whether
each provider is enabled (§19). Update `config.example.toml` and the README (prerequisites, cost
line, the note that server-side fallback is enabled, dashboard spend limits as the real backstop).
4. **Budget and concurrency** (§5): one `UsageMeter` per provider, each with its own `max_daily_usd`.
The budget day is the **UTC date of the run's `started_at`**, preloaded by summing
`runs.provider_costs_json` for earlier runs that UTC day; this replaces `db::spend_for_date`
(remove it). Write `runs.provider_costs_json` (`{"deepseek": {...usage, cost_usd}, "anthropic": {...}}`)
and `runs.config_json` (the resolved `[curation]`, `[editorial]`, model names and prompt versions,
as JSON) in `finish_run`. Keep `runs.cost_usd` as the total across providers. Existing Stage A
scoring batches run through `futures::stream::iter(...).buffer_unordered(max_concurrent_requests)`
with the budget check before each request is spawned; a tripped meter skips remaining calls,
lets in-flight finish, records the number of unscored candidates in the report, and continues.
5. **The editor** (§13): `EDITOR_INSTRUCTIONS` replaces `SELECT_INSTRUCTIONS` verbatim from the plan
with `{soft_target}`/`{hard_max}` substituted. Runs on `editor_or_bulk()`; on refusal/error fall
back to the same prompt on the bulk client; if that fails too, `select_without_llm`. Per-item
rendering follows §13 with what exists today (Stage A score/category/rationale instead of
quality/fit/facets — those arrive in step 5; render `flags: always-include | excerpt only`;
`opening:` first 60 words). Do not put the numeric blend in the prompt. `assemble()` keeps section
validation, unique lead, auto-include reinsertion, duplicate-id defence, malformed-response
fallback and the `hard_max` trim (by today's ranking key, utility arrives in step 5).
**Delete the "too few: top up" branch.** `--max-articles N` is a ceiling:
`hard_max = min(config.curation.max_article_count, N)`, `soft_target = min(target_article_count, hard_max)`.
`select_without_llm` respects `soft_target` as its size. Each pick's `why` (≤14 words) lands on
`Pick.why: Option<String>` and in `issue_articles.why` (`replace_issue_articles`).
6. **Editorial** (§14): summaries run on the editor client per `editorial.summary_model` with the
input budget from config (3,000 tokens), concurrency 4 via `buffer_unordered`, fallback per article:
bulk client, then the excerpt. `BRIEF_INSTRUCTIONS` replaces `FRONT_PAGE_INSTRUCTIONS` verbatim
from §14.2 (JSON `{"brief": "..."}`); input is the lineup with sections, each pick's title, feed,
`why`, summary and the Stage A score. `Editorial.section_intros` is removed (or always empty) and
`section.xhtml` renders only the section name; `front_page.xhtml` renders the brief under the
masthead; fallback stays `fallback_front_page_html`. The weekly profile rebuild runs on
`editor_or_bulk()` (pipeline and the `profile rebuild` CLI).
7. **Paper** (§15.1): `chapter.xhtml` gets a small italic `Why it's here: <why>` line under the meta
line; the In-this-issue page shows the `why` line under each summary. `Colophon` gains
`provider_costs` and `models`; the colophon template prints per-provider cost lines and the models
(editor and summaries model, bulk model). `StageCounts`/`RunReport` carry per-provider usage;
`print_report` in `main.rs` prints per-provider cost.
8. Remove `Vote`-era leftovers you notice only if they are in your files; do not touch embeddings,
signals, telemetry or triage (steps 3–5).
Tests to add/adapt (§20 "Anthropic backend", "Editor", parts of "Pipeline"): request body has the
system block with `cache_control`, no `temperature`, `output_config.effort`, `fallbacks`, the beta
header; usage fields parsed into cost with cache read/write prices; `stop_reason: refusal` surfaces as
the fallback-triggering error; 429 retried, 400 not (use a mock backend or a local `axum`/`tokio`
listener — never the real network); a nine-pick response is published as nine; `hard_max` trims;
`--max-articles` is a ceiling; auto-includes reinserted; `why` lines land on picks and in
`issue_articles.why`; refusal/error on the Anthropic mock falls back to the DeepSeek mock with the
same prompt; the brief is parsed and rendered, section intros gone; `provider_costs_json` and
`config_json` are written; budget day preload sums by UTC date.
+103
View File
@@ -0,0 +1,103 @@
---
# YOUR STEP: 3 — Embeddings, signals, telemetry (plan §21 step 3)
You are working in a git **worktree** on branch `curation-v2-step3`, which branches from the step 1
commit. Another agent is implementing step 2 (the Claude editor) in parallel on the main checkout;
you will not see its changes and must not need them. Read `git log -3` and the current
`src/curate/`, `src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs` first.
Do not run `git checkout`, `git stash`, `git worktree` or `git commit`.
Scope (plan §4.3, §7.1, §7.2, §7.4, §7.5, §9, §12.2 normalization only, §12.4 preliminary blend,
§15.2 `explain`, §15.4 log line parts, §16, §17 Voyage row, §19 `[voyage]` + `[curation.ranking]`).
The old heuristic prefilter still gates in this step; signals are computed for every article that
passes hygiene and persisted, nothing about *selection* changes yet.
1. **`src/curate/embedding.rs`** (§4.3, §7.1, §7.2): `EmbeddingBackend` trait mirroring `ChatBackend`
(with a mock for tests), `VoyageBackend` (`POST {base_url}/embeddings`, `Authorization: Bearer`,
body `{input, model, input_type: "document"|"query", truncation: true, output_dimension, output_dtype: "float"}`,
response mapped by index and length-checked, usage `total_tokens` metered), batching by `batch_size`
with `buffer_unordered(max_concurrent_requests)`, retry 429/5xx via `RetryPolicy`, f32
little-endian BLOB encode/decode with length and finiteness checks, `dot(a, b)` with a dimension
check, and cache orchestration: `article_embeddings` keyed by `(model, dimension, input_hash = sha256
of the embedded text)`, overwritten on change. Embedded text = `"Title: {title}\n\n{plain body}"`
via `curate::prompt_text`, whitespace collapsed, cut at `max_input_chars` on a char boundary —
**no feed name, author or scores**. Interests go to `interest_embeddings` with `input_type = "query"`
and the bare interest string. A failed batch leaves those articles without embeddings; never fatal.
A Voyage `UsageMeter` with `price_per_mtok = 0.02` and `max_daily_usd` (0.50) as a runaway guard.
Key only from `DAILY_EPUB_VOYAGE__API_KEY`. Raw vectors never reach logs or reports.
2. **Config** (§4.3, §19): `[voyage]` (`enabled`, `base_url = "https://api.voyageai.com/v1"`,
`model = "voyage-4-lite"`, `output_dimension = 512`, `batch_size = 32`, `max_concurrent_requests = 4`,
`max_input_chars = 60000`, `max_daily_usd = 0.50`) and the complete `[curation.ranking]` block from
§19 (`triage_max`, `deep_keep`, `shortlist_keep`, `assessment_reuse_days`, `rating_lookback_days`,
`rating_half_life_days`, `neighbour_k`, `negative_coefficient`, `knn_floor/full`, `feed_floor/full`,
`semantic_min_words`, `exploration_slots`, `embedding_retention_days`, `telemetry_retention_days`,
`[curation.ranking.quotas]`, `[curation.ranking.weights.preliminary]`, `[curation.ranking.weights.utility]`,
`[curation.ranking.diversity]`) with the plan's defaults and the validation rules in §19 (weights
non-negative; `deep_keep ≥ shortlist_keep ≥ target_article_count`; `*_full > *_floor ≥ 0`;
`0 ≤ cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; batch sizes ≥ 1; dimension ∈ {256,512,1024,2048}).
Steps 4–5 will *use* the ranking keys; you only add and validate them. Keep `prefilter_keep` for now.
Startup logs whether Voyage is enabled. Update `config.example.toml` and the README.
3. **`src/curate/signals.rs`** (§9): a `Signals` type where every signal is `Option<f64>` (absent ≠ 0):
`interest` (§9.1 z-scored interest match, top-three interests recorded; raw top-1 cosine fallback
under 30 embedded articles, logged), `knn` (§9.2 preference state from `db::current_ratings(rating_lookback_days)`
joined to `article_embeddings`, decayed weights `value × 0.5^(age/half_life)`, top-k positives and
negatives, `knn = pos − negative_coefficient × neg`, top-three neighbours recorded, gate ramp
`clamp((n − knn_floor)/(knn_full − knn_floor), 0, 1)` multiplying the weight, absent at 0),
`feed` (§9.3 Beta-smoothed decayed rate credited to distinct direct `SourceKind::Feed` feeds, mean
over the article's rated feeds, gate `feed_floor/feed_full`), `social` (existing composite; absent
with no rows), `heuristic` (`longform_points(word_count)` − excerpt-only − roundup penalty, from
`prefilter.rs` **without** the social, Scour/HN, multi-source terms; expose those pieces as
functions). Log once per run:
`preference: N rated articles with embeddings → knn gate X; feed gate Y (n=…)`.
Also implement the **mid-rank percentile normalizer** of §12.2 and the **preliminary blend** of
§12.4 (present-and-active weights renormalized) so `signals_json.norm`/`weights` can be written now.
4. **Pipeline**: after social enrichment, for every article that passes hygiene, compute embeddings
(cached), signals, and the preliminary blend, then run the old prefilter as today. New stage
timings `embed` and `signals`. `--skip-embeddings` uses cached embeddings only (zero Voyage calls);
`--skip-llm` unchanged. `StageCounts` gains `eligible`, `embedded`, `rated_with_embeddings`, and
the report/`print_report` show them. The paper must still build when Voyage is down or unconfigured
(§17): `interest`/`knn` absent, never a penalty.
5. **`src/curate/telemetry.rs`** (§7.4, §7.5): the `candidate_runs` writer. One row per considered
article per run, upserted with `INSERT … ON CONFLICT(run_id, article_id) DO UPDATE` setting every
column. Hygiene-excluded articles get thin rows (`stage='excluded'`, `excluded_reason` ∈
`blocked | published_before | recently_rejected`, `signals_json='{}'`). In this step map the old
pipeline onto the stage vocabulary: passed hygiene → `eligible`; cut by the prefilter →
`eligible` + `excluded_reason='not_admitted'`; prefilter survivors → `admitted`; Stage-A scored →
`assessed`; sent to selection → `shortlisted`; picked → `selected` (+ `editor_why` if a `why`
exists on the pick, else NULL); unpicked shortlist → `excluded_reason='not_selected'`. `signals_json`
follows §7.5 exactly (`v: 1`, `raw`, `norm`, `present`, `weights` = effective preliminary weights,
`top_interests`, `neighbours`, `exploration: false`, `auto_include`, `notes`). `utility` and
`rank_utility` stay NULL until step 5.
6. **CLI** (§15.2, §16) in `src/main.rs`:
`explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]` and
`explain --date YYYY-MM-DD --near-misses [N]` printing the persisted `candidate_runs` row for the
latest non-dry run of that date (or `--run-id`): stage and reason; every raw/normalized signal with
presence and effective weight; top interests with z; nearest rated neighbours; assessments from
`article_assessments` when present (empty until step 4); admitted_by; the editor's `why`. `--url`
canonicalizes (`dedupe`) and looks the article up; if absent from `articles`, print that it was never
ingested. `--near-misses` lists the top N by preliminary blend (utility from step 5 when present)
that were not selected. `features backfill [--days 30] [--rated-only] [--all] [--yes]` embeds rated
and published articles first, then interests, then other recent articles only under `--all`, prints
an estimate and asks for confirmation above 5M tokens unless `--yes`, and is idempotent (warm cache
⇒ zero calls). `features prune` removes `article_embeddings` rows for articles neither rated nor
published older than `embedding_retention_days` and `candidate_runs` rows whose run started more
than `telemetry_retention_days` ago. `features backfill` takes no lock yet (step 6 adds `lock.rs`).
7. Types (§18): `Signals`, `Signal` helpers, `RatedArticle` already exists — extend if needed. Do not
create `Candidate` yet (step 4/5).
Tests (§20 "Embeddings", "Interest z-scores", "Preference", "Normalization", plus telemetry/explain):
BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed
text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the
others; embedded text contains no feed name or author; a broad interest with uniformly high cosine does
not dominate while a specific interest with one strong match does; raw fallback under 30 articles; one
loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high
(anti-centroid); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate
0 below `knn_floor`, 1 at `knn_full`, linear between; feed credit sums to 1 across direct feeds; feed
affinity uses the mean; constant signal → 0.5 for everyone; ties get equal percentiles (400 identical
zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate
missing a signal is scored on the rest; `candidate_runs` rows written for every considered article with
the right stage/reason on a mocked run; `--skip-embeddings` makes zero Voyage calls; a Voyage failure
still publishes; `explain` renders from persisted rows and reports "never ingested"; `features prune`
respects rated/published.
+89
View File
@@ -0,0 +1,89 @@
---
# YOUR STEP: 4 — Triage replaces the gate (plan §21 step 4)
Steps 1–3 have landed: three-way votes and `rating_events`; the Claude editor (`Llms { bulk, editor }`,
`AnthropicBackend`, per-provider `UsageMeter`s, `why` lines, the Brief); Voyage embeddings,
`signals.rs` (interest/knn/feed/social/heuristic, percentile normalizer, preliminary blend),
`telemetry.rs` (`candidate_runs` writer), `explain`, `features backfill|prune`, and the full
`[curation.ranking]` config. Read `git log -6`, then the current `src/curate/` (all files),
`src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs`, `src/types.rs`.
Scope (plan §7.3 reuse and churn rules, §8.1 hygiene, §10 triage, §11 admission and exploration,
§12.4 preliminary blend already exists, §15.4 log line parts, §17 DeepSeek row, §19 `deep_keep`):
1. **Migration `migrations/0003_drop_scores.sql`**: `DROP TABLE scores;` (deferred from step 1).
Remove `db::upsert_score`, `recently_low_scored_ids` and every `scores` reader. The churn rule
now reads `article_assessments`: an article whose latest `triage` `score < recent_rejection_floor`
(3.0) or `deep` `score < recent_rejection_floor` within `recent_rejection_days` (7) is
`recently_rejected`, unless auto-include (§7.3, §8.1).
2. **`src/curate/triage.rs`** (§10): `TRIAGE_INSTRUCTIONS` and `TRIAGE_PROMPT_VERSION = 1` verbatim
from the plan; per-article block exactly as §10 (id, title, feed (category), author, length ·
excerpt only, opening = first 200 words via `prompt_text` + `truncate_words`, `matches interests:`
from `top_interests` with z ≥ 1.5 labelled strong (z ≥ 2.5) / weak, `closest rated:` from
neighbours with cosine ≥ 0.55; omit lines with nothing). Batches of `deepseek.triage_batch_size`
(25) through `buffer_unordered(max_concurrent_requests)` on the **bulk** client, budget check
before each batch. Tolerant parsing generalized from `score.rs::parse_score_response`: a
malformed item never sinks the batch, ids not in the batch are dropped, unknown `kind` → `other`,
`interest` clamped 0–10. Persist every result to `article_assessments (stage='triage', model,
prompt_version, profile_version, score, kind, rationale=why, assessed_at)`.
**Cache** (§7.3): skip articles with a reusable `triage` or `deep` assessment (same `model` and
`prompt_version`, `assessed_at` within `assessment_reuse_days`); `generate --rescore` ignores the
cache. A failed batch leaves `triage` absent for its articles.
**Pool cap** (§10): if eligible > `triage_max` (800), triage the union of top `0.7 × triage_max`
by preliminary blend, top 100 by `interest`, top 100 by `knn` (if active), all auto-includes,
filled to `triage_max` by blend; the rest get `stage='eligible'`, `excluded_reason='not_admitted'`.
3. **`src/curate/admit.rs`** (§8.1, §11): hygiene moves here (`blocked`, `published_before` = in
`issue_articles` for any issue date before this run's date, `recently_rejected` per item 1;
auto-includes never excluded) writing the thin `candidate_runs` rows, and the union admission
into `deep_keep` (120) slots in this order, each retriever taking its top-N by its own signal among
not-yet-admitted, not-excluded articles, recording every retriever that would have taken an
article in `admitted_by` (first = the one that admitted it):
`auto_include` (uncapped) → `triage` (quota 60, floor `interest ≥ 5`) → `interest` (20; floors
`word_count ≥ semantic_min_words`, not `looks_like_roundup`, triage `interest ≥ 3` if triaged)
→ `knn` (20, gate > 0, same floors) → `exploration` (5, §11.1) → `blend` (remaining).
Inactive retrievers release their quota to `blend`. Not admitted → `stage='triaged'` (or
`'eligible'` if never triaged) + `excluded_reason='not_admitted'`.
**Exploration** (§11.1): five slots for articles ranked between `deep_keep` and `deep_keep × 2.5`
by the preliminary blend with `word_count ≥ 300`, not roundups, triage `interest ≥ 4`; ordered by
`sha256(run_date || article_id)`; flagged `exploration = true` through to the editor prompt.
4. **Prefilter reduced** (§18): `prefilter.rs` keeps only hygiene helpers (`is_blocked`,
`is_auto_include`, `looks_like_roundup`) and the text heuristic pieces `signals.rs` uses. Delete
`prefilter::run`, `score_article`'s social/Scour/HN/multi-source terms, `PrefilterContext`, the
`prefilter_keep` config key (replace uses with `curation.ranking.deep_keep`; the config validation
`deep_keep ≥ shortlist_keep ≥ target_article_count` already exists), and `Curator::prefilter`.
A stale `prefilter_keep` key in a config file must fail loudly with a message naming
`curation.ranking.deep_keep` (follow the `bookorbit_dir` precedent in `config.rs`).
5. **Pipeline**: hygiene → embeddings → signals → preliminary blend → triage → admission → the
*existing* Stage A scoring (`score.rs`, unchanged this step) over the admitted deep set → the
editor → editorial, as today. The deep set (`admitted`) is what Stage A scores and what the editor
sees; step 5 replaces Stage A with deep assessment and adds the shortlist. Telemetry stages now:
`excluded | eligible | triaged | admitted | assessed | shortlisted | selected` with reasons per §7.4;
`admitted_by` and `exploration` written; `signals_json.raw/norm/present` gain `triage` (÷10 for norm).
Stage timings `triage` and `admit`; `StageCounts` gains `triaged`, `admitted`, `admitted_by` (map),
`exploration_admitted`, `exploration_selected`. Log the admission line from §15.4:
`admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0`.
`--skip-llm` skips triage; DeepSeek down or tripped ⇒ no triage, admission by
`interest`/`knn`/`blend`, editor still runs (§17). The editor prompt (§13 rendering) shows the
triage score and `flags: exploration` where present. `explain` prints the triage assessment
(score, kind, why) and `admitted_by`.
6. `types.rs` (§18): introduce `Assessment { triage: Option<Triage>, deep: Option<Deep> }` and
`Triage { interest, kind, why, model, prompt_version, assessed_at }` (keep `Deep`/`LlmScore`
compatible with the existing Stage A output for now), and `Candidate { article, auto_include,
exploration, signals, assessment, utility: Option<f64>, cluster: Option<...>, admitted_by: Vec<String>,
stage, excluded_reason }` replacing `ScoredArticle` where the pipeline flows through admission.
Keep the change mechanical where `score.rs`/`select.rs` still consume the old shape (step 5
retires them); an adapter is acceptable.
7. `runs.config_json` gains `TRIAGE_PROMPT_VERSION`; `config.example.toml` and README updated
(`prefilter_keep` removed, `[deepseek].triage_batch_size`, `--rescore`).
Tests (§20 "Triage and deep parsing" triage half, "Admission", churn/cache): realistic triage
fixture parsed; malformed items do not sink a batch; unknown kind → other; every `kind` token in the
prompt round-trips; cached triage reused within `assessment_reuse_days` and ignored with `--rescore`;
churn rule excludes a recent low triage score and spares auto-includes; a strong-interest,
weak-heuristic, no-social article reaches the deep set; a 60-word stub with high interest similarity
is not admitted by `interest`/`knn`; quotas honoured; inactive retrievers release quota; exploration
deterministic per date and rotating across dates; auto-includes always admitted; excluded articles get
thin rows with the right reason; pool cap path yields `not_admitted` rows; the mocked full pipeline
still publishes with DeepSeek failing; migration 0003 drops `scores`.
+69
View File
@@ -0,0 +1,69 @@
---
# YOUR STEP: 5 — Deep assessment, utility, diversity (plan §21 step 5)
Steps 1–4 have landed; triage and union admission now gate the deep set and the old prefilter is
reduced to hygiene helpers. Read `git log -8`, then all of `src/curate/`, `src/pipeline.rs`,
`src/types.rs`, `src/config.rs`, `src/report.rs`, `src/main.rs`, `src/db.rs`.
Scope (plan §12 in full, §13 rendering with facets/neighbours, §7.3 deep rows, §18 module retirements):
1. **`src/curate/assess.rs`** replaces `score.rs` (§12.1): `DEEP_INSTRUCTIONS` and
`DEEP_PROMPT_VERSION = 1` verbatim from the plan (the section palette is substituted from
`curation.sections`). Per article: title, feed, author, length, excerpt-only flag, the triage
`why`, the interest and neighbour hint lines from §10, and a **representative sample** of
~2,000 tokens: body under ~1,500 words sent whole; otherwise first 600 words, 500 around the
midpoint, last 400, with visible `[BEGINNING]`/`[MIDDLE]`/`[END]` markers, split on word
boundaries. Batches of `deepseek.deep_batch_size` (8, replaces `score_batch_size`) through
`buffer_unordered`, bulk client, budget check per batch. Auto-includes are assessed too.
Tolerant parsing: `quality`/`fit` clamped 0–10, `category` validated against the palette (invalid →
`None`, the editor decides), `paywalled_guess`, and `facets` where every unknown enum token degrades
to `None` and `specific_topics` is capped at 3. **Every enum token in the prompt must round-trip
through the parser (test).** Persist to `article_assessments (stage='deep', score=quality, fit,
kind=facets.format, facets_json, rationale, category, paywalled_guess, ...)`; reuse per §7.3 with
`--rescore` bypass. Facets are shown to the editor, the profile rebuild and `explain`; **not** a
numeric signal.
2. **`src/curate/rank.rs`** (§12.2–§12.5): normalization (LLM scores ÷ 10; everything else mid-rank
percentile over the deep set — reuse/move the step-3 normalizer), the **utility** blend over
present signals with `[curation.ranking.weights.utility]` renormalized and learned signals
multiplied by their gate ramp first, stored 0–100; the preliminary blend stays for the eligible set.
**Diversified shortlist** (§12.5): leader clustering by embedding cosine with `cluster_threshold`
(0.85) and `per_cluster_cap` (2): sort by utility desc then article id asc; assign each article to
the first existing cluster whose *leader* has cosine ≥ threshold, else make it a new leader;
articles without embeddings are singletons; admit in order while the cluster's admitted count is
below the cap until `shortlist_keep` (60); the top `utility_protected` (10) by utility and all
auto-includes are admitted regardless and still count toward their cluster; exploration picks that
reached the deep set get up to 3 reserved slots; if short, relax to cap 3, then uncapped. Persist
`cluster_id`, `cluster_rank`, `rank_utility`, `utility`, `excluded_reason ∈ cluster_suppressed |
shortlist_cap`, stage `shortlisted`.
3. **`src/curate/editor.rs`** replaces `select.rs`: the §13 rendering in full —
`quality 8.5 · fit 7.0 · triage 8.0 — <deep rationale>`, `facets: format · depth · evidence ·
technicality · topic_group`, `matches:`, `closest rated:`, `flags: exploration | always-include |
excerpt only`, `opening:` first 60 words. The editor sees the 60-item shortlist. `assemble()`'s
`hard_max` trim and `select_without_llm` now order by **utility**, falling back to the preliminary
blend. Delete `ScoredArticle::combined_score()` and `ScoredArticle` itself; `Candidate` is the only
flow type. Move the tests from `score.rs`/`select.rs` into `assess.rs`/`editor.rs` and delete the
old files (update `curate/mod.rs`, `Curator`).
4. **Editorial inputs**: the Brief and the summaries use quality/fit where the plan says so (§14.2
input lists quality/fit). The weekly profile rebuild's rated lines now get real facets.
5. **Pipeline/report**: stage timings `assess` and `rank`; `StageCounts` gains `assessed`,
`shortlisted`, `clusters`; the §15.4 curation line
`curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected`.
`explain` prints the deep assessment (quality, fit, category, rationale, facets), utility and rank,
cluster id and what suppressed it; `--near-misses` orders by utility. `runs.config_json` gains
`DEEP_PROMPT_VERSION`. `--skip-llm`/DeepSeek down ⇒ no deep assessment, utility over present
signals (triage/interest dominate, §12.3), editor still runs on what it has (§17).
6. `config.example.toml`/README: `deep_batch_size` replaces `score_batch_size`; a stale
`score_batch_size` key fails loudly naming `deep_batch_size`.
Tests (§20 "Triage and deep parsing" deep half, "Normalization", "Clustering", "Editor" utility
parts): realistic deep fixture; malformed items do not sink a batch; unknown facet tokens → `None`;
every enum token round-trips; representative sample has the three markers and respects word
boundaries, short bodies are sent whole; cached deep rows reused / bypassed with `--rescore`; constant
signal → 0.5; ties equal; absent values do not shift others; effective weights sum to 1; a candidate
missing a signal is scored on the rest; near-duplicates share a cluster and the third is suppressed;
protected top-N survive and count; the bridge case (A~C, B~C, A≁B, utility A>B>C) yields two clusters;
articles without embeddings are never suppressed; `hard_max` trims by utility; `select_without_llm`
orders by utility; the mocked full pipeline publishes with DeepSeek down and writes `shortlisted`
rows with cluster ids.
+62
View File
@@ -0,0 +1,62 @@
---
# YOUR STEP: 6 — Paper telemetry, stats, lock (plan §21 step 6)
Steps 1–5 have landed: the full v2 ranking pipeline (hygiene → embeddings → signals → triage →
admission → deep assessment → utility → cluster-capped shortlist → Claude editor → editorial),
`candidate_runs` telemetry, `explain`, `features`, `ratings`. Read `git log -10`, then
`src/pipeline.rs`, `src/report.rs`, `src/main.rs`, `src/curate/telemetry.rs`, `src/epub/chapters.rs`
and `src/epub/templates/`, `src/types.rs` (`Colophon`), `src/db.rs`, README, `config.example.toml`.
Scope (plan §5 lock, §15.1 Behind the paper + In-this-issue, §15.3 `stats`, §15.4 report/log block,
§16 lock coverage, §19 startup logging, README/config docs):
1. **Behind the paper** (§15.1): a new short chapter after the World Briefing and before the colophon
(`src/epub/templates/behind.xhtml`, `render_behind_the_paper`) with exactly the content shape in
the plan: the considered/eligible/triaged/read-closely/shortlisted/selected line; the
`Admitted via:` line; the `Learned signals:` line (rated-with-embeddings count and knn percentage,
feed affinity state); `Near misses` — the 10 highest-utility non-selected articles as
`<title> — <feed> · quality X · fit Y · <stage/reason>`; the `Models:` line with triage/assessment,
editor/summaries and embeddings models; cost and generation time. Both editions (X4 gets the same
text, no links). The data comes from the run's `RunReport`/`StageCounts` and the run's
`candidate_runs` rows (add a `BehindThePaper` struct to `types.rs` filled by the pipeline; keep
templates pure). Chapter ids stable (`behind`).
2. **In this issue** page: verify each entry shows the `why` line under the summary (step 2 was asked
to do this; complete it if missing). The colophon keeps its fields and shows per-provider cost
lines and models (verify; complete if missing).
3. **`stats`** (§15.3): `daily-epub stats [--days 14]` printing issues, articles published, explicit
ratings by label, ratings per issue, up/down ratio per admitting retriever (`admitted_by[0]` of
rated picks, from `candidate_runs` joined to `rating_events` via the latest explicit event),
exploration yield (exploration picks selected / admitted, and how many were rated positively),
mean issue size, cost per day per provider (from `runs.provider_costs_json`), mean generation time.
Plain text, one fact per line, no tables wider than 80 columns.
4. **Run report and log block** (§15.4): make sure `StageCounts` has every field listed —
`eligible`, `embedded`, `triaged`, `admitted`, `admitted_by` (map), `assessed`, `shortlisted`,
`clusters`, `exploration_admitted`, `exploration_selected`, `verdicts_in_prompt`,
`rated_with_embeddings`, per-provider usage — and that stage timings include `embed`, `signals`,
`triage`, `admit`, `assess`, `rank`, `editor`, `summaries`, `brief`. Emit the four-line info block
from §15.4 once per run (`curation:`, `admission:`, `preference:`, `providers:`). `print_report`
in `main.rs` prints the same lines.
5. **`src/lock.rs`** (§5): `flock(LOCK_EX | LOCK_NB)` on `<database_path>.lock`, taken in `main`
for `generate`, `profile rebuild`, `features backfill`, and `backfill-social`; a second invocation
exits with `generate is already running` (name the command that holds it if cheap, else the
generic message). `serve`, `explain`, `stats`, `ratings`, `db migrate`, `features prune` do not
take it. Use `libc`/`rustix` only if already in the dependency tree (check `Cargo.lock`); otherwise
`std::fs::File` + the `fs2`-free approach via `libc::flock` is acceptable to add as a tiny dep —
say which in the report. Twenty lines, no table, no TTL.
6. **Docs**: README rewritten where it describes the pipeline, feedback links, costs, prerequisites
(Anthropic and Voyage keys, dashboard spend limits as the real backstop, the server-side fallback
note), the CLI (`ratings`, `explain`, `stats`, `features`, `--rescore`, `--skip-embeddings`,
`--max-articles` as ceiling), and `data/profile.md`. `config.example.toml` carries every key from
plan §19 with comments. Startup logs resolved models and per-provider enabled state (verify).
7. `systemd/daily-epub-generate.service`: no change needed unless an env var name changed; note the
two new env vars in the README's systemd section.
Tests (§20 "Lock", "Pipeline" behind-the-paper, stats): two processes/one wins (spawn the binary or
use two `File` handles from separate threads with `try_lock` semantics — no sleeping longer than a
second), a killed holder frees the lock, `serve` does not take it (parse-level or by checking no lock
file is created); the behind-the-paper chapter renders with the counts and near misses on the mocked
pipeline run and is parseable XHTML in both editions (extend `tests/m4_epub.rs`); `stats` runs on a
temp DB with a couple of runs, issues and rating events and prints every line; the info block appears
in the log (capture with `tracing` test subscriber or assert on the formatted strings).
+29
View File
@@ -0,0 +1,29 @@
---
# YOUR STEP: 7 — Cleanup (plan §21 step 7)
Steps 1–6 have landed. This step removes what the plan retired and updates the implementation
notes. Read `git log -12` and skim every file under `src/` for leftovers.
1. **Dead code**: anything only the old prefilter/scores/select/feed-prior paths used — unused
functions, types (`LlmScore` if nothing reads it, `ScoredArticle`, `FeedPrior`, `Rating`),
`kv` keys no longer read, config aliases and `#[serde(alias)]`s introduced as transition aids,
`// filled in by step N` comments, stale doc comments that still describe 👍/👎, "From the Editor",
section intros, prefilter gating, or `async-openai`. `cargo clippy --all-targets -- -W dead_code`
must be clean; do not add `#[allow(dead_code)]`.
2. **Prune paths** (§7.1, §7.4): confirm `features prune` covers `article_embeddings`
(`embedding_retention_days`) and `candidate_runs` (`telemetry_retention_days`); add
`article_assessments` older than `telemetry_retention_days` to it. Make `generate` call the prune
once per run after publishing (best effort, logged).
3. **Docs**: update `docs/plans/2026-08-15-implementation-notes.md`: fix item 5 (hand-rolled
`reqwest` client, not `async-openai`), add the new providers (Anthropic Messages API facts from
plan §4.2, Voyage facts from §4.3, each with "verified 2026-09-02"), the new tables, the lock, the
budget-day rule, and a short "curation v2" section pointing at
`docs/plans/2026-09-02-personalized-curation-v2.md`. This is the one step allowed to edit `docs/`.
Do not edit the plan itself.
4. **Consistency pass**: `config.example.toml` matches `Config::default()` key for key (write a test
that loads the example and compares every `[curation.*]`, `[anthropic]`, `[voyage]`, `[editorial]`
value to the defaults, since the defaults are the plan's numbers); README CLI section matches
`daily-epub --help` output for every subcommand (spot check by running the binary).
5. Run `cargo fmt`, `cargo clippy --all-targets`, `cargo test`; report the summary lines.
+283
View File
@@ -0,0 +1,283 @@
# Runbook — upgrading the server from v1 to curation v2
**Written:** 2026-09-02, for the `curation-v2` branch (all seven plan steps plus the provider
registry). Every command below is meant to be run on the server, as `root` via `sudo`, unless it
says otherwise. Expect the whole thing to take about an hour, most of it waiting on the embedding
backfill and one dry run.
What changes for the operator, in one paragraph: the binary is replaced; the SQLite schema gains
tables and drops `ratings`, `feed_priors` and `scores` (the migration copies your ratings first);
`config.toml` loses a few keys and gains the `[llm]` / `[providers.*]` registry plus a
`profile_path`; the env file gains two API keys and renames the DeepSeek one; a hand-maintained
`profile.md` is installed next to the OPML; the systemd units are unchanged.
## 0. Before touching the server
On the dev box:
```sh
git checkout curation-v2
cargo test # expect green
cargo build --release # or build on the server, as you do today
```
The crate uses `edition = "2024"` and let-chains, so the server's toolchain must be current
(`rustup update stable`). Read `data/profile.md` once; it is the reader profile you will be
editing by hand from now on, and the first thing to tune when the paper feels off.
## 1. Freeze the timer, back everything up
```sh
sudo systemctl stop daily-epub-generate.timer daily-epub-generate.service
sudo systemctl stop daily-epub.service # the server also runs migrations on start
sudo install -d -m0750 -o daily-epub -g daily-epub /var/lib/daily-epub/backup
sudo -u daily-epub sqlite3 /var/lib/daily-epub/daily-epub.db \
".backup '/var/lib/daily-epub/backup/daily-epub-pre-v2-$(date +%F).db'"
sudo cp -a /etc/daily-epub/config.toml /etc/daily-epub/config.toml.v1
sudo cp -a /etc/daily-epub/env /etc/daily-epub/env.v1
sudo cp -a /usr/local/bin/daily-epub /usr/local/bin/daily-epub.v1
```
`.backup` is the safe way to copy a WAL-mode database; a plain `cp` while anything is open can
miss the WAL. The migration is one-way (`ratings` is dropped after being copied), so this backup
is the rollback path.
## 2. Install the new binary
```sh
sudo install -m0755 target/release/daily-epub /usr/local/bin/
daily-epub --version
daily-epub --help # confirm `config`, `stats`, `explain`, `ratings`, `features` exist
```
Do **not** start any service yet: every subcommand except `config check` opens the database and
applies pending migrations, and you want the config right first.
The units in `systemd/` did not change. If you want to be sure your installed copies match:
```sh
diff systemd/daily-epub-generate.service /etc/systemd/system/daily-epub-generate.service
diff systemd/daily-epub.service /etc/systemd/system/daily-epub.service
```
## 3. Edit `config.toml` in place
Everything you do not mention keeps its documented default, so the edit is small. Open it with
`sudo -e /etc/daily-epub/config.toml` and apply this checklist.
**Remove** (each one now fails loudly at startup, naming its replacement):
| Old key | Why |
|---|---|
| `prefilter_keep = …` (top level) | replaced by `curation.ranking.deep_keep` (default 120) |
| `max_daily_usd = …` (top level) | now per provider: `providers.deepseek.max_daily_usd` |
| the whole `[deepseek]` table | becomes `[providers.deepseek]` + `[llm]` (see below) |
| any `[anthropic]` table (only if you added one from an interim build) | becomes `[providers.anthropic]` |
**Add** near the top, next to `interests_opml`:
```toml
profile_path = "/var/lib/daily-epub/data/profile.md"
```
Use an absolute path. The default is `data/profile.md` *relative to the working directory*, which
under the unit is `/var/lib/daily-epub`, so the default would resolve to the same place, but an
explicit path survives running one-off commands from another directory. Point
`interests_opml` at an absolute path too if it is still relative.
**Add** the LLM registry. Carry over the `base_url`, `model` and `price_*` values from your old
`[deepseek]` table if you had changed them; the values shown are the defaults.
```toml
[llm]
bulk = "deepseek" # triage, deep assessment, and the fallback for every editor call
editor = "anthropic" # lineup, summaries, the Brief, the weekly profile rebuild
triage_batch_size = 25
deep_batch_size = 8
score_temperature = 0.3
editorial_temperature = 0.8
[providers.deepseek]
kind = "openai"
base_url = "https://api.deepseek.com/v1"
model = "deepseek-v4-flash"
max_daily_usd = 2.0
max_concurrent_requests = 4
price_input_per_mtok = 0.14
price_cache_read_per_mtok = 0.0028
price_cache_write_per_mtok = 0.0
price_output_per_mtok = 0.28
[providers.anthropic]
kind = "anthropic"
base_url = "https://api.anthropic.com"
model = "claude-opus-5"
effort = "high"
max_daily_usd = 3.0
max_concurrent_requests = 4
price_input_per_mtok = 5.0
price_cache_read_per_mtok = 0.5
price_cache_write_per_mtok = 6.25
price_output_per_mtok = 25.0
```
Optional, for the Gemini comparison (section 9):
```toml
[providers.gemini]
kind = "openai" # Gemini's OpenAI-compatible endpoint
base_url = "https://generativelanguage.googleapis.com/v1beta/openai"
model = "gemini-3.8-flash"
effort = "high"
max_daily_usd = 3.0
max_concurrent_requests = 4
price_input_per_mtok = 0.75 # $1.50 from 2027-01-01
price_cache_read_per_mtok = 0.075 # $0.15 from 2027-01-01
price_cache_write_per_mtok = 0.0
price_output_per_mtok = 3.75 # includes thinking tokens; $7.50 from 2027-01-01
```
**Leave alone** `[miniflux]`, `[server]`, `[publish]`, `[xtc]`, `[world]`, `[curation]
sections`, `always_include_feeds`, `blocked_domains`. `[voyage]`, `[editorial]`,
`[curation.feedback]` and `[curation.ranking]` all have sensible defaults; copy a section from
`config.example.toml` only when you want to change a value in it.
## 4. Edit the env file
```sh
sudo -e /etc/daily-epub/env
```
| Variable | Action |
|---|---|
| `DAILY_EPUB_DEEPSEEK__API_KEY` | **rename** to `DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY` (the old name is rejected at startup so it cannot silently disable the bulk model) |
| `DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY` | add |
| `DAILY_EPUB_VOYAGE__API_KEY` | add |
| `DAILY_EPUB_PROVIDERS__GEMINI__API_KEY` | add only if you configured `[providers.gemini]` |
| `DAILY_EPUB_MINIFLUX__API_KEY`, `DAILY_EPUB_SERVER__HMAC_SECRET` | unchanged |
Keep it `0600 daily-epub:daily-epub`. Then set hard spend limits in the DeepSeek, Anthropic and
Voyage dashboards: the in-app `max_daily_usd` meters are runaway guards, not accounting.
## 5. Install the reader profile
```sh
sudo install -d -m0750 -o daily-epub -g daily-epub /var/lib/daily-epub/data
sudo install -m0640 -o daily-epub -g daily-epub data/profile.md /var/lib/daily-epub/data/profile.md
# if the OPML is not already there:
sudo install -m0640 -o daily-epub -g daily-epub data/scour-interests.opml /var/lib/daily-epub/data/
```
If the file is missing the run does not fail; it logs a warning and uses the OPML interests only,
which is a much worse prompt. `config check` in the next step tells you whether it was found.
## 6. Check the config as the service user
The units run as `daily-epub` with the env file loaded, so check the same way. This helper runs
one command in that identity, with the env file and the working directory the unit uses, and no
other hardening:
```sh
de() { sudo systemd-run --quiet --wait --pty --collect \
--uid=daily-epub --gid=daily-epub \
-p WorkingDirectory=/var/lib/daily-epub -p EnvironmentFile=/etc/daily-epub/env \
/usr/local/bin/daily-epub --config /etc/daily-epub/config.toml "$@"; }
de config check
```
Expected: every line is a fact, no line starts with `!`. Fix anything marked `MISSING` (a key
name, a path) before continuing. A stale key in the TOML is reported as an error naming its
replacement; go back to section 3.
## 7. Migrate the schema
```sh
de db migrate
sudo -u daily-epub sqlite3 /var/lib/daily-epub/daily-epub.db '.tables'
# expect rating_events, article_embeddings, interest_embeddings, article_assessments,
# candidate_runs; no ratings, feed_priors or scores
sudo -u daily-epub sqlite3 /var/lib/daily-epub/daily-epub.db \
"select label, count(*) from rating_events group by label;"
# your old votes: up → loved, down → not_for_me, source = 'migration'
```
## 8. Warm the embedding cache, then a dry run
```sh
de features backfill --rated-only # the learned set; prints a token estimate first
de features backfill --days 30 # recent articles, so day one is not all cache misses
de generate --dry-run --out /var/lib/daily-epub/out-check
```
The dry run makes real DeepSeek, Claude and Voyage calls but publishes nothing and writes no
`issues` row. Read the printed lineup and the four report lines (`curation:`, `admission:`,
`preference:`, `providers:`); the cost should be well under $1. Then read the EPUB it wrote
(Calibre or KOReader): The Brief, the `Why it's here` line under each headline, and the new
Behind-the-paper chapter before the colophon. Finally:
```sh
de explain --date "$(date +%F)" --near-misses
```
Remove `/var/lib/daily-epub/out-check` when done.
## 9. Go live
```sh
sudo systemctl daemon-reload
sudo systemctl start daily-epub.service
sudo systemctl enable --now daily-epub-generate.timer
sudo systemctl list-timers daily-epub-generate.timer
```
If you want today's paper regenerated by the new pipeline now rather than tomorrow at 05:30:
`sudo systemctl start daily-epub-generate`. A same-date rerun replaces today's issue and is a
new run id in telemetry.
Afterwards:
```sh
journalctl -u daily-epub-generate -n 60 --no-pager # the four-line info block near the end
de stats --days 14
```
## 10. Comparing editors (Claude Opus 5 vs Gemini 3.8 Flash)
With `[providers.gemini]` and its key in place, an A/B needs no config edit: environment variables
override the TOML, so
```sh
sudo systemd-run --quiet --wait --pty --collect --uid=daily-epub --gid=daily-epub \
-p WorkingDirectory=/var/lib/daily-epub -p EnvironmentFile=/etc/daily-epub/env \
-E DAILY_EPUB_LLM__EDITOR=gemini \
/usr/local/bin/daily-epub --config /etc/daily-epub/config.toml \
generate --dry-run --date "$(date +%F)" --out /var/lib/daily-epub/out-gemini
```
produces the same date's paper with Gemini as the editor. Triage and deep assessments are cached
per article for three days, so the second run re-requests only articles that have no row yet:
ones ingested since the first run, and ones whose earlier batch was rejected by the provider's
content filter (those are bisected, retried once on the editor provider, and then recorded as
`provider_rejected` so they are not retried daily). The `triage:` and `assess:` log lines say how
many were reused versus requested. The bulk-side cost of the second run is therefore small; the
editor, summaries and the Brief are the real spend. Compare the two
lineups, the `why` lines and the Brief side by side, and the `providers:` cost line. To switch
for good, set `editor = "gemini"` in `[llm]` (and `summary_model` stays `editor`, so summaries
move with it). The same trick works for the bulk role: `DAILY_EPUB_LLM__BULK=gemini`.
## Rollback
```sh
sudo systemctl stop daily-epub-generate.timer daily-epub.service
sudo install -m0755 /usr/local/bin/daily-epub.v1 /usr/local/bin/daily-epub
sudo cp -a /etc/daily-epub/config.toml.v1 /etc/daily-epub/config.toml
sudo cp -a /etc/daily-epub/env.v1 /etc/daily-epub/env
sudo -u daily-epub cp /var/lib/daily-epub/backup/daily-epub-pre-v2-<date>.db /var/lib/daily-epub/daily-epub.db
sudo rm -f /var/lib/daily-epub/daily-epub.db-wal /var/lib/daily-epub/daily-epub.db-shm
sudo systemctl start daily-epub.service daily-epub-generate.timer
```
The database restore is required, not optional: the v1 binary expects `ratings`, `feed_priors`
and `scores`, which v2's migrations drop.
+2 -1
View File
@@ -369,6 +369,7 @@ fn pick_for(target: &Target, content_html: String) -> Pick {
section: target.issue.clone(),
position: 0,
is_lead: false,
why: None,
summary: None,
llm: None,
discussion: None,
@@ -412,11 +413,11 @@ fn write_audit_epub(out_dir: &Path, picks: &[Pick], assets: &[ImageAsset]) {
Articles are re-extracted live; editorial, discussions and the \
world briefing are absent by design.</p>"
.into(),
section_intros: Default::default(),
summaries: Default::default(),
},
world_briefing: None,
colophon: Colophon::default(),
behind: Default::default(),
};
let cfg = daily_epub::config::Config::default();
match daily_epub::epub::build_edition_with_images(
+1
View File
@@ -0,0 +1 @@
DROP TABLE scores;
+1292 -45
View File
File diff suppressed because it is too large. Load diff
+542
View File
@@ -0,0 +1,542 @@
//! Hygiene and union admission into the deep set (plan §8.1, §11).
use std::collections::{BTreeMap, HashSet};
use jiff::Timestamp;
use jiff::civil::Date;
use sha2::{Digest, Sha256};
use sqlx::Row as _;
use super::{prefilter, telemetry};
use crate::config::{CurationConfig, RankingConfig};
use crate::db::{Db, fmt_ts};
use crate::types::{Article, ArticleId, Candidate};
/// Run hygiene before embeddings and write thin telemetry rows for exclusions.
pub async fn hygiene(
db: &Db,
run_id: i64,
articles: Vec<Article>,
date: Date,
config: &CurationConfig,
now: Timestamp,
) -> anyhow::Result<Vec<Candidate>> {
let published = db
.previously_published_ids_before(date)
.await?
.into_iter()
.collect::<HashSet<_>>();
let since = now - jiff::Span::new().hours(config.recent_rejection_days.max(0) * 24);
let rows = sqlx::query(
"SELECT DISTINCT article_id FROM article_assessments
WHERE stage IN ('triage', 'deep') AND score IS NOT NULL AND score < ?
AND assessed_at >= ?",
)
.bind(config.recent_rejection_floor)
.bind(fmt_ts(since))
.fetch_all(db.pool())
.await?;
let rejected = rows
.iter()
.map(|row| row.get::<i64, _>("article_id"))
.collect::<HashSet<_>>();
let mut eligible = Vec::new();
for article in articles {
let auto_include = prefilter::is_auto_include(&article, config);
let reason = if auto_include {
None
} else if prefilter::is_blocked(&article, config) {
Some("blocked")
} else if published.contains(&article.id) {
Some("published_before")
} else if rejected.contains(&article.id) {
Some("recently_rejected")
} else {
None
};
if let Some(reason) = reason {
telemetry::thin_excluded(db, run_id, article.id, reason).await?;
} else {
eligible.push(Candidate::new(article, auto_include));
}
}
Ok(eligible)
}
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AdmissionSummary {
pub admitted: usize,
pub admitted_by: BTreeMap<String, usize>,
pub exploration_admitted: usize,
}
pub fn admit(
candidates: &mut [Candidate],
date: Date,
ranking: &RankingConfig,
) -> AdmissionSummary {
for candidate in candidates.iter_mut() {
candidate.admitted_by.clear();
candidate.exploration = false;
if candidate.excluded_reason.as_deref() != Some("not_admitted") {
candidate.excluded_reason = None;
}
}
let mut admitted = HashSet::new();
for (index, candidate) in candidates.iter_mut().enumerate() {
if candidate.excluded_reason.is_none() && candidate.auto_include {
candidate.admitted_by.push("auto_include".into());
admitted.insert(index);
}
}
let capacity = |admitted: &HashSet<usize>| ranking.deep_keep.saturating_sub(admitted.len());
let triage = ranked(candidates, |candidate| {
candidate
.assessment
.triage
.as_ref()
.filter(|triage| triage.interest >= 5.0)
.map(|triage| triage.interest)
});
let quota = ranking.quotas.triage.min(capacity(&admitted));
take_retriever(
candidates,
&mut admitted,
&triage,
ranking.quotas.triage,
quota,
"triage",
);
let interest = ranked(candidates, |candidate| {
semantic_floor(candidate, ranking)
.then_some(candidate.signals.interest)
.flatten()
});
if !interest.is_empty() {
let quota = ranking.quotas.interest.min(capacity(&admitted));
take_retriever(
candidates,
&mut admitted,
&interest,
ranking.quotas.interest,
quota,
"interest",
);
}
let knn = ranked(candidates, |candidate| {
semantic_floor(candidate, ranking)
.then_some(candidate.signals.knn.filter(|score| *score > 0.0))
.flatten()
});
if !knn.is_empty() {
let quota = ranking.quotas.knn.min(capacity(&admitted));
take_retriever(
candidates,
&mut admitted,
&knn,
ranking.quotas.knn,
quota,
"knn",
);
}
let mut by_blend = ranked(candidates, |candidate| candidate.signals.preliminary);
let band_end = ((ranking.deep_keep as f64) * 2.5).ceil() as usize;
let band_start = ranking.deep_keep.min(by_blend.len());
by_blend.truncate(band_end.min(by_blend.len()));
let mut exploration = by_blend
.into_iter()
.skip(band_start)
.filter(|index| {
let candidate = &candidates[*index];
candidate.article.word_count >= 300
&& !prefilter::looks_like_roundup(&candidate.article.title)
&& candidate
.assessment
.triage
.as_ref()
.is_some_and(|triage| triage.interest >= 4.0)
})
.collect::<Vec<_>>();
exploration.sort_by_key(|index| exploration_key(date, candidates[*index].article.id));
let quota = ranking.exploration_slots.min(capacity(&admitted));
take_retriever(
candidates,
&mut admitted,
&exploration,
ranking.exploration_slots,
quota,
"exploration",
);
for index in &admitted {
if candidates[*index]
.admitted_by
.first()
.is_some_and(|name| name == "exploration")
{
candidates[*index].exploration = true;
}
}
let blend = ranked(candidates, |candidate| candidate.signals.preliminary);
let quota = capacity(&admitted);
take_retriever(candidates, &mut admitted, &blend, quota, quota, "blend");
for (index, candidate) in candidates.iter_mut().enumerate() {
if admitted.contains(&index) {
candidate.stage = "admitted".into();
candidate.excluded_reason = None;
} else if candidate.excluded_reason.is_none() {
candidate.stage = if candidate.assessment.triage.is_some() {
"triaged".into()
} else {
"eligible".into()
};
candidate.excluded_reason = Some("not_admitted".into());
}
}
let mut summary = AdmissionSummary {
admitted: admitted.len(),
exploration_admitted: admitted
.iter()
.filter(|index| candidates[**index].exploration)
.count(),
..AdmissionSummary::default()
};
for index in admitted {
if let Some(first) = candidates[index].admitted_by.first() {
*summary.admitted_by.entry(first.clone()).or_default() += 1;
}
}
summary
}
fn semantic_floor(candidate: &Candidate, ranking: &RankingConfig) -> bool {
candidate.article.word_count >= ranking.semantic_min_words
&& !prefilter::looks_like_roundup(&candidate.article.title)
&& candidate
.assessment
.triage
.as_ref()
.is_none_or(|triage| triage.interest >= 3.0)
}
fn ranked(candidates: &[Candidate], signal: impl Fn(&Candidate) -> Option<f64>) -> Vec<usize> {
let mut values = candidates
.iter()
.enumerate()
.filter(|(_, candidate)| candidate.excluded_reason.is_none())
.filter_map(|(index, candidate)| signal(candidate).map(|value| (index, value)))
.collect::<Vec<_>>();
values.sort_by(|(left_index, left), (right_index, right)| {
right.total_cmp(left).then_with(|| {
candidates[*left_index]
.article
.id
.cmp(&candidates[*right_index].article.id)
})
});
values.into_iter().map(|(index, _)| index).collect()
}
fn take_retriever(
candidates: &mut [Candidate],
admitted: &mut HashSet<usize>,
ranked: &[usize],
would_take: usize,
admit_quota: usize,
name: &str,
) {
// Record overlap among this retriever's own top-N.
for index in ranked.iter().take(would_take) {
if admitted.contains(index)
&& !candidates[*index]
.admitted_by
.iter()
.any(|value| value == name)
{
candidates[*index].admitted_by.push(name.into());
}
}
let mut taken = 0;
for index in ranked {
if admitted.contains(index) {
continue;
}
if taken >= admit_quota {
break;
}
candidates[*index].admitted_by.push(name.into());
admitted.insert(*index);
taken += 1;
}
}
fn exploration_key(date: Date, article_id: ArticleId) -> [u8; 32] {
let mut hasher = Sha256::new();
hasher.update(date.to_string().as_bytes());
hasher.update(article_id.to_string().as_bytes());
hasher.finalize().into()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::{CurationConfig, RankingConfig, RankingQuotas};
use crate::curate::prefilter::tests::article;
use crate::types::Triage;
fn candidate(
id: i64,
words: i64,
interest: Option<f64>,
knn: Option<f64>,
triage: Option<f64>,
) -> Candidate {
let mut candidate = Candidate::new(article(id, &format!("article {id}"), words), false);
candidate.signals.interest = interest;
candidate.signals.knn = knn;
candidate.signals.preliminary = Some(id as f64);
candidate.assessment.triage = triage.map(|interest| Triage {
interest,
kind: "essay".into(),
why: "specific".into(),
model: "mock".into(),
prompt_version: 1,
assessed_at: "2026-09-02T05:30:00Z".parse().expect("timestamp"),
});
candidate
}
#[test]
fn semantic_retrievers_reject_stubs_and_honor_quotas() {
let config = RankingConfig {
deep_keep: 4,
exploration_slots: 0,
quotas: RankingQuotas {
triage: 0,
interest: 2,
knn: 1,
},
..RankingConfig::default()
};
let mut candidates = vec![
candidate(1, 60, Some(100.0), Some(100.0), None),
candidate(2, 600, Some(9.0), None, None),
candidate(3, 600, Some(8.0), None, None),
candidate(4, 600, None, Some(0.8), None),
candidate(5, 600, None, None, None),
];
let summary = admit(
&mut candidates,
"2026-09-02".parse().expect("date"),
&config,
);
assert_eq!(summary.admitted_by.get("interest"), Some(&2));
assert_eq!(summary.admitted_by.get("knn"), Some(&1));
assert!(
!candidates[0]
.admitted_by
.iter()
.any(|by| by == "interest" || by == "knn")
);
assert_eq!(summary.admitted, 4);
assert_eq!(summary.admitted_by.get("blend"), Some(&1));
}
#[test]
fn strong_interest_weak_heuristic_reaches_deep_set_and_auto_always_wins() {
let config = RankingConfig {
deep_keep: 2,
exploration_slots: 0,
quotas: RankingQuotas {
triage: 0,
interest: 1,
knn: 0,
},
..RankingConfig::default()
};
let mut candidates = vec![
candidate(1, 400, Some(9.0), None, None),
candidate(2, 100, None, None, None),
candidate(3, 4000, None, None, None),
];
candidates[0].signals.heuristic = Some(0.0);
candidates[1].auto_include = true;
let summary = admit(
&mut candidates,
"2026-09-02".parse().expect("date"),
&config,
);
assert_eq!(summary.admitted, 2);
assert_eq!(
candidates[0].admitted_by.first().map(String::as_str),
Some("interest")
);
assert_eq!(
candidates[1].admitted_by.first().map(String::as_str),
Some("auto_include")
);
}
#[test]
fn inactive_retrievers_release_slots_to_blend() {
let config = RankingConfig {
deep_keep: 3,
exploration_slots: 0,
..RankingConfig::default()
};
let mut candidates = (1..=5)
.map(|id| candidate(id, 500, None, None, None))
.collect::<Vec<_>>();
let summary = admit(
&mut candidates,
"2026-09-02".parse().expect("date"),
&config,
);
assert_eq!(summary.admitted_by.get("blend"), Some(&3));
}
#[test]
fn exploration_is_stable_for_a_date_and_rotates() {
let config = RankingConfig {
deep_keep: 4,
exploration_slots: 2,
quotas: RankingQuotas {
triage: 0,
interest: 0,
knn: 0,
},
..RankingConfig::default()
};
let base = (1..=12)
.map(|id| candidate(id, 500, None, None, Some(5.0)))
.collect::<Vec<_>>();
let mut first = base.clone();
admit(&mut first, "2026-09-02".parse().expect("date"), &config);
let ids = |items: &[Candidate]| {
items
.iter()
.filter(|candidate| candidate.exploration)
.map(|candidate| candidate.article.id)
.collect::<Vec<_>>()
};
let expected = ids(&first);
let mut again = base.clone();
admit(&mut again, "2026-09-02".parse().expect("date"), &config);
assert_eq!(ids(&again), expected);
let mut next = base;
admit(&mut next, "2026-09-03".parse().expect("date"), &config);
assert_ne!(ids(&next), expected);
}
#[tokio::test]
async fn recent_low_triage_is_excluded_but_auto_include_is_spared() {
let dir = tempfile::tempdir().expect("tempdir");
let db = Db::open_and_migrate(&dir.path().join("hygiene.db"))
.await
.expect("db");
sqlx::query(
"INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
(1, 'https://example.com/1', 'Rejected', '2026-09-02T00:00:00Z'),
(2, 'https://example.com/2', 'Auto', '2026-09-02T00:00:00Z')",
)
.execute(db.pool())
.await
.expect("articles");
sqlx::query(
"INSERT INTO article_assessments
(article_id, stage, model, prompt_version, score, assessed_at) VALUES
(1, 'triage', 'model', 1, 2.0, '2026-09-02T04:00:00Z'),
(2, 'triage', 'model', 1, 1.0, '2026-09-02T04:00:00Z')",
)
.execute(db.pool())
.await
.expect("assessments");
let run_id = db
.start_run(
"2026-09-02".parse().expect("date"),
"2026-09-02T05:30:00Z".parse().expect("timestamp"),
)
.await
.expect("run");
let config = CurationConfig {
always_include_feeds: vec!["99".into()],
..CurationConfig::default()
};
let normal = article(1, "Rejected", 500);
let mut auto = article(2, "Auto", 500);
auto.feed_id = 99;
let eligible = hygiene(
&db,
run_id,
vec![normal, auto],
"2026-09-02".parse().expect("date"),
&config,
"2026-09-02T05:30:00Z".parse().expect("timestamp"),
)
.await
.expect("hygiene");
assert_eq!(eligible.len(), 1);
assert_eq!(eligible[0].article.id, 2);
assert!(eligible[0].auto_include);
let reason: String = sqlx::query_scalar(
"SELECT excluded_reason FROM candidate_runs WHERE run_id = ? AND article_id = 1",
)
.bind(run_id)
.fetch_one(db.pool())
.await
.expect("thin row");
assert_eq!(reason, "recently_rejected");
}
#[tokio::test]
async fn provider_rejected_rows_do_not_mark_an_article_recently_rejected() {
let dir = tempfile::tempdir().expect("tempdir");
let db = Db::open_and_migrate(&dir.path().join("hygiene.db"))
.await
.expect("db");
sqlx::query(
"INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
(1, 'https://example.com/1', 'Refused', '2026-09-02T00:00:00Z')",
)
.execute(db.pool())
.await
.expect("article");
sqlx::query(
"INSERT INTO article_assessments
(article_id, stage, model, prompt_version, score, fit, kind, rationale, assessed_at)
VALUES (1, 'triage', 'model', 1, NULL, NULL, 'provider_rejected',
'deepseek: Content Exists Risk', '2026-09-02T04:00:00Z'),
(1, 'deep', 'model', 1, NULL, NULL, 'provider_rejected',
'deepseek: Content Exists Risk', '2026-09-02T04:00:00Z')",
)
.execute(db.pool())
.await
.expect("rejection rows");
let run_id = db
.start_run(
"2026-09-02".parse().expect("date"),
"2026-09-02T05:30:00Z".parse().expect("timestamp"),
)
.await
.expect("run");
let eligible = hygiene(
&db,
run_id,
vec![article(1, "Refused", 500)],
"2026-09-02".parse().expect("date"),
&CurationConfig::default(),
"2026-09-02T05:30:00Z".parse().expect("timestamp"),
)
.await
.expect("hygiene");
assert_eq!(eligible.len(), 1, "a NULL score is not a low score");
assert_eq!(eligible[0].article.id, 1);
}
}
+1324
View File
File diff suppressed because it is too large. Load diff
+839
View File
@@ -0,0 +1,839 @@
//! Bisecting batch runner shared by triage and deep assessment.
//!
//! DeepSeek's input content filter rejects a whole request when any one
//! article in the batch trips it (`400 Content Exists Risk`) and does not say
//! which. Sending the batch once and giving up cost every other article its
//! assessment and retried them all the next day. Instead, a batch that fails
//! with a non-transient error is split in half and both halves are sent
//! again, down to single articles. A single article that still fails is
//! "rejected": it is retried once on the editor client when that runs on a
//! different provider, and otherwise reported so the caller can persist a
//! `provider_rejected` row and stop asking (§10, §12.1, §17).
//!
//! Transient errors are retried inside [`LlmClient::complete`] and, once
//! exhausted, leave the batch unassessed as before; a budget trip stops the
//! stage. Concurrency is the caller's `max_concurrent_requests` across the
//! original batches; the bisection inside one batch runs sequentially.
use std::collections::HashSet;
use futures::{StreamExt, stream};
use super::llm::{LlmClient, LlmError};
use crate::types::{ArticleId, Candidate};
/// Longest rejection message kept in `article_assessments.rationale`.
pub const REJECTION_MESSAGE_CHARS: usize = 200;
/// A parsed per-article item the runner can attribute to a candidate.
pub trait Assessed {
fn article_id(&self) -> ArticleId;
}
/// One stage's prompt builder, parser and clients.
pub struct BatchRunner<'a, T> {
/// The bulk client every batch goes to first.
pub llm: &'a LlmClient,
/// The editor client, tried once per rejected article when it runs on a
/// different provider than `llm` and its meter is not tripped.
pub fallback: Option<&'a LlmClient>,
pub temperature: f32,
pub build_prompt: &'a (dyn Fn(&[&Candidate]) -> String + Sync),
pub parse: &'a (dyn Fn(&str) -> Vec<T> + Sync),
}
impl<T> BatchRunner<'_, T> {
/// The editor client when it is a real alternative to the bulk one.
pub fn usable_fallback(&self) -> Option<&LlmClient> {
self.fallback
.filter(|fallback| fallback.provider() != self.llm.provider())
.filter(|fallback| fallback.meter.check_budget().is_ok())
}
/// The name of the provider rejected articles are retried on, if any.
pub fn fallback_provider(&self) -> Option<String> {
self.fallback
.filter(|fallback| fallback.provider() != self.llm.provider())
.map(|fallback| fallback.provider().to_string())
}
}
/// An item together with the model that produced it (bulk or editor).
#[derive(Debug, Clone, PartialEq)]
pub struct Scored<T> {
pub item: T,
pub model: String,
}
/// A single article both the bulk provider and the fallback refused.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Rejection {
pub id: ArticleId,
/// The bulk provider's name.
pub provider: String,
/// The bulk provider's message, at most [`REJECTION_MESSAGE_CHARS`] long.
pub message: String,
}
impl Rejection {
/// `<provider>: <message>`, the `rationale` of a `provider_rejected` row.
pub fn rationale(&self) -> String {
format!("{}: {}", self.provider, self.message)
}
}
/// What a set of batches produced.
#[derive(Debug, Clone, PartialEq)]
pub struct BatchOutcome<T> {
pub items: Vec<Scored<T>>,
/// Articles nobody would assess; the caller persists these.
pub rejections: Vec<Rejection>,
/// Single articles the bulk provider rejected, recovered or not.
pub rejected: usize,
/// Of `rejected`, those the fallback client assessed.
pub recovered: usize,
/// Requests made on the bulk client.
pub requests: usize,
/// True when the bulk budget tripped and work was left undone.
pub budget_stopped: bool,
}
impl<T> Default for BatchOutcome<T> {
fn default() -> Self {
Self {
items: Vec::new(),
rejections: Vec::new(),
rejected: 0,
recovered: 0,
requests: 0,
budget_stopped: false,
}
}
}
impl<T> BatchOutcome<T> {
fn absorb(&mut self, other: Self) {
self.items.extend(other.items);
self.rejections.extend(other.rejections);
self.rejected += other.rejected;
self.recovered += other.recovered;
self.requests += other.requests;
self.budget_stopped |= other.budget_stopped;
}
}
/// The per-stage totals behind the `triage:` / `assess:` log line and the
/// run report's `*_reused` / `*_rejected` counts.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct StageSummary {
pub stage: &'static str,
/// Articles the stage was responsible for this run.
pub pool: usize,
/// Served from cached rows within `assessment_reuse_days`.
pub reused: usize,
/// Skipped because a `provider_rejected` row is still fresh.
pub known_rejected: usize,
/// Sent to the bulk provider.
pub requested: usize,
/// Original batches before any bisection.
pub batches: usize,
/// Assessments written this run (bulk and editor).
pub applied: usize,
/// Single articles the bulk provider rejected this run.
pub rejected: usize,
/// Of `rejected`, those the editor client assessed instead.
pub recovered: usize,
pub fallback_provider: Option<String>,
}
impl StageSummary {
/// Articles carrying this stage's assessment after the run.
pub fn assessed(&self) -> usize {
self.reused + self.applied
}
/// Articles without an assessment because a provider rejected them:
/// this run's unrecovered rejections plus the cached ones.
pub fn rejected_total(&self) -> usize {
self.known_rejected + self.rejected.saturating_sub(self.recovered)
}
/// `triage: 398 in pool · 210 reused · 188 requested in 8 batches · 3 rejected (2 recovered on gemini)`
pub fn info_line(&self) -> String {
let mut line = format!(
"{}: {} in pool · {} reused · {} requested in {} batches · {} rejected",
self.stage, self.pool, self.reused, self.requested, self.batches, self.rejected
);
if self.recovered > 0 {
line.push_str(&format!(
" ({} recovered on {})",
self.recovered,
self.fallback_provider.as_deref().unwrap_or("editor")
));
}
if self.known_rejected > 0 {
line.push_str(&format!(" · {} known rejected", self.known_rejected));
}
line
}
}
/// Errors the provider will keep returning for the same input.
fn is_rejection(error: &LlmError) -> bool {
matches!(
error,
LlmError::Api { .. } | LlmError::Refusal { .. } | LlmError::EmptyResponse { .. }
)
}
/// The provider's own words, without the `<provider> request failed:` prefix.
fn rejection_message(error: &LlmError) -> String {
match error {
LlmError::Api { message, .. } => message.clone(),
LlmError::Refusal { .. } => "returned a refusal".into(),
LlmError::EmptyResponse { .. } => "returned an empty completion".into(),
other => other.to_string(),
}
}
fn truncate_chars(text: &str, max_chars: usize) -> String {
let collapsed = text.split_whitespace().collect::<Vec<_>>().join(" ");
if collapsed.chars().count() <= max_chars {
return collapsed;
}
let mut cut = collapsed
.chars()
.take(max_chars.saturating_sub(1))
.collect::<String>();
cut.push('…');
cut
}
enum Failure {
Rejected(String),
NoItems,
}
struct Work<'c> {
batch: Vec<&'c Candidate>,
/// A batch that parses to zero items is split once; its halves are not.
zero_split: bool,
}
/// Run every batch on the bulk client, `max_concurrent` in flight.
pub async fn run_batches<T: Assessed>(
runner: &BatchRunner<'_, T>,
batches: Vec<Vec<&Candidate>>,
max_concurrent: usize,
) -> BatchOutcome<T> {
let outcomes = stream::iter(batches)
.map(|batch| run_one(runner, batch))
.buffer_unordered(max_concurrent.max(1))
.collect::<Vec<_>>()
.await;
let mut merged = BatchOutcome::default();
for outcome in outcomes {
merged.absorb(outcome);
}
merged
}
/// One original batch, bisected as far as it needs to be.
async fn run_one<T: Assessed>(
runner: &BatchRunner<'_, T>,
batch: Vec<&Candidate>,
) -> BatchOutcome<T> {
let original = batch.len();
let provider = runner.llm.provider();
let mut out = BatchOutcome::default();
let mut warned = false;
// Explicit LIFO stack instead of recursion: the left half is pushed last
// so it runs first, which keeps the request order predictable.
let mut stack = vec![Work {
batch,
zero_split: true,
}];
while let Some(work) = stack.pop() {
if let Err(error) = runner.llm.meter.check_budget() {
tracing::warn!(%error, "bulk budget tripped; skipping the rest of the batch");
out.budget_stopped = true;
return out;
}
let size = work.batch.len();
let prompt = (runner.build_prompt)(&work.batch);
let allowed = work
.batch
.iter()
.map(|candidate| candidate.article.id)
.collect::<HashSet<_>>();
out.requests += 1;
let failure = match runner.llm.complete(&prompt, runner.temperature, true).await {
Ok(raw) => {
let items = (runner.parse)(&raw)
.into_iter()
.filter(|item| allowed.contains(&item.article_id()))
.collect::<Vec<_>>();
if items.is_empty() {
Failure::NoItems
} else {
out.items.extend(items.into_iter().map(|item| Scored {
item,
model: runner.llm.model.clone(),
}));
continue;
}
}
Err(LlmError::BudgetExceeded { spent, limit }) => {
tracing::warn!(
spent,
limit,
"bulk budget tripped; skipping the rest of the batch"
);
out.budget_stopped = true;
return out;
}
Err(error) if is_rejection(&error) => Failure::Rejected(rejection_message(&error)),
Err(error) => {
tracing::warn!(%error, size, "batch failed; its articles remain unassessed");
continue;
}
};
if size == 1 {
let candidate = work.batch[0];
let message = match failure {
Failure::Rejected(message) => message,
Failure::NoItems => "returned no assessment for the article".into(),
};
resolve_single(runner, candidate, &prompt, message, &mut out).await;
continue;
}
match failure {
Failure::Rejected(message) => {
if !warned {
tracing::warn!(
provider,
message = %truncate_chars(&message, REJECTION_MESSAGE_CHARS),
"batch of {original} rejected by {provider}; bisecting"
);
warned = true;
}
push_halves(&mut stack, work.batch, work.zero_split);
}
Failure::NoItems if work.zero_split => {
tracing::warn!(provider, size, "batch parsed to zero items; bisecting once");
push_halves(&mut stack, work.batch, false);
}
Failure::NoItems => {
tracing::warn!(
provider,
size,
"half batch parsed to zero items again; its articles remain unassessed"
);
}
}
}
out
}
fn push_halves<'c>(stack: &mut Vec<Work<'c>>, batch: Vec<&'c Candidate>, zero_split: bool) {
let mut left = batch;
let right = left.split_off(left.len() / 2);
stack.push(Work {
batch: right,
zero_split,
});
stack.push(Work {
batch: left,
zero_split,
});
}
/// A single article the bulk provider would not assess: try the editor
/// client once, else report it as rejected.
async fn resolve_single<T: Assessed>(
runner: &BatchRunner<'_, T>,
candidate: &Candidate,
prompt: &str,
message: String,
out: &mut BatchOutcome<T>,
) {
let id = candidate.article.id;
let provider = runner.llm.provider();
out.rejected += 1;
if let Some(fallback) = runner.usable_fallback() {
let fallback_provider = fallback.provider();
match fallback.complete(prompt, runner.temperature, true).await {
Ok(raw) => {
if let Some(item) = (runner.parse)(&raw)
.into_iter()
.find(|item| item.article_id() == id)
{
tracing::info!(
article_id = id,
provider = fallback_provider,
"article rejected by {provider}; assessed on {fallback_provider} instead"
);
out.items.push(Scored {
item,
model: fallback.model.clone(),
});
out.recovered += 1;
return;
}
tracing::warn!(
article_id = id,
provider = fallback_provider,
"fallback returned no assessment for the rejected article"
);
}
Err(error) => {
tracing::warn!(
article_id = id,
provider = fallback_provider,
%error,
"fallback failed for the rejected article"
);
}
}
}
let message = truncate_chars(&message, REJECTION_MESSAGE_CHARS);
tracing::warn!(
article_id = id,
provider,
%message,
"article rejected by the provider; recorded so it is not retried"
);
out.rejections.push(Rejection {
id,
provider: provider.to_string(),
message,
});
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
use crate::config::ProviderConfig;
use crate::curate::llm::{
ChatBackend, ChatCompletion, ChatRequest, MockBackend, PriceTable, UsageMeter,
};
use crate::curate::prefilter::tests::article;
use crate::curate::triage::{TriageItem, build_batch_prompt, parse_triage_response};
use crate::http::RetryPolicy;
use crate::types::TokenUsage;
use std::future::Future;
use std::pin::Pin;
use std::sync::{Arc, Mutex};
use std::time::Duration;
/// A backend that answers for every id it finds in the prompt unless the
/// prompt mentions a forbidden title, in which case it rejects the whole
/// request the way DeepSeek's content filter does.
#[derive(Debug)]
pub(crate) struct FilterBackend {
forbidden: Vec<String>,
/// Prompts answered with `{"articles": []}` (by request index).
empty_on: Vec<usize>,
answer: fn(&[i64]) -> String,
seen: Mutex<Vec<String>>,
}
impl FilterBackend {
/// Answers in the triage shape.
pub(crate) fn new(forbidden: &[&str], empty_on: &[usize]) -> Arc<Self> {
Self::with_answer(forbidden, empty_on, answer_for)
}
/// Answers in the deep-assessment shape.
pub(crate) fn deep(forbidden: &[&str]) -> Arc<Self> {
Self::with_answer(forbidden, &[], deep_answer_for)
}
fn with_answer(
forbidden: &[&str],
empty_on: &[usize],
answer: fn(&[i64]) -> String,
) -> Arc<Self> {
Arc::new(Self {
forbidden: forbidden.iter().map(|s| s.to_string()).collect(),
empty_on: empty_on.to_vec(),
answer,
seen: Mutex::new(Vec::new()),
})
}
pub(crate) fn calls(&self) -> usize {
self.seen.lock().map(|seen| seen.len()).unwrap_or(0)
}
/// The prompt of the first request that carried only `id`.
pub(crate) fn prompts_for_single(&self, id: i64) -> String {
self.seen
.lock()
.ok()
.and_then(|seen| {
seen.iter()
.find(|prompt| ids_in(prompt) == vec![id])
.cloned()
})
.unwrap_or_default()
}
/// The ids in each request, in order.
pub(crate) fn requests(&self) -> Vec<Vec<i64>> {
self.seen
.lock()
.map(|seen| seen.iter().map(|prompt| ids_in(prompt)).collect())
.unwrap_or_default()
}
}
fn ids_in(prompt: &str) -> Vec<i64> {
prompt
.lines()
.filter_map(|line| line.strip_prefix("--- id: "))
.filter_map(|id| id.trim().parse().ok())
.collect()
}
/// A triage answer for `ids`.
pub(crate) fn answer_for(ids: &[i64]) -> String {
let items = ids
.iter()
.map(|id| format!(r#"{{"id":{id},"interest":6,"kind":"essay","why":"fine"}}"#))
.collect::<Vec<_>>()
.join(",");
format!(r#"{{"articles":[{items}]}}"#)
}
/// A deep-assessment answer for `ids`.
pub(crate) fn deep_answer_for(ids: &[i64]) -> String {
let items = ids
.iter()
.map(|id| {
format!(
r#"{{"id":{id},"quality":7,"fit":6,"category":"Top Stories","rationale":"fine","facets":{{"format":"analysis_essay"}}}}"#
)
})
.collect::<Vec<_>>()
.join(",");
format!(r#"{{"articles":[{items}]}}"#)
}
impl ChatBackend for FilterBackend {
fn complete<'a>(
&'a self,
req: ChatRequest,
) -> Pin<Box<dyn Future<Output = Result<ChatCompletion, LlmError>> + Send + 'a>> {
Box::pin(async move {
let index = self.calls();
if let Ok(mut seen) = self.seen.lock() {
seen.push(req.user.clone());
}
if self.forbidden.iter().any(|title| req.user.contains(title)) {
return Err(LlmError::api(
"deepseek",
"400 Bad Request: {\"error\":{\"message\":\"Content Exists Risk\"}}",
));
}
let content = if self.empty_on.contains(&index) {
r#"{"articles":[]}"#.to_string()
} else {
(self.answer)(&ids_in(&req.user))
};
Ok(ChatCompletion {
content,
usage: TokenUsage::default(),
})
})
}
}
fn client(provider: &str, model: &str, backend: Arc<dyn ChatBackend>) -> LlmClient {
LlmClient::with_backend_options(
provider,
model,
"SYSTEM".into(),
None,
UsageMeter::with_prices(PriceTable::from(&ProviderConfig::deepseek()), 10.0),
backend,
)
}
fn candidates(n: i64) -> Vec<Candidate> {
(1..=n)
.map(|id| Candidate::new(article(id, &format!("Title number {id}"), 800), false))
.collect()
}
async fn run(
llm: &LlmClient,
fallback: Option<&LlmClient>,
candidates: &[Candidate],
batch_size: usize,
) -> BatchOutcome<TriageItem> {
let runner = BatchRunner {
llm,
fallback,
temperature: 0.3,
build_prompt: &build_batch_prompt,
parse: &parse_triage_response,
};
let all = candidates.iter().collect::<Vec<_>>();
let batches = all.chunks(batch_size).map(<[&Candidate]>::to_vec).collect();
run_batches(&runner, batches, 4).await
}
fn sorted_ids(outcome: &BatchOutcome<TriageItem>) -> Vec<i64> {
let mut ids = outcome
.items
.iter()
.map(|scored| scored.item.id)
.collect::<Vec<_>>();
ids.sort_unstable();
ids
}
#[tokio::test]
async fn one_bad_article_in_four_costs_five_requests_and_one_rejection() {
let backend = FilterBackend::new(&["Title number 3"], &[]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let candidates = candidates(4);
let outcome = run(&llm, None, &candidates, 4).await;
// [1,2,3,4] is rejected → split. [1,2] succeeds. [3,4] is rejected →
// split. [3] alone is rejected → recorded. [4] succeeds. 1 + 2 + 2 = 5.
assert_eq!(
backend.requests(),
vec![vec![1, 2, 3, 4], vec![1, 2], vec![3, 4], vec![3], vec![4]]
);
assert_eq!(outcome.requests, 5);
assert_eq!(sorted_ids(&outcome), vec![1, 2, 4]);
assert!(outcome.items.iter().all(|s| s.model == "deepseek-v4-flash"));
assert_eq!(outcome.rejected, 1);
assert_eq!(outcome.recovered, 0);
assert_eq!(outcome.rejections.len(), 1);
let rejection = &outcome.rejections[0];
assert_eq!(rejection.id, 3);
assert_eq!(rejection.provider, "deepseek");
assert!(rejection.message.contains("Content Exists Risk"));
assert!(
rejection
.rationale()
.starts_with("deepseek: 400 Bad Request"),
"{}",
rejection.rationale()
);
assert!(!outcome.budget_stopped);
}
#[tokio::test]
async fn every_article_rejected_gives_every_article_a_rejection_in_2n_minus_1_requests() {
let titles = (1..=8)
.map(|id| format!("Title number {id}"))
.collect::<Vec<_>>();
let backend =
FilterBackend::new(&titles.iter().map(String::as_str).collect::<Vec<_>>(), &[]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let candidates = candidates(8);
let outcome = run(&llm, None, &candidates, 8).await;
assert_eq!(backend.calls(), 2 * 8 - 1);
assert_eq!(outcome.requests, 15);
assert!(outcome.items.is_empty());
let mut rejected = outcome
.rejections
.iter()
.map(|rejection| rejection.id)
.collect::<Vec<_>>();
rejected.sort_unstable();
assert_eq!(rejected, (1..=8).collect::<Vec<_>>());
assert_eq!(outcome.rejected, 8);
}
#[tokio::test]
async fn a_rejected_article_is_recovered_on_an_editor_of_another_provider() {
let backend = FilterBackend::new(&["Title number 2"], &[]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let editor_backend = Arc::new(MockBackend::new());
editor_backend.push(answer_for(&[2]), TokenUsage::default());
let editor = client("gemini", "gemini-3.8-flash", editor_backend.clone());
let candidates = candidates(2);
let outcome = run(&llm, Some(&editor), &candidates, 2).await;
assert_eq!(backend.requests(), vec![vec![1, 2], vec![1], vec![2]]);
assert_eq!(editor_backend.calls(), 1);
let prompt = editor_backend.prompts()[0].clone();
assert_eq!(prompt.system.as_str(), "SYSTEM", "same system prompt");
assert!(prompt.user.contains("--- id: 2\n"), "same article prompt");
assert_eq!(sorted_ids(&outcome), vec![1, 2]);
let recovered = outcome
.items
.iter()
.find(|scored| scored.item.id == 2)
.expect("recovered");
assert_eq!(recovered.model, "gemini-3.8-flash");
assert!(outcome.rejections.is_empty(), "no rejection row");
assert_eq!((outcome.rejected, outcome.recovered), (1, 1));
// The same provider name is no alternative: no fallback attempt.
let backend = FilterBackend::new(&["Title number 2"], &[]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let same_backend = Arc::new(MockBackend::new());
same_backend.push(answer_for(&[2]), TokenUsage::default());
let same = client("deepseek", "deepseek-v4-flash", same_backend.clone());
let outcome = run(&llm, Some(&same), &candidates, 2).await;
assert_eq!(same_backend.calls(), 0);
assert_eq!(outcome.rejections.len(), 1);
assert_eq!(outcome.recovered, 0);
// A tripped editor meter is skipped too, and a failing editor still
// yields a rejection with the bulk provider's message.
let backend = FilterBackend::new(&["Title number 2"], &[]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let tripped_backend = Arc::new(MockBackend::new());
let tripped = client("gemini", "gemini-3.8-flash", tripped_backend.clone());
tripped.meter.preload_cost(100.0);
let outcome = run(&llm, Some(&tripped), &candidates, 2).await;
assert_eq!(tripped_backend.calls(), 0);
assert_eq!(outcome.rejections.len(), 1);
let backend = FilterBackend::new(&["Title number 2"], &[]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let failing_backend = Arc::new(MockBackend::new());
failing_backend.push_llm_error(LlmError::refusal("gemini"));
let failing = client("gemini", "gemini-3.8-flash", failing_backend.clone());
let outcome = run(&llm, Some(&failing), &candidates, 2).await;
assert_eq!(failing_backend.calls(), 1);
assert_eq!(outcome.rejections.len(), 1);
assert_eq!(outcome.rejections[0].provider, "deepseek");
assert!(
outcome.rejections[0]
.message
.contains("Content Exists Risk")
);
}
#[tokio::test]
async fn zero_items_bisect_once_and_a_single_zero_is_a_rejection() {
// Batch of 4 parses to zero items → its two halves are sent; both fine.
let backend = FilterBackend::new(&[], &[0]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let candidates = candidates(4);
let outcome = run(&llm, None, &candidates, 4).await;
assert_eq!(
backend.requests(),
vec![vec![1, 2, 3, 4], vec![1, 2], vec![3, 4]]
);
assert_eq!(sorted_ids(&outcome), vec![1, 2, 3, 4]);
assert!(outcome.rejections.is_empty());
// A half that parses to zero items again is not split further: its
// articles simply stay unassessed for this run.
let backend = FilterBackend::new(&[], &[0, 1]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let outcome = run(&llm, None, &candidates, 4).await;
assert_eq!(backend.calls(), 3);
assert_eq!(sorted_ids(&outcome), vec![3, 4]);
assert!(outcome.rejections.is_empty());
// A single article that parses to zero items is a rejection.
let backend = FilterBackend::new(&[], &[0]);
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
let outcome = run(&llm, None, &candidates[..1], 1).await;
assert_eq!(backend.calls(), 1);
assert!(outcome.items.is_empty());
assert_eq!(outcome.rejections.len(), 1);
assert_eq!(
outcome.rejections[0].rationale(),
"deepseek: returned no assessment for the article"
);
}
#[tokio::test]
async fn transient_and_budget_failures_are_not_bisected() {
let backend = Arc::new(MockBackend::new());
for _ in 0..3 {
backend.push_llm_error(LlmError::Transient {
provider: "deepseek".into(),
message: "503".into(),
});
}
let llm =
client("deepseek", "deepseek-v4-flash", backend.clone()).with_retry(RetryPolicy {
max_attempts: 3,
base_delay: Duration::from_millis(1),
max_delay: Duration::from_millis(2),
});
let candidates = candidates(4);
// Transient errors are retried inside the client and, once exhausted,
// leave the batch unassessed without bisecting it.
let outcome = run(&llm, None, &candidates, 4).await;
assert_eq!(backend.calls(), 3, "three attempts, one batch");
assert!(outcome.items.is_empty());
assert!(
outcome.rejections.is_empty(),
"transient is not a rejection"
);
assert_eq!(outcome.rejected, 0);
let backend = Arc::new(MockBackend::new());
let llm = client("deepseek", "deepseek-v4-flash", backend.clone());
llm.meter.preload_cost(100.0);
let outcome = run(&llm, None, &candidates, 2).await;
assert_eq!(backend.calls(), 0);
assert!(outcome.budget_stopped);
}
#[test]
fn summary_line_matches_the_documented_shape() {
let summary = StageSummary {
stage: "triage",
pool: 398,
reused: 210,
known_rejected: 0,
requested: 188,
batches: 8,
applied: 187,
rejected: 3,
recovered: 2,
fallback_provider: Some("gemini".into()),
};
assert_eq!(
summary.info_line(),
"triage: 398 in pool · 210 reused · 188 requested in 8 batches · 3 rejected (2 recovered on gemini)"
);
assert_eq!(summary.assessed(), 397);
assert_eq!(summary.rejected_total(), 1);
let quiet = StageSummary {
stage: "assess",
pool: 120,
reused: 20,
known_rejected: 2,
requested: 98,
batches: 13,
applied: 98,
..StageSummary::default()
};
assert_eq!(
quiet.info_line(),
"assess: 120 in pool · 20 reused · 98 requested in 13 batches · 0 rejected · 2 known rejected"
);
assert_eq!(quiet.rejected_total(), 2);
}
#[test]
fn rejection_messages_are_short_and_single_line() {
let long = "x".repeat(500);
let cut = truncate_chars(&long, REJECTION_MESSAGE_CHARS);
assert_eq!(cut.chars().count(), REJECTION_MESSAGE_CHARS);
assert!(cut.ends_with('…'));
assert_eq!(truncate_chars("a\n b", 10), "a b");
assert_eq!(
rejection_message(&LlmError::refusal("anthropic")),
"returned a refusal"
);
assert_eq!(
rejection_message(&LlmError::api("deepseek", "400: nope")),
"400: nope"
);
}
}
+1219
View File
File diff suppressed because it is too large. Load diff
+358 -312
View File
@@ -1,34 +1,18 @@
//! Stage C — summaries, section intros and the front page (spec §3.6).
//!
//! Voice: warm, literate, a little playful; never fabricates facts that are not
//! present in the summaries.
//!
//! Everything here is best-effort. If the cost ceiling trips mid-way (§3.6) or a
//! call fails, the affected article silently falls back to its own opening words
//! and the run continues — an issue with plain excerpts is far better than no
//! issue at all.
//! Editor-first summaries and The Brief, with per-call bulk fallback (§14).
use std::collections::BTreeMap;
use std::fmt::Write as _;
use serde::{Deserialize, Serialize};
use futures::{StreamExt, stream};
use serde::Deserialize;
use super::llm::{LlmClient, LlmError};
use super::llm::{LlmClient, LlmError, Llms};
use super::{escape_html, prompt_text, text_to_paragraphs, truncate_tokens, truncate_words};
use crate::config::{EditorialConfig, SummaryModel};
use crate::types::{ArticleId, Editorial, Lineup, Pick};
/// Article text is truncated to roughly this many tokens per summary call (§3.6).
pub const SUMMARY_INPUT_TOKEN_BUDGET: usize = 5000;
/// Target length of the "From the Editor" front page, in words (§3.6).
pub const FRONT_PAGE_WORDS: (usize, usize) = (250, 400);
/// Words of body text used when a summary has to fall back to the excerpt.
pub const FALLBACK_SUMMARY_WORDS: usize = 45;
// ---------------------------------------------------------------------------
// Prompts (reusable instructions here; per-call material in the user message)
// ---------------------------------------------------------------------------
/// Per-article summary instructions (§3.6 stage C).
pub const SUMMARY_INSTRUCTIONS: &str = "\
TASK: write the newspaper abstract for one article in today's issue.
@@ -57,66 +41,35 @@ excerpt.
Return JSON exactly: {\"summary\": \"<two or three sentences>\"}";
/// Front-page + section-intro instructions (§3.6 stage C).
pub const FRONT_PAGE_INSTRUCTIONS: &str = "\
TASK: write the front page of today's issue of The Daily EPUB.
/// The Brief instructions (§14.2).
pub const BRIEF_INSTRUCTIONS: &str = r#"TASK: write "The Brief" for today's issue — the note at the top of the paper.
You are given the whole lineup: sections, headlines, sources and the abstract \
written for each article. Everything you write must come from those abstracts — \
you have not read the articles themselves, and inventing a fact would be worse \
than saying less.
120-200 words, one or two paragraphs. It must earn its place: if a reader skipped
it, what would he miss? Name at least three of today's picks by title and say the
specific thing that makes each worth his time (the result, the argument, the scale,
the person). If there is a thread connecting several pieces, say it in one sentence;
if there is not, do not invent one. If the issue is short, say why in one clause.
Produce two things.
Do not: welcome the reader, describe the weather, summarize every section, use
"delve", "dive", "explore", "a mix of", "something for everyone", or any sentence
that could introduce any other issue. No headings. No bullet points.
1. \"from_the_editor\" — 250 to 400 words of prose addressed to the paper's one \
reader. Find the two or three threads that actually run through today's lineup \
(a shared question, an argument between two pieces, an accidental theme) and use \
them to guide the read: what to start with over coffee, what to save for the \
commute, what rewards patience. Name the lead story and say why it leads. It is \
fine — good, even — to note when a day is quiet or lopsided. Voice: warm, \
literate, lightly playful, never breathless; a real editor writing to someone \
whose taste he knows. No bullet lists, no headings, no emoji, 2–4 paragraphs \
separated by a blank line.
Return JSON exactly: {"brief": "<the text, plain prose>"}"#;
2. \"section_intros\" — for EACH section name given below, two or three \
sentences (35–60 words) introducing what is in it today. Concrete, specific to \
these articles, no filler like \"a variety of interesting stories\". Use the \
section names exactly as spelled in the lineup.
Return JSON exactly:
{\"from_the_editor\": \"<paragraphs separated by \\n\\n>\", \
\"section_intros\": {\"<section name>\": \"<2-3 sentences>\"}}";
/// The single front-page call's JSON response (§3.6).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct FrontPageResponse {
/// "From the Editor", 250–400 words.
pub from_the_editor: String,
/// Section name → 2–3 sentence intro.
#[derive(Debug, Clone, PartialEq, Deserialize)]
pub struct BriefResponse {
#[serde(default)]
pub section_intros: BTreeMap<String, String>,
pub brief: String,
}
/// The per-article summary call's JSON response.
#[derive(Debug, Clone, Default, Deserialize)]
struct SummaryResponse {
#[serde(default)]
summary: String,
}
// ---------------------------------------------------------------------------
// Per-article summaries
// ---------------------------------------------------------------------------
/// One 2–3 sentence newspaper abstract: what it argues, why it's worth reading (§3.6).
pub async fn summarize_article(
llm: &LlmClient,
title: &str,
body_html: &str,
temperature: f32,
) -> Result<String, LlmError> {
llm.meter.check_budget()?;
let body = truncate_tokens(&prompt_text(body_html), SUMMARY_INPUT_TOKEN_BUDGET);
fn summary_prompt(title: &str, body_html: &str, input_tokens: usize) -> String {
let body = truncate_tokens(&prompt_text(body_html), input_tokens);
let mut prompt = String::with_capacity(body.len() + SUMMARY_INSTRUCTIONS.len() + 256);
prompt.push_str(SUMMARY_INSTRUCTIONS);
let _ = write!(
@@ -129,184 +82,186 @@ pub async fn summarize_article(
""
},
if body.is_empty() {
"(no body text was extracted; summarize from the headline alone and say the \
full text was unavailable)"
"(no body text was extracted; summarize from the headline alone and say the full text was unavailable)"
} else {
&body
}
);
prompt
}
pub async fn summarize_article(
llm: &LlmClient,
title: &str,
body_html: &str,
input_tokens: usize,
temperature: f32,
) -> Result<String, LlmError> {
let prompt = summary_prompt(title, body_html, input_tokens);
let response: SummaryResponse = llm.complete_json(&prompt, temperature).await?;
let summary = response.summary.trim().to_string();
if summary.is_empty() {
return Err(LlmError::EmptyResponse);
return Err(LlmError::empty_response(llm.provider()));
}
Ok(summary)
}
/// Summarize every pick, returning `article_id → summary` (§3.6).
///
/// Stops early and returns what it has when the cost guardrail trips (§3.6).
pub async fn summarize_all(
llm: &LlmClient,
lineup: &Lineup,
temperature: f32,
) -> BTreeMap<ArticleId, String> {
let mut out = BTreeMap::new();
for (n, pick) in lineup.picks.iter().enumerate() {
if llm.meter.budget_exceeded() {
tracing::error!(
summarized = out.len(),
remaining = lineup.picks.len() - n,
spent_usd = llm.meter.cost_usd(),
"COST CEILING HIT during stage C — the remaining articles fall back to \
feed excerpts as summaries"
);
break;
/// `(primary, fallback)` for the summaries per `editorial.summary_model` (§14.1).
fn summary_clients(llms: &Llms, model: SummaryModel) -> (Option<&LlmClient>, Option<&LlmClient>) {
match model {
SummaryModel::Bulk => (llms.bulk.as_ref(), None),
SummaryModel::Editor => {
let primary = llms.editor_or_bulk();
let fallback = primary.and_then(|client| {
llms.bulk
.as_ref()
.filter(|bulk| bulk.provider != client.provider)
});
(primary, fallback)
}
}
}
async fn summarize_pick(
pick: &Pick,
primary: Option<&LlmClient>,
fallback: Option<&LlmClient>,
config: &EditorialConfig,
temperature: f32,
) -> Option<String> {
let primary = primary?;
match summarize_article(
llm,
primary,
&pick.article.title,
&pick.article.content_html,
config.summary_input_tokens,
temperature,
)
.await
{
Ok(summary) => {
out.insert(pick.article.id, summary);
}
Err(LlmError::BudgetExceeded { spent, limit }) => {
tracing::error!(spent, limit, "COST CEILING HIT during stage C");
break;
}
Err(e) => {
tracing::warn!(
article_id = pick.article.id,
title = %pick.article.title,
error = %e,
"summary failed; falling back to the article's own opening"
);
Ok(summary) => Some(summary),
Err(error) => {
let Some(fallback) = fallback else {
tracing::warn!(article_id = pick.article.id, %error, "summary failed; using excerpt");
return None;
};
tracing::warn!(article_id = pick.article.id, %error, "editor summary failed; retrying on bulk");
summarize_article(
fallback,
&pick.article.title,
&pick.article.content_html,
config.summary_input_tokens,
temperature,
)
.await
.map_err(|fallback_error| {
tracing::warn!(article_id = pick.article.id, %fallback_error, "bulk summary failed; using excerpt");
})
.ok()
}
}
}
tracing::info!(
summarized = out.len(),
picks = lineup.picks.len(),
"stage C summaries complete"
);
out
}
// ---------------------------------------------------------------------------
// Front page
// ---------------------------------------------------------------------------
/// The single front-page + section-intro call (§3.6).
pub async fn front_page(
llm: &LlmClient,
pub async fn summarize_all(
llms: &Llms,
lineup: &Lineup,
summaries: &BTreeMap<ArticleId, String>,
config: &EditorialConfig,
temperature: f32,
) -> Result<FrontPageResponse, LlmError> {
llm.meter.check_budget()?;
let prompt = build_front_page_prompt(lineup, summaries);
tracing::debug!(
approx_tokens = super::approx_tokens(&prompt),
"stage C front-page request"
);
let mut response: FrontPageResponse = llm.complete_json(&prompt, temperature).await?;
response.from_the_editor = response.from_the_editor.trim().to_string();
if response.from_the_editor.is_empty() {
return Err(LlmError::EmptyResponse);
}
// Keep only intros for sections that actually exist in the issue.
response
.section_intros
.retain(|name, text| lineup.section_order.contains(name) && !text.trim().is_empty());
Ok(response)
) -> BTreeMap<ArticleId, String> {
let (primary, fallback) = summary_clients(llms, config.summary_model);
// The summary provider's own `max_concurrent_requests` bounds the fan-out.
let concurrency = primary
.map(|client| client.max_concurrent_requests)
.unwrap_or(1)
.max(1);
stream::iter(lineup.picks.iter())
.map(|pick| async move {
let summary = summarize_pick(pick, primary, fallback, config, temperature).await;
(pick.article.id, summary)
})
.buffer_unordered(concurrency)
.filter_map(|(id, summary)| async move { summary.map(|summary| (id, summary)) })
.collect()
.await
}
/// Render the front-page user prompt: the whole lineup with its abstracts (§3.6).
pub fn build_front_page_prompt(lineup: &Lineup, summaries: &BTreeMap<ArticleId, String>) -> String {
pub fn build_brief_prompt(lineup: &Lineup, summaries: &BTreeMap<ArticleId, String>) -> String {
let mut prompt = String::with_capacity(4096);
prompt.push_str(FRONT_PAGE_INSTRUCTIONS);
let minutes: i64 = lineup
.picks
.iter()
.map(|p| p.article.reading_minutes())
.sum();
prompt.push_str(BRIEF_INSTRUCTIONS);
let _ = write!(
prompt,
"\n\nISSUE: {} · {} articles across {} sections · about {} minutes of reading\n\
SECTIONS, in order: {}\n\nLINEUP\n",
"\n\nISSUE: {} · {} articles\n\nLINEUP\n",
lineup.date,
lineup.picks.len(),
lineup.section_order.len(),
minutes,
lineup.section_order.join(" | ")
lineup.picks.len()
);
for section in &lineup.section_order {
let _ = write!(prompt, "\n## {section}\n");
let _ = writeln!(prompt, "\n## {section}");
for pick in lineup.section_picks(section) {
let _ = write!(prompt, "{}", render_pick(pick, summaries));
let quality = pick
.llm
.as_ref()
.map(|assessment| format!("{:.1}", assessment.quality))
.unwrap_or_else(|| "unassessed".into());
let fit = pick
.llm
.as_ref()
.map(|assessment| format!("{:.1}", assessment.fit))
.unwrap_or_else(|| "unassessed".into());
let summary = summaries
.get(&pick.article.id)
.cloned()
.unwrap_or_else(|| excerpt_summary(pick));
let _ = writeln!(
prompt,
"- {}\n feed: {}\n why: {}\n quality: {}\n fit: {}\n summary: {}",
pick.article.title.trim(),
pick.article.feed_title.trim(),
pick.why.as_deref().unwrap_or("not supplied"),
quality,
fit,
summary
);
}
}
prompt
}
fn render_pick(pick: &Pick, summaries: &BTreeMap<ArticleId, String>) -> String {
let a = &pick.article;
let mut block = String::with_capacity(400);
let _ = writeln!(
block,
"\n- {}{}",
a.title.trim(),
if pick.is_lead { " [LEAD STORY]" } else { "" }
);
let _ = writeln!(
block,
" source: {} · {} words (~{} min){}",
if a.feed_title.is_empty() {
"unknown"
} else {
a.feed_title.trim()
},
a.word_count,
a.reading_minutes(),
social_note(pick)
);
let abstract_text = summaries
.get(&a.id)
.cloned()
.unwrap_or_else(|| excerpt_summary(pick));
let _ = writeln!(block, " abstract: {abstract_text}");
block
}
fn social_note(pick: &Pick) -> String {
if pick.article.social.is_empty() {
return String::new();
pub async fn brief(
llms: &Llms,
lineup: &Lineup,
summaries: &BTreeMap<ArticleId, String>,
temperature: f32,
) -> Result<String, LlmError> {
let prompt = build_brief_prompt(lineup, summaries);
let Some(primary) = llms.editor_or_bulk() else {
return Err(LlmError::api("editorial", "no provider configured"));
};
let response = match primary
.complete_json::<BriefResponse>(&prompt, temperature)
.await
{
Ok(response) => response,
Err(error) => {
let Some(fallback) = llms
.bulk
.as_ref()
.filter(|bulk| bulk.provider != primary.provider)
else {
return Err(error);
};
tracing::warn!(%error, "brief failed on editor; retrying on bulk");
fallback
.complete_json::<BriefResponse>(&prompt, temperature)
.await?
}
let parts: Vec<String> = pick
.article
.social
.iter()
.map(|s| {
format!(
"{} {} pts/{} comments",
s.source.display_name(),
s.score,
s.num_comments
)
})
.collect();
format!(" · {}", parts.join(", "))
};
let brief = response.brief.trim().to_string();
if brief.is_empty() {
return Err(LlmError::empty_response(primary.provider()));
}
Ok(brief)
}
// ---------------------------------------------------------------------------
// Fallbacks (§3.6, notes §6)
// ---------------------------------------------------------------------------
/// The article's own opening words, used when no LLM summary exists (§3.6).
pub fn excerpt_summary(pick: &Pick) -> String {
let text = truncate_words(
&prompt_text(&pick.article.content_html),
@@ -326,17 +281,14 @@ pub fn excerpt_summary(pick: &Pick) -> String {
}
}
/// A plain, factual front page used when the model is unavailable (§3.6, notes §6).
pub fn fallback_front_page_html(lineup: &Lineup) -> String {
let minutes: i64 = lineup
.picks
.iter()
.map(|p| p.article.reading_minutes())
.map(|pick| pick.article.reading_minutes())
.sum();
let mut text = format!(
"Today's issue collects {} articles across {} sections — about {} minutes of \
reading. Editorial notes are unavailable for this issue, so the lineup speaks \
for itself.",
"Today's issue collects {} articles across {} sections — about {} minutes of reading. Editorial notes are unavailable for this issue, so the lineup speaks for itself.",
lineup.picks.len(),
lineup.section_order.len(),
minutes
@@ -346,29 +298,15 @@ pub fn fallback_front_page_html(lineup: &Lineup) -> String {
text,
"\n\nLeading today: “{}” ({}).",
lead.article.title.trim(),
if lead.article.feed_title.is_empty() {
"source unknown"
} else {
lead.article.feed_title.trim()
}
);
}
if !lineup.section_order.is_empty() {
let _ = write!(
text,
"\n\nIn this issue: {}.",
lineup.section_order.join(", ")
);
}
text_to_paragraphs(&text)
}
/// `--skip-llm` / budget-exceeded fallback: feed excerpts stand in for summaries
/// and the front page is a plain stats line (§3.6, notes §6).
pub fn fallback_editorial(lineup: &Lineup) -> Editorial {
Editorial {
front_page_html: fallback_front_page_html(lineup),
section_intros: BTreeMap::new(),
summaries: lineup
.picks
.iter()
@@ -377,54 +315,62 @@ pub fn fallback_editorial(lineup: &Lineup) -> Editorial {
}
}
// ---------------------------------------------------------------------------
// Stage driver
// ---------------------------------------------------------------------------
/// Wall-clock milliseconds of the two editorial calls, for the run report's
/// `summaries` and `brief` stage timings (§15.4).
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct EditorialTimings {
pub summaries_ms: i64,
pub brief_ms: i64,
}
/// Stage C end to end: summaries, then one front-page call, with excerpts filling
/// every gap (§3.6).
pub async fn run(llm: &LlmClient, lineup: &Lineup, temperature: f32) -> Editorial {
pub async fn run(
llms: &Llms,
lineup: &Lineup,
config: &EditorialConfig,
temperature: f32,
) -> Editorial {
run_timed(llms, lineup, config, temperature).await.0
}
/// [`run`], also reporting how long the summaries and the brief took.
pub async fn run_timed(
llms: &Llms,
lineup: &Lineup,
config: &EditorialConfig,
temperature: f32,
) -> (Editorial, EditorialTimings) {
if lineup.picks.is_empty() {
return fallback_editorial(lineup);
}
let mut summaries = summarize_all(llm, lineup, temperature).await;
let missing: Vec<&Pick> = lineup
.picks
.iter()
.filter(|p| !summaries.contains_key(&p.article.id))
.collect();
if !missing.is_empty() {
tracing::warn!(
count = missing.len(),
"using feed excerpts as summaries for articles the model did not cover"
);
for pick in missing {
summaries.insert(pick.article.id, excerpt_summary(pick));
return (fallback_editorial(lineup), EditorialTimings::default());
}
let started = std::time::Instant::now();
let mut summaries = summarize_all(llms, lineup, config, temperature).await;
for pick in &lineup.picks {
summaries
.entry(pick.article.id)
.or_insert_with(|| excerpt_summary(pick));
}
let (front_page_html, section_intros) =
match front_page(llm, lineup, &summaries, temperature).await {
Ok(response) => (
text_to_paragraphs(&response.from_the_editor),
response.section_intros,
),
Err(e) => {
tracing::error!(error = %e,
"front-page generation failed; using the plain front page");
(fallback_front_page_html(lineup), BTreeMap::new())
let summaries_ms = started.elapsed().as_millis() as i64;
let started = std::time::Instant::now();
let front_page_html = match brief(llms, lineup, &summaries, temperature).await {
Ok(text) => text_to_paragraphs(&text),
Err(error) => {
tracing::warn!(%error, "brief failed; using fallback front page");
fallback_front_page_html(lineup)
}
};
let brief_ms = started.elapsed().as_millis() as i64;
(
Editorial {
front_page_html,
section_intros,
summaries,
}
},
EditorialTimings {
summaries_ms,
brief_ms,
},
)
}
/// Escape-and-wrap helper for callers rendering a summary straight into XHTML.
pub fn summary_to_html(summary: &str) -> String {
format!("<p>{}</p>", escape_html(summary.trim()))
}
@@ -432,15 +378,15 @@ pub fn summary_to_html(summary: &str) -> String {
#[cfg(test)]
mod tests {
use super::*;
use crate::config::DeepseekConfig;
use crate::curate::llm::{MockBackend, UsageMeter};
use crate::config::ProviderConfig;
use crate::curate::llm::{ChatBackend, MockBackend, PriceTable, UsageMeter};
use crate::curate::prefilter::tests::article;
use crate::types::TokenUsage;
use std::sync::Arc;
const FRONT_PAGE_FIXTURE: &str = include_str!(concat!(
const BRIEF_FIXTURE: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/deepseek_front_page.json"
"/tests/fixtures/claude_brief.json"
));
fn pick(id: i64, title: &str, section: &str, is_lead: bool) -> Pick {
@@ -451,6 +397,7 @@ mod tests {
section: section.into(),
position: 1,
is_lead,
why: Some(format!("the {title} piece you'd argue with")),
summary: None,
llm: None,
discussion: None,
@@ -468,15 +415,40 @@ mod tests {
}
}
fn client(backend: Arc<MockBackend>, limit: f64) -> LlmClient {
LlmClient::with_backend(
"deepseek-v4-flash",
fn mock(provider: &'static str, backend: Arc<MockBackend>, limit: f64) -> LlmClient {
let prices = if provider == "anthropic" {
PriceTable::from(&ProviderConfig::anthropic())
} else {
PriceTable::from(&ProviderConfig::deepseek())
};
LlmClient::with_backend_options(
provider,
"model",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), limit),
backend,
None,
UsageMeter::with_prices(prices, limit),
backend as Arc<dyn ChatBackend>,
)
}
fn bulk_only(backend: Arc<MockBackend>, limit: f64) -> Llms {
Llms {
bulk: Some(mock("deepseek", backend, limit)),
editor: None,
}
}
fn editor_and_bulk(editor: Arc<MockBackend>, bulk: Arc<MockBackend>) -> Llms {
Llms {
bulk: Some(mock("deepseek", bulk, 2.0)),
editor: Some(mock("anthropic", editor, 3.0)),
}
}
fn config() -> EditorialConfig {
EditorialConfig::default()
}
#[tokio::test]
async fn summary_prompt_carries_headline_and_truncated_body() {
let backend = Arc::new(MockBackend::new());
@@ -484,9 +456,9 @@ mod tests {
r#"{"summary": "A team moves 40TB of relational data off Postgres and documents every rollback."}"#,
TokenUsage::default(),
);
let llm = client(Arc::clone(&backend), 2.0);
let llm = mock("deepseek", Arc::clone(&backend), 2.0);
let body = format!("<p>{}</p>", "word ".repeat(20_000));
let summary = summarize_article(&llm, "Migrating 40TB", &body, 0.8)
let summary = summarize_article(&llm, "Migrating 40TB", &body, 3_000, 0.8)
.await
.expect("summary");
assert!(summary.starts_with("A team moves 40TB"));
@@ -495,60 +467,132 @@ mod tests {
assert!(prompt.starts_with(SUMMARY_INSTRUCTIONS));
assert!(prompt.contains("HEADLINE: Migrating 40TB"));
assert!(prompt.contains("(truncated for length)"));
// ~5k tokens ≈ 20k characters of body, not the full 100k.
assert!(prompt.len() < 26_000, "prompt was {} bytes", prompt.len());
// 3k tokens ≈ 12k characters of body, not the full 100k.
assert!(prompt.len() < 16_000, "prompt was {} bytes", prompt.len());
}
#[tokio::test]
async fn front_page_parses_and_filters_unknown_sections() {
async fn the_brief_is_parsed_and_rendered() {
let backend = Arc::new(MockBackend::new());
backend.push(FRONT_PAGE_FIXTURE, TokenUsage::default());
let llm = client(Arc::clone(&backend), 2.0);
backend.push(BRIEF_FIXTURE, TokenUsage::default());
let llms = bulk_only(Arc::clone(&backend), 2.0);
let lineup = lineup();
let summaries = BTreeMap::from([
(1, "A migration story with numbers.".to_string()),
(2, "Transit data, charted.".to_string()),
]);
let response = front_page(&llm, &lineup, &summaries, 0.8)
.await
.expect("front page");
assert!(response.from_the_editor.split_whitespace().count() > 40);
assert_eq!(response.section_intros.len(), 2);
assert!(response.section_intros.contains_key("Top Stories"));
assert!(
!response.section_intros.contains_key("Niche Corner"),
"intros for absent sections are dropped"
);
let text = brief(&llms, &lineup, &summaries, 0.8).await.expect("brief");
assert!(text.split_whitespace().count() > 100);
assert!(text.contains("Migrating 40TB off Postgres"));
let prompt = &backend.prompts()[0].user;
assert!(prompt.starts_with(FRONT_PAGE_INSTRUCTIONS));
assert!(prompt.starts_with(BRIEF_INSTRUCTIONS));
assert!(prompt.contains("## Top Stories"));
assert!(prompt.contains("[LEAD STORY]"));
assert!(prompt.contains("abstract: A migration story with numbers."));
assert!(prompt.contains("## Boston & Local"));
assert!(prompt.contains("- Migrating 40TB off Postgres"));
assert!(prompt.contains("why: the Migrating 40TB off Postgres piece you'd argue with"));
assert!(prompt.contains("summary: A migration story with numbers."));
assert!(prompt.contains("quality: unassessed"));
assert!(prompt.contains("fit: unassessed"));
assert!(prompt.contains("2026-08-15"));
assert!(
!prompt.contains("section_intros"),
"section intros are gone"
);
}
#[tokio::test]
async fn full_stage_c_produces_summaries_intros_and_front_page() {
async fn full_stage_c_produces_summaries_and_the_brief() {
let backend = Arc::new(MockBackend::new());
backend.push(r#"{"summary": "First abstract."}"#, TokenUsage::default());
backend.push(r#"{"summary": "Second abstract."}"#, TokenUsage::default());
backend.push(FRONT_PAGE_FIXTURE, TokenUsage::default());
let llm = client(Arc::clone(&backend), 2.0);
backend.push(BRIEF_FIXTURE, TokenUsage::default());
let llms = bulk_only(Arc::clone(&backend), 2.0);
let editorial = run(&llm, &lineup(), 0.8).await;
assert_eq!(
backend.calls(),
3,
"one call per article plus the front page"
);
let editorial = run(&llms, &lineup(), &config(), 0.8).await;
assert_eq!(backend.calls(), 3, "one call per article plus the brief");
assert_eq!(editorial.summaries.len(), 2);
assert_eq!(editorial.summaries[&1], "First abstract.");
assert!(editorial.front_page_html.starts_with("<p>"));
assert!(editorial.front_page_html.contains("</p>"));
assert!(
editorial
.front_page_html
.contains("Migrating 40TB off Postgres")
);
assert!(!editorial.front_page_html.contains("<script"));
assert_eq!(editorial.section_intros.len(), 2);
}
#[tokio::test]
async fn summaries_run_on_the_editor_and_fall_back_per_article() {
let editor = Arc::new(MockBackend::new());
editor.push(
r#"{"summary": "Opus wrote this one."}"#,
TokenUsage::default(),
);
editor.push_llm_error(LlmError::refusal("anthropic"));
editor.push(BRIEF_FIXTURE, TokenUsage::default());
let bulk = Arc::new(MockBackend::new());
bulk.push(
r#"{"summary": "DeepSeek covered the refusal."}"#,
TokenUsage::default(),
);
let llms = editor_and_bulk(Arc::clone(&editor), Arc::clone(&bulk));
let editorial = run(&llms, &lineup(), &config(), 0.8).await;
assert_eq!(
editor.calls(),
3,
"two summaries and the brief on the editor"
);
assert_eq!(bulk.calls(), 1, "only the refused summary went to bulk");
assert_eq!(editorial.summaries[&1], "Opus wrote this one.");
assert_eq!(editorial.summaries[&2], "DeepSeek covered the refusal.");
assert_eq!(
editor.prompts()[1].user,
bulk.prompts()[0].user,
"the bulk client gets the identical summary prompt"
);
assert!(
editorial
.front_page_html
.contains("Migrating 40TB off Postgres")
);
}
#[tokio::test]
async fn the_brief_falls_back_to_bulk_with_the_same_prompt() {
let editor = Arc::new(MockBackend::new());
editor.push_error("500 opus is down");
let bulk = Arc::new(MockBackend::new());
bulk.push(BRIEF_FIXTURE, TokenUsage::default());
let llms = editor_and_bulk(Arc::clone(&editor), Arc::clone(&bulk));
let text = brief(&llms, &lineup(), &BTreeMap::new(), 0.8)
.await
.expect("bulk brief");
assert!(text.contains("Migrating 40TB off Postgres"));
assert_eq!(editor.prompts()[0].user, bulk.prompts()[0].user);
}
#[tokio::test]
async fn summary_model_bulk_skips_the_editor_for_summaries() {
let editor = Arc::new(MockBackend::new());
editor.push(BRIEF_FIXTURE, TokenUsage::default());
let bulk = Arc::new(MockBackend::new());
bulk.push(r#"{"summary": "First abstract."}"#, TokenUsage::default());
bulk.push(r#"{"summary": "Second abstract."}"#, TokenUsage::default());
let llms = editor_and_bulk(Arc::clone(&editor), Arc::clone(&bulk));
let config = EditorialConfig {
summary_model: SummaryModel::Bulk,
..EditorialConfig::default()
};
let editorial = run(&llms, &lineup(), &config, 0.8).await;
assert_eq!(bulk.calls(), 2);
assert_eq!(editor.calls(), 1, "the brief still runs on the editor");
assert_eq!(editorial.summaries[&2], "Second abstract.");
}
#[tokio::test]
@@ -560,14 +604,15 @@ mod tests {
TokenUsage {
input_tokens: 1_000_000,
cached_tokens: 0,
cache_write_tokens: 0,
output_tokens: 0,
},
);
let llm = client(Arc::clone(&backend), 0.05);
let llms = bulk_only(Arc::clone(&backend), 0.05);
let editorial = run(&llm, &lineup(), 0.8).await;
let editorial = run(&llms, &lineup(), &config(), 0.8).await;
assert_eq!(backend.calls(), 1, "no further calls after the ceiling");
assert!(llm.meter.budget_exceeded());
assert!(llms.bulk.as_ref().expect("bulk").meter.budget_exceeded());
assert_eq!(
editorial.summaries.len(),
2,
@@ -581,7 +626,6 @@ mod tests {
);
// The front page degraded to the plain version.
assert!(editorial.front_page_html.contains("2 articles"));
assert!(editorial.section_intros.is_empty());
}
#[tokio::test]
@@ -589,28 +633,30 @@ mod tests {
let backend = Arc::new(MockBackend::new());
backend.push_error("400 bad request");
backend.push(r#"{"summary": "Second abstract."}"#, TokenUsage::default());
backend.push_error("500 front page exploded");
let llm = client(Arc::clone(&backend), 2.0);
backend.push_error("500 brief exploded");
let llms = bulk_only(Arc::clone(&backend), 2.0);
let editorial = run(&llm, &lineup(), 0.8).await;
let editorial = run(&llms, &lineup(), &config(), 0.8).await;
assert_eq!(editorial.summaries.len(), 2);
assert!(editorial.summaries[&1].contains("opens with a specific"));
assert_eq!(editorial.summaries[&2], "Second abstract.");
assert!(editorial.front_page_html.contains("Leading today"));
}
#[tokio::test]
async fn no_provider_means_the_fallback_editorial() {
let editorial = run(&Llms::default(), &lineup(), &config(), 0.8).await;
assert_eq!(editorial.summaries.len(), 2);
assert!(editorial.front_page_html.contains("2 articles"));
}
#[test]
fn fallback_editorial_covers_every_pick() {
let lineup = lineup();
let editorial = fallback_editorial(&lineup);
assert_eq!(editorial.summaries.len(), lineup.picks.len());
assert!(editorial.section_intros.is_empty());
assert!(editorial.front_page_html.contains("2 articles"));
assert!(
editorial
.front_page_html
.contains("Top Stories, Boston &amp; Local")
);
assert!(editorial.front_page_html.contains("2 sections"));
assert!(editorial.front_page_html.starts_with("<p>"));
// An empty lineup is still a valid editorial.
File diff suppressed because it is too large. Load diff
+1169 -214
View File
File diff suppressed because it is too large. Load diff
+106 -107
View File
@@ -1,154 +1,153 @@
//! Curation pipeline: pre-filter → LLM scoring → selection → editorial (spec §3.5, §3.6).
//! Personalized curation: signals → triage → admission → assessment → editor.
//!
//! ```text
//! ~400 articles ─prefilter─▶ ~120 candidates ─stage A─▶ scored ─stage B─▶ lineup ─stage C─▶ editorial
//! ~400 eligible ─triage─▶ admission (120) ─assess/rank─▶ shortlist (60) ─editor─▶ editorial
//! ```
//!
//! [`Curator`] is the thin orchestration layer the `generate` pipeline calls; the
//! interesting logic lives in the stage modules. Every stage is safe to run with
//! `llm == None` (`--skip-llm`): the prefilter order stands in for selection and
//! feed excerpts stand in for summaries (notes §6).
//! no provider at all (`--skip-llm`): the cheap-signal blend stands in for selection
//! and feed excerpts stand in for summaries (notes §6). Scoring runs on the bulk
//! client; selection and editorial on the editor with per-call bulk fallback.
pub mod admit;
pub mod assess;
pub mod batch;
pub mod editor;
pub mod editorial;
pub mod embedding;
pub mod llm;
pub mod prefilter;
pub mod profile;
pub mod score;
pub mod select;
pub mod rank;
pub mod signals;
pub mod telemetry;
pub mod triage;
use jiff::civil::Date;
use jiff::{Timestamp, civil::Date};
use crate::config::Config;
use crate::db::Db;
use crate::types::{Article, Editorial, Lineup, ScoredArticle};
use crate::types::{Candidate, Editorial, Lineup};
/// Runs the three curation stages against one day's articles (§3.5, §3.6).
/// Runs the LLM curation stages against one day's articles (§12, §13, §14).
pub struct Curator {
pub config: Config,
pub db: Db,
pub llm: Option<llm::LlmClient>,
pub llms: llm::Llms,
}
impl Curator {
/// `llm == None` corresponds to `--skip-llm`: prefilter order is used for
/// selection and feed excerpts stand in for summaries (notes §6).
pub fn new(config: Config, db: Db, llm: Option<llm::LlmClient>) -> Self {
Self { config, db, llm }
/// An empty [`llm::Llms`] corresponds to `--skip-llm`: cheap-signal order is
/// used for selection and feed excerpts stand in for summaries (notes §6).
/// With only `bulk`, every editor call runs on the bulk provider (§4.2).
pub fn new(config: Config, db: Db, llms: llm::Llms) -> Self {
Self { config, db, llms }
}
/// Heuristic pre-filter: 300–500 articles → `prefilter_keep` (§3.5).
/// Deep assessment of the admitted set on the bulk client (§12.1), reusing
/// cached `article_assessments` rows within `assessment_reuse_days`.
///
/// Also persists each candidate's `prefilter_score` for the day so that a
/// re-run of the same date is idempotent (notes §12).
pub async fn prefilter(
/// A no-op under `--skip-llm`: like triage, nothing is read or written and
/// utility falls back to the present signals (§12.3, §17). When the bulk
/// provider is down or its budget trips, cached rows are still reused and
/// the failed batches simply stay unassessed. A batch the provider's
/// content filter rejects is bisected, and a rejected single article is
/// retried on the editor when that is another provider (`curate::batch`).
pub async fn assess(
&self,
articles: Vec<Article>,
date: Date,
) -> anyhow::Result<Vec<ScoredArticle>> {
let span = tracing::info_span!("prefilter", articles = articles.len());
let _guard = span.enter();
let ctx = prefilter::PrefilterContext::load(&self.db, date).await?;
let candidates = prefilter::run(articles, &ctx, &self.config);
for candidate in &candidates {
if candidate.article.id == 0 {
continue; // not persisted yet (dry run over synthetic articles)
}
if let Err(e) = self
.db
.upsert_score(
candidate.article.id,
date,
Some(candidate.prefilter_score),
None,
)
.await
{
tracing::warn!(article_id = candidate.article.id, error = %e,
"could not persist the prefilter score");
}
}
Ok(candidates)
}
/// Stage A: batched LLM scoring of the surviving candidates (§3.6).
///
/// A no-op under `--skip-llm`. Scores are persisted per `(article, date)`.
pub async fn score(&self, candidates: &mut [ScoredArticle], date: Date) -> anyhow::Result<()> {
let Some(llm) = self.llm.as_ref() else {
tracing::info!("--skip-llm: stage A scoring skipped");
return Ok(());
candidates: &mut [Candidate],
rescore: bool,
profile_version: Option<i64>,
assessed_at: Timestamp,
) -> anyhow::Result<batch::StageSummary> {
let Some(bulk) = self.llms.bulk.as_ref() else {
tracing::info!("--skip-llm: deep assessment skipped");
return Ok(batch::StageSummary::default());
};
let span = tracing::info_span!("llm_score", candidates = candidates.len());
let admitted = candidates
.iter()
.filter(|candidate| candidate.stage == "admitted")
.count();
let span = tracing::info_span!("llm_assess", candidates = admitted);
let _guard = span.enter();
let scored = score::score_all(
llm,
assess::run(
&self.db,
Some(bulk),
self.llms.editor.as_ref(),
&bulk.model,
candidates,
self.config.deepseek.score_batch_size,
self.config.llm.deep_batch_size,
bulk.max_concurrent_requests,
self.config.curation.ranking.assessment_reuse_days,
rescore,
profile_version,
assessed_at,
self.config.llm.score_temperature,
&self.config.curation.sections,
self.config.deepseek.score_temperature,
)
.await?;
tracing::info!(scored, total = candidates.len(), "stage A complete");
for candidate in candidates.iter() {
if candidate.article.id == 0 {
continue;
.await
}
if let Some(llm_score) = candidate.llm.as_ref()
&& let Err(e) = self
.db
.upsert_score(candidate.article.id, date, None, Some(llm_score))
/// The editor: one call that assembles the issue from the shortlist (§13).
pub async fn select(&self, candidates: Vec<Candidate>, date: Date) -> anyhow::Result<Lineup> {
let sections = &self.config.curation.sections;
let soft_target = self.config.target_article_count;
let hard_max = self.config.curation.max_article_count;
let span = tracing::info_span!("llm_editor", candidates = candidates.len());
let _guard = span.enter();
match editor::select(
&self.llms,
candidates.clone(),
sections,
soft_target,
hard_max,
date,
)
.await
{
tracing::warn!(article_id = candidate.article.id, error = %e,
"could not persist the llm score");
}
}
Ok(())
}
/// Stage B: single-call lineup selection into sections (§3.6).
pub async fn select(
&self,
candidates: Vec<ScoredArticle>,
date: Date,
) -> anyhow::Result<Lineup> {
let sections = &self.config.curation.sections;
let target = self.config.target_article_count;
let Some(llm) = self.llm.as_ref() else {
tracing::info!("--skip-llm: selecting by prefilter order");
return Ok(select::select_without_llm(
candidates, sections, target, date,
));
};
let span = tracing::info_span!("llm_select", candidates = candidates.len());
let _guard = span.enter();
match select::select(llm, candidates.clone(), sections, target, date).await {
Ok(lineup) => Ok(lineup),
Err(e) => {
tracing::error!(error = %e,
"stage B selection failed; falling back to prefilter order");
Ok(select::select_without_llm(
candidates, sections, target, date,
Err(error) => {
tracing::error!(%error, "editor and bulk fallback failed; selecting heuristically");
Ok(editor::select_without_llm(
candidates,
sections,
soft_target,
hard_max,
date,
))
}
}
}
/// Stage C: per-article summaries, section intros and the front page (§3.6).
/// Stage C: per-article summaries and the Brief (§14).
///
/// Never fails the run: a budget trip or an API error degrades to excerpts.
pub async fn editorial(&self, lineup: &Lineup) -> anyhow::Result<Editorial> {
let Some(llm) = self.llm.as_ref() else {
Ok(self.editorial_timed(lineup).await?.0)
}
/// [`Self::editorial`] plus the `summaries` / `brief` stage timings (§15.4).
pub async fn editorial_timed(
&self,
lineup: &Lineup,
) -> anyhow::Result<(Editorial, editorial::EditorialTimings)> {
if self.llms.editor_or_bulk().is_none() {
tracing::info!("--skip-llm: using feed excerpts as summaries");
return Ok(editorial::fallback_editorial(lineup));
};
return Ok((
editorial::fallback_editorial(lineup),
editorial::EditorialTimings::default(),
));
}
let span = tracing::info_span!("llm_editorial", picks = lineup.picks.len());
let _guard = span.enter();
Ok(editorial::run(llm, lineup, self.config.deepseek.editorial_temperature).await)
Ok(editorial::run_timed(
&self.llms,
lineup,
&self.config.editorial,
self.config.llm.editorial_temperature,
)
.await)
}
}
@@ -157,7 +156,7 @@ impl Curator {
// ---------------------------------------------------------------------------
/// Crude token estimate: DeepSeek averages ~4 characters per token for English
/// prose. Only used to size prompt budgets (§3.6 stage C).
/// prose. Only used to size prompt budgets.
pub fn approx_tokens(text: &str) -> usize {
text.len().div_ceil(4)
}
@@ -224,7 +223,7 @@ pub fn truncate_words(text: &str, max_words: usize) -> String {
out
}
/// Truncate to roughly `max_tokens` tokens on a word boundary (§3.6 stage C).
/// Truncate to roughly `max_tokens` tokens on a word boundary.
pub fn truncate_tokens(text: &str, max_tokens: usize) -> String {
let max_chars = max_tokens.saturating_mul(4);
if text.len() <= max_chars {
+51 -438
View File
@@ -1,28 +1,12 @@
//! Heuristic pre-filter: 300–500 articles → ~120 candidates (spec §3.5).
//! Hygiene matchers and the text-only heuristic used by personalized ranking.
//!
//! Pure Rust and free: this is what keeps LLM cost flat as feed volume grows.
//!
//! The 0–100 score is a sum of bounded components so that no single signal can
//! dominate, and every component is monotonic in its input:
//!
//! | component | range | source |
//! |---|---|---|
//! | long-form word count | 0 … +35 | §3.5 "0 pts <300 words, max at ~2500+" |
//! | social proof | 0 … +25 | §3.4 composite, log-scaled again |
//! | came via Scour | +8 | §3.5 (already matched a stated interest) |
//! | came via HN frontpage | +8 | §3.5 |
//! | carried by several feeds | 0 … +8 | §3.2 (multi-source *is* social proof) |
//! | excerpt only | −20 | §3.5 (penalized, never banned — §7) |
//! | roundup/release-notes title | −15 | §3.5 |
//! | blocked domain | excluded | §3.5 |
//! This module no longer gates the candidate pool. Admission lives in
//! `curate::admit`; these helpers remain here because hygiene and cheap signals
//! share them (plan §8.1, §9, §18).
use std::collections::HashSet;
use crate::config::CurationConfig;
use crate::types::{Article, FeedId};
use crate::config::{Config, CurationConfig};
use crate::types::{Article, ArticleId, FeedId, ScoredArticle, SourceKind};
/// Title patterns that mark low-effort posts: link roundups, release notes,
/// sponsor posts (§3.5).
pub const PENALTY_TITLE_PATTERNS: &[&str] = &[
"link roundup",
"links for",
@@ -48,68 +32,12 @@ pub const PENALTY_TITLE_PATTERNS: &[&str] = &[
"digest #",
];
/// Word count at which the long-form bonus saturates (§3.5).
pub const LONGFORM_SATURATION_WORDS: i64 = 2500;
/// Below this word count the long-form bonus is zero (§3.5).
pub const LONGFORM_FLOOR_WORDS: i64 = 300;
/// Articles the LLM scored below this within the last week are not re-scored (§3.5).
pub const STALE_LOW_SCORE: f64 = 3.0;
/// Lookback for the "don't re-score churn" rule (§3.5).
pub const STALE_LOOKBACK_DAYS: i64 = 7;
/// Maximum contribution of each scoring component (§3.5).
pub const MAX_LONGFORM_POINTS: f64 = 35.0;
pub const MAX_SOCIAL_POINTS: f64 = 25.0;
pub const SCOUR_BONUS: f64 = 8.0;
pub const HN_FRONTPAGE_BONUS: f64 = 8.0;
pub const MAX_MULTI_SOURCE_POINTS: f64 = 8.0;
pub const EXCERPT_ONLY_PENALTY: f64 = 20.0;
pub const ROUNDUP_TITLE_PENALTY: f64 = 15.0;
/// `composite_social_score` value that earns the full social bonus. Empirically
/// ~6.0 is a 1,000-point HN story with 500 comments (§3.4 formula).
const SOCIAL_SATURATION: f64 = 6.0;
/// Everything the pre-filter needs beyond the articles themselves (§3.5, §3.9).
#[derive(Debug, Clone, Default)]
pub struct PrefilterContext {
/// Article ids already published in a previous issue (§3.5).
pub already_published: Vec<ArticleId>,
/// Article ids the LLM scored < [`STALE_LOW_SCORE`] recently (§3.5).
pub recently_rejected: Vec<ArticleId>,
}
impl PrefilterContext {
/// Load the history/priors context from SQLite (§3.5 dedup-vs-history, §3.9).
///
/// `today` anchors the [`STALE_LOOKBACK_DAYS`] window.
pub async fn load(
db: &crate::db::Db,
today: jiff::civil::Date,
) -> Result<Self, crate::db::DbError> {
let since = today
.checked_sub(jiff::Span::new().days(STALE_LOOKBACK_DAYS))
.unwrap_or(today);
let already_published = db.previously_published_ids().await?;
let recently_rejected = db.recently_low_scored_ids(STALE_LOW_SCORE, since).await?;
tracing::debug!(
published = already_published.len(),
rejected = recently_rejected.len(),
"loaded prefilter context"
);
Ok(Self {
already_published,
recently_rejected,
})
}
}
/// True when the article's feed is in `curation.always_include_feeds` (§3.5).
///
/// Entries are matched either as a Miniflux feed id (any feed in the cluster) or
/// as a case-insensitive substring of the article/site URL.
///
/// Auto-includes are still LLM-scored (for section + summary) but can't be dropped.
pub fn is_auto_include(article: &Article, cfg: &CurationConfig) -> bool {
if cfg.always_include_feeds.is_empty() {
return false;
@@ -122,38 +50,30 @@ pub fn is_auto_include(article: &Article, cfg: &CurationConfig) -> bool {
return false;
}
if let Ok(id) = needle.parse::<FeedId>()
&& (article.feed_id == id || article.sources.iter().any(|s| s.feed_id == id))
&& (article.feed_id == id || article.sources.iter().any(|source| source.feed_id == id))
{
return true;
}
let needle = needle.to_lowercase();
// Bare host or full site URL: compare against both URLs we hold.
let needle = needle
.to_lowercase()
.trim_start_matches("https://")
.trim_start_matches("http://")
.trim_end_matches('/');
!needle.is_empty() && (url.contains(needle) || canonical.contains(needle))
.trim_end_matches('/')
.to_string();
!needle.is_empty() && (url.contains(&needle) || canonical.contains(&needle))
})
}
/// True when the article's host matches `curation.blocked_domains` (§3.5).
pub fn is_blocked(article: &Article, cfg: &CurationConfig) -> bool {
if cfg.blocked_domains.is_empty() {
return false;
}
let host = host_of(&article.canonical_url)
.or_else(|| host_of(&article.url))
.unwrap_or_default();
if host.is_empty() {
return false;
}
cfg.blocked_domains.iter().any(|raw| {
let blocked = raw.trim().trim_start_matches('.').to_lowercase();
!blocked.is_empty() && (host == blocked || host.ends_with(&format!(".{blocked}")))
})
}
/// Lowercased host of a URL, `www.` stripped.
fn host_of(url: &str) -> Option<String> {
let rest = url
.split_once("://")
@@ -161,17 +81,12 @@ fn host_of(url: &str) -> Option<String> {
.unwrap_or(url)
.split(['/', '?', '#'])
.next()?;
let host = rest.rsplit_once('@').map(|(_, h)| h).unwrap_or(rest);
let host = host.split_once(':').map(|(h, _)| h).unwrap_or(host);
let host = rest.rsplit_once('@').map(|(_, host)| host).unwrap_or(rest);
let host = host.split_once(':').map(|(host, _)| host).unwrap_or(host);
let host = host.trim().to_lowercase();
if host.is_empty() {
None
} else {
Some(host.trim_start_matches("www.").to_string())
}
(!host.is_empty()).then(|| host.trim_start_matches("www.").to_string())
}
/// True when the title reads like a link roundup / release note / sponsor post (§3.5).
pub fn looks_like_roundup(title: &str) -> bool {
let lower = title.to_lowercase();
PENALTY_TITLE_PATTERNS
@@ -179,134 +94,38 @@ pub fn looks_like_roundup(title: &str) -> bool {
.any(|pattern| lower.contains(pattern))
}
/// Long-form bonus: zero below [`LONGFORM_FLOOR_WORDS`], saturating at
/// [`LONGFORM_SATURATION_WORDS`], with a concave curve so that the jump from a
/// 400-word note to a 1,200-word piece matters more than 2,000 → 2,500 (§3.5).
pub fn longform_points(word_count: i64) -> f64 {
let span = (LONGFORM_SATURATION_WORDS - LONGFORM_FLOOR_WORDS) as f64;
let over = (word_count - LONGFORM_FLOOR_WORDS).max(0) as f64;
MAX_LONGFORM_POINTS * (over / span).min(1.0).powf(0.65)
}
/// Social proof, log-scaled a second time so that a viral story cannot swamp the
/// long-form preference (§3.4, §3.5).
pub fn social_points(social_score: f64) -> f64 {
if social_score <= 0.0 {
return 0.0;
}
MAX_SOCIAL_POINTS * (social_score / SOCIAL_SATURATION).min(1.0).sqrt()
pub fn text_heuristic(article: &Article) -> f64 {
longform_points(article.word_count)
- excerpt_only_penalty(article)
- roundup_penalty(&article.title)
}
/// Score one article 0–100 from word count, social proof, source signals,
/// and the excerpt/roundup/blocklist penalties (§3.5).
pub fn score_article(article: &Article, _ctx: &PrefilterContext, cfg: &Config) -> f64 {
if is_blocked(article, &cfg.curation) {
return 0.0;
}
let mut score = longform_points(article.word_count);
score += social_points(article.social_score());
if article.came_via(SourceKind::Scour) {
score += SCOUR_BONUS;
}
if article.came_via(SourceKind::HnFrontpage) {
score += HN_FRONTPAGE_BONUS;
}
let extra_feeds = article.sources.len().saturating_sub(1) as f64;
score += (extra_feeds * 4.0).min(MAX_MULTI_SOURCE_POINTS);
pub fn excerpt_only_penalty(article: &Article) -> f64 {
if article.excerpt_only {
score -= EXCERPT_ONLY_PENALTY;
}
if looks_like_roundup(&article.title) {
score -= ROUNDUP_TITLE_PENALTY;
}
score.clamp(0.0, 100.0)
}
/// Apply [`score_article`] to everything, drop history duplicates, then keep the
/// top `prefilter_keep` plus every auto-include (§3.5).
pub fn run(articles: Vec<Article>, ctx: &PrefilterContext, cfg: &Config) -> Vec<ScoredArticle> {
let published: HashSet<ArticleId> = ctx.already_published.iter().copied().collect();
let rejected: HashSet<ArticleId> = ctx.recently_rejected.iter().copied().collect();
let total = articles.len();
let (mut dropped_history, mut dropped_blocked) = (0usize, 0usize);
let mut scored: Vec<ScoredArticle> = Vec::with_capacity(total);
for article in articles {
let auto_include = is_auto_include(&article, &cfg.curation);
// Never print the same story twice, not even from an always-include feed.
if published.contains(&article.id) {
dropped_history += 1;
continue;
}
// "Don't re-score churn" (§3.5) — but an always-include feed still gets in.
if !auto_include && rejected.contains(&article.id) {
dropped_history += 1;
continue;
}
if !auto_include && is_blocked(&article, &cfg.curation) {
dropped_blocked += 1;
continue;
}
let prefilter_score = score_article(&article, ctx, cfg);
let social_score = article.social_score();
scored.push(ScoredArticle {
article,
prefilter_score,
social_score,
llm: None,
auto_include,
});
}
// Descending by score; ties broken by word count then id so the order is
// deterministic across runs (notes §12).
sort_by_prefilter(&mut scored);
let keep = cfg.prefilter_keep.max(cfg.target_article_count);
let kept: Vec<ScoredArticle> = if scored.len() <= keep {
scored
EXCERPT_ONLY_PENALTY
} else {
let (head, tail) = scored.split_at(keep);
let mut kept = head.to_vec();
// Auto-includes below the cut are pulled back in — they can't be dropped.
kept.extend(tail.iter().filter(|s| s.auto_include).cloned());
sort_by_prefilter(&mut kept);
kept
};
tracing::info!(
input = total,
kept = kept.len(),
auto_includes = kept.iter().filter(|s| s.auto_include).count(),
dropped_history,
dropped_blocked,
"pre-filter complete"
);
kept
0.0
}
}
/// Deterministic ranking: score desc, then longer, then lowest id (notes §12).
pub fn sort_by_prefilter(scored: &mut [ScoredArticle]) {
scored.sort_by(|a, b| {
b.prefilter_score
.partial_cmp(&a.prefilter_score)
.unwrap_or(std::cmp::Ordering::Equal)
.then_with(|| b.article.word_count.cmp(&a.article.word_count))
.then_with(|| a.article.id.cmp(&b.article.id))
});
pub fn roundup_penalty(title: &str) -> f64 {
if looks_like_roundup(title) {
ROUNDUP_TITLE_PENALTY
} else {
0.0
}
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
use crate::types::{ExtractMethod, SocialRef, SocialSource, SourceRef};
use crate::types::{ArticleId, ExtractMethod, SocialRef, SocialSource, SourceKind, SourceRef};
use jiff::Timestamp;
pub(crate) fn ts() -> Timestamp {
@@ -315,7 +134,6 @@ pub(crate) mod tests {
.expect("static timestamp parses")
}
/// A plain 800-word article from feed 7 with no social proof.
pub(crate) fn article(id: ArticleId, title: &str, word_count: i64) -> Article {
Article {
id,
@@ -347,245 +165,40 @@ pub(crate) mod tests {
}
}
pub(crate) fn with_social(mut a: Article, points: i64, comments: i64) -> Article {
a.social = vec![SocialRef {
article_id: a.id,
pub(crate) fn with_social(mut article: Article, points: i64, comments: i64) -> Article {
article.social = vec![SocialRef {
article_id: article.id,
source: SocialSource::Hn,
item_id: Some("1".into()),
score: points,
num_comments: comments,
item_url: Some("https://news.ycombinator.com/item?id=1".into()),
item_url: None,
fetched_at: ts(),
}];
a
}
pub(crate) fn via(mut a: Article, kind: SourceKind, feed_id: FeedId) -> Article {
a.sources.push(SourceRef {
entry_id: a.best_entry_id,
feed_id,
feed_title: format!("{kind:?} feed"),
category: None,
kind,
});
a
}
fn cfg() -> Config {
Config {
prefilter_keep: 3,
target_article_count: 2,
..Config::default()
}
article
}
#[test]
fn longform_curve_is_monotonic_and_bounded() {
assert_eq!(longform_points(0), 0.0);
assert_eq!(longform_points(LONGFORM_FLOOR_WORDS), 0.0);
let mut prev = -1.0;
for wc in [0, 100, 299, 300, 500, 900, 1500, 2200, 2500, 9000] {
let pts = longform_points(wc);
assert!(pts >= prev, "not monotonic at {wc}");
assert!(pts <= MAX_LONGFORM_POINTS);
prev = pts;
}
assert!((longform_points(2500) - MAX_LONGFORM_POINTS).abs() < 1e-9);
assert!((longform_points(50_000) - MAX_LONGFORM_POINTS).abs() < 1e-9);
fn text_heuristic_has_only_text_terms() {
let quiet = article(1, "An essay", 1200);
let loud = with_social(quiet.clone(), 500, 200);
assert_eq!(text_heuristic(&quiet), text_heuristic(&loud));
assert!(text_heuristic(&article(2, "This Week in Rust", 1200)) < text_heuristic(&quiet));
}
#[test]
fn social_curve_is_monotonic_and_bounded() {
let mut prev = -1.0;
for s in [0.0, 0.5, 1.0, 2.0, 4.0, 6.0, 20.0] {
let pts = social_points(s);
assert!(pts >= prev);
assert!(pts <= MAX_SOCIAL_POINTS);
prev = pts;
}
assert_eq!(social_points(0.0), 0.0);
assert!((social_points(6.0) - MAX_SOCIAL_POINTS).abs() < 1e-9);
}
#[test]
fn score_rises_with_length_and_social_proof() {
let (ctx, cfg) = (PrefilterContext::default(), cfg());
let short = score_article(&article(1, "A thought", 200), &ctx, &cfg);
let medium = score_article(&article(2, "An essay", 1200), &ctx, &cfg);
let long = score_article(&article(3, "A treatise", 3000), &ctx, &cfg);
assert!(short < medium, "{short} !< {medium}");
assert!(medium < long, "{medium} !< {long}");
let quiet = score_article(&article(4, "An essay", 1200), &ctx, &cfg);
let loud = score_article(
&with_social(article(5, "An essay", 1200), 400, 250),
&ctx,
&cfg,
);
assert!(loud > quiet);
assert!(loud <= 100.0);
}
#[test]
fn source_bonuses_and_penalties_apply() {
let (ctx, cfg) = (PrefilterContext::default(), cfg());
// Long enough that the penalties do not run into the 0 floor.
let plain = score_article(&article(1, "Deep dive", 3000), &ctx, &cfg);
assert!(plain > EXCERPT_ONLY_PENALTY);
let scoured = score_article(
&via(article(2, "Deep dive", 3000), SourceKind::Scour, 42),
&ctx,
&cfg,
);
// Scour bonus + one extra feed in the cluster.
assert!(scoured > plain + SCOUR_BONUS - 0.001);
let mut excerpt = article(3, "Deep dive", 3000);
excerpt.excerpt_only = true;
assert!(
(score_article(&excerpt, &ctx, &cfg) - (plain - EXCERPT_ONLY_PENALTY)).abs() < 1e-9
);
let roundup = article(4, "This Week in Rust #612", 3000);
assert!(looks_like_roundup(&roundup.title));
assert!(
(score_article(&roundup, &ctx, &cfg) - (plain - ROUNDUP_TITLE_PENALTY)).abs() < 1e-9
);
}
#[test]
fn blocked_domains_and_auto_includes_match_urls_and_ids() {
let mut cfg = cfg();
cfg.curation.blocked_domains = vec!["spam.example".into()];
cfg.curation.always_include_feeds = vec!["99".into(), "tyler.blog".into()];
let mut blocked = article(1, "Buy now", 1200);
blocked.canonical_url = "https://news.spam.example/post".into();
blocked.url.clone_from(&blocked.canonical_url);
assert!(is_blocked(&blocked, &cfg.curation));
assert_eq!(
score_article(&blocked, &PrefilterContext::default(), &cfg),
0.0
);
let mut by_url = article(2, "A rare post", 900);
by_url.url = "https://tyler.blog/2026/rare".into();
assert!(is_auto_include(&by_url, &cfg.curation));
let mut by_id = article(3, "Another rare post", 900);
by_id.feed_id = 99;
assert!(is_auto_include(&by_id, &cfg.curation));
assert!(!is_auto_include(&article(4, "Normal", 900), &cfg.curation));
}
#[test]
fn keeps_top_n_plus_auto_includes_and_drops_history() {
let mut cfg = cfg();
cfg.prefilter_keep = 2;
cfg.curation.always_include_feeds = vec!["99".into()];
let mut auto = article(5, "A short personal note", 120);
fn blocked_and_auto_include_match() {
let cfg = CurationConfig {
blocked_domains: vec!["spam.example".into()],
always_include_feeds: vec!["99".into(), "tyler.blog".into()],
..CurationConfig::default()
};
let mut blocked = article(1, "spam", 100);
blocked.url = "https://news.spam.example/a".into();
blocked.canonical_url.clone_from(&blocked.url);
assert!(is_blocked(&blocked, &cfg));
let mut auto = article(2, "post", 100);
auto.feed_id = 99;
let articles = vec![
article(1, "Long treatise", 4000),
article(2, "Medium essay", 1500),
article(3, "Shorter piece", 700),
article(4, "Already printed", 5000),
auto,
article(6, "Rejected yesterday", 3000),
];
let ctx = PrefilterContext {
already_published: vec![4],
recently_rejected: vec![6],
};
let kept = run(articles, &ctx, &cfg);
let ids: Vec<ArticleId> = kept.iter().map(|s| s.article.id).collect();
assert!(!ids.contains(&4), "previously published must be dropped");
assert!(!ids.contains(&6), "recently rejected must be dropped");
assert!(ids.contains(&5), "auto-include survives below the cut");
assert!(ids.contains(&1) && ids.contains(&2));
assert!(!ids.contains(&3), "cut at prefilter_keep");
assert_eq!(kept.len(), 3); // 2 kept + 1 auto-include
// Sorted by score, descending.
for pair in kept.windows(2) {
assert!(pair[0].prefilter_score >= pair[1].prefilter_score);
}
assert!(
kept.iter()
.find(|s| s.article.id == 5)
.is_some_and(|s| s.auto_include)
);
}
#[test]
fn auto_include_survives_the_recently_rejected_list_but_not_republication() {
let mut cfg = cfg();
cfg.curation.always_include_feeds = vec!["99".into()];
let mut a = article(1, "Personal note", 200);
a.feed_id = 99;
let mut b = article(2, "Personal note two", 200);
b.feed_id = 99;
let ctx = PrefilterContext {
recently_rejected: vec![1],
already_published: vec![2],
};
let kept = run(vec![a, b], &ctx, &cfg);
let ids: Vec<ArticleId> = kept.iter().map(|s| s.article.id).collect();
assert_eq!(ids, vec![1]);
}
#[tokio::test]
async fn context_loads_history_from_sqlite() {
let dir = tempfile::tempdir().expect("tempdir");
let db = crate::db::Db::open_and_migrate(&dir.path().join("t.db"))
.await
.expect("db");
let date: jiff::civil::Date = "2026-08-15".parse().expect("date");
sqlx::query(
"INSERT INTO articles (id, canonical_url, title, first_seen) VALUES
(42, 'https://example.com/42', 'Printed', '2026-08-14T00:00:00Z'),
(43, 'https://example.com/43', 'Rejected', '2026-08-14T00:00:00Z'),
(44, 'https://example.com/44', 'Ancient', '2020-01-01T00:00:00Z')",
)
.execute(db.pool())
.await
.expect("articles");
db.upsert_issue(
"2026-08-14".parse().expect("date"),
1,
ts(),
None,
None,
None,
None,
None,
)
.await
.expect("issue");
sqlx::query(
"INSERT INTO issue_articles (issue_date, article_id, section, position, is_lead)
VALUES ('2026-08-14', 42, 'Top Stories', 1, 0)",
)
.execute(db.pool())
.await
.expect("issue article");
sqlx::query(
"INSERT INTO scores (article_id, run_date, llm_score) VALUES (43, '2026-08-14', 1.5),
(44, '2020-01-01', 1.0)",
)
.execute(db.pool())
.await
.expect("scores");
let ctx = PrefilterContext::load(&db, date).await.expect("context");
assert_eq!(ctx.already_published, vec![42]);
assert_eq!(ctx.recently_rejected, vec![43], "old rejects age out");
assert!(is_auto_include(&auto, &cfg));
}
}
+18 -3
View File
@@ -16,7 +16,10 @@ use crate::db::{Db, KV_PROFILE_VERSION, KV_TASTE_PROFILE};
use crate::types::{Facets, RatedArticle, TasteProfile};
pub const REBUILD_INTERVAL_DAYS: i64 = 7;
pub const RATINGS_LOOKBACK_DAYS: i64 = 36_500;
/// The verdict block and the weekly rebuild are bounded by count
/// (`verdicts_in_prompt`, [`MAX_RATINGS_IN_REBUILD`]), not by age (§8.3, §8.4),
/// so their `current_ratings` lookback is effectively unbounded.
const RATINGS_LOOKBACK_DAYS: i64 = 36_500;
pub const KV_LEARNED_ADJUSTMENTS: &str = "taste_profile_learned";
const MAX_RATINGS_IN_REBUILD: usize = 200;
@@ -125,6 +128,16 @@ pub fn load_profile(path: &Path) -> anyhow::Result<ProfileFile> {
}
}
/// Load the exact standing-interest union used in the system prompt.
pub fn load_standing_interests(
opml_path: &Path,
profile_path: &Path,
) -> anyhow::Result<Vec<String>> {
let opml = parse_interests(opml_path)?;
let profile = load_profile(profile_path)?;
Ok(union_interests(opml, profile.interests))
}
fn union_interests(opml: Vec<String>, profile: Vec<String>) -> Vec<String> {
let mut seen = BTreeSet::new();
let mut out = Vec::new();
@@ -294,6 +307,7 @@ pub async fn load_or_build(
),
version,
built_at,
verdicts: ratings.len().min(verdict_limit),
};
db.kv_set(KV_TASTE_PROFILE, &profile.text).await?;
tracing::debug!(
@@ -462,6 +476,7 @@ pub async fn rebuild(
),
version: next_version,
built_at,
verdicts: current.len().min(verdict_limit),
};
db.kv_set(KV_TASTE_PROFILE, &profile.text).await?;
tracing::info!(
@@ -609,7 +624,7 @@ mod tests {
use std::sync::Arc;
use super::super::llm::{MockBackend, UsageMeter};
use crate::config::DeepseekConfig;
use crate::config::ProviderConfig;
use crate::types::{RatingEvent, TokenUsage};
let dir = tempfile::tempdir().unwrap();
@@ -652,7 +667,7 @@ mod tests {
let llm = LlmClient::with_backend(
"deepseek-v4-flash",
initial.text,
UsageMeter::new(&DeepseekConfig::default(), 2.0),
UsageMeter::for_provider(&ProviderConfig::deepseek()),
backend.clone(),
);
let rebuilt = rebuild(&db, &llm, &opml, &profile_path, 60).await.unwrap();
+586
View File
@@ -0,0 +1,586 @@
//! Utility normalization, leader clustering and diversified shortlisting (§12.2–§12.5).
use std::collections::{HashMap, HashSet};
use crate::config::{RankingConfig, UtilityWeights};
use crate::curate::embedding::dot;
use crate::curate::signals;
use crate::types::{ArticleId, Candidate};
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct RankSummary {
pub shortlisted: usize,
pub clusters: usize,
}
/// Re-normalize cheap signals over the deep set and calculate utility on 0–100.
pub fn calculate_utility(candidates: &mut [Candidate], configured: &UtilityWeights) {
let indices = (0..candidates.len()).collect::<Vec<_>>();
calculate_utility_for(candidates, &indices, configured);
}
fn calculate_utility_for(
candidates: &mut [Candidate],
indices: &[usize],
configured: &UtilityWeights,
) {
let mut normalized = indices
.iter()
.map(|index| candidates[*index].signals.clone())
.collect::<Vec<_>>();
for signals in &mut normalized {
signals.norm.clear();
signals.weights.clear();
}
let mut signal_refs = normalized.iter_mut().collect::<Vec<_>>();
signals::normalize(&mut signal_refs);
for (index, signals) in indices.iter().zip(normalized) {
candidates[*index].signals.norm = signals.norm;
candidates[*index].signals.weights.clear();
}
for index in indices {
let candidate = &mut candidates[*index];
if let Some(triage) = &candidate.assessment.triage {
candidate
.signals
.norm
.insert("triage".into(), (triage.interest / 10.0).clamp(0.0, 1.0));
}
if let Some(deep) = &candidate.assessment.deep {
candidate
.signals
.norm
.insert("quality".into(), (deep.quality / 10.0).clamp(0.0, 1.0));
candidate
.signals
.norm
.insert("fit".into(), (deep.fit / 10.0).clamp(0.0, 1.0));
}
let weighted = [
("quality", configured.quality, 1.0),
("fit", configured.fit, 1.0),
("knn", configured.knn, candidate.signals.knn_gate),
("interest", configured.interest, 1.0),
("feed", configured.feed, candidate.signals.feed_gate),
("triage", configured.triage, 1.0),
("social", configured.social, 1.0),
("heuristic", configured.heuristic, 1.0),
]
.into_iter()
.filter_map(|(name, weight, gate)| {
let value = candidate.signals.norm.get(name).copied()?;
let effective = weight * gate;
(effective > 0.0).then_some((name, effective, value))
})
.collect::<Vec<_>>();
let total = weighted.iter().map(|(_, weight, _)| weight).sum::<f64>();
if total <= 0.0 {
candidate.utility = None;
continue;
}
candidate.signals.weights = weighted
.iter()
.map(|(name, weight, _)| ((*name).to_string(), weight / total))
.collect();
candidate.utility = Some(
weighted
.iter()
.map(|(_, weight, value)| weight / total * value)
.sum::<f64>()
* 100.0,
);
}
}
fn ranked_indices(candidates: &[Candidate], indices: &[usize]) -> Vec<usize> {
let mut sorted = indices.to_vec();
sorted.sort_by(|left, right| {
candidates[*right]
.utility
.unwrap_or(f64::NEG_INFINITY)
.total_cmp(&candidates[*left].utility.unwrap_or(f64::NEG_INFINITY))
.then_with(|| {
candidates[*left]
.article
.id
.cmp(&candidates[*right].article.id)
})
});
sorted
}
#[derive(Debug)]
struct Cluster {
id: i64,
leader: usize,
}
/// Rank the admitted deep set and leave only the diversified shortlist at
/// `stage = shortlisted`. All deep-set articles receive ranks and clusters.
pub fn shortlist(
candidates: &mut [Candidate],
embeddings: &HashMap<ArticleId, Vec<f32>>,
ranking: &RankingConfig,
) -> RankSummary {
let deep = candidates
.iter()
.enumerate()
.filter(|(_, candidate)| matches!(candidate.stage.as_str(), "admitted" | "assessed"))
.map(|(index, _)| index)
.collect::<Vec<_>>();
calculate_utility_for(candidates, &deep, &ranking.weights.utility);
let sorted = ranked_indices(candidates, &deep);
for (rank, index) in sorted.iter().enumerate() {
candidates[*index].rank_utility = Some(rank as i64 + 1);
candidates[*index].cluster = None;
candidates[*index].cluster_rank = None;
}
let mut clusters = Vec::<Cluster>::new();
let mut members: HashMap<i64, Vec<usize>> = HashMap::new();
for index in &sorted {
let assigned = embeddings
.get(&candidates[*index].article.id)
.and_then(|vector| {
clusters.iter().find_map(|cluster| {
let leader_id = candidates[cluster.leader].article.id;
let leader = embeddings.get(&leader_id)?;
dot(vector, leader)
.ok()
.filter(|cosine| *cosine >= ranking.diversity.cluster_threshold)
.map(|_| cluster.id)
})
});
let cluster_id = assigned.unwrap_or_else(|| {
let id = clusters.len() as i64 + 1;
clusters.push(Cluster { id, leader: *index });
id
});
let cluster_members = members.entry(cluster_id).or_default();
cluster_members.push(*index);
candidates[*index].cluster = Some(cluster_id);
candidates[*index].cluster_rank = Some(cluster_members.len() as i64);
}
let protected = sorted
.iter()
.take(ranking.diversity.utility_protected)
.copied()
.collect::<HashSet<_>>();
let mut admitted = HashSet::new();
let mut admitted_per_cluster = HashMap::<i64, usize>::new();
let admit = |index: usize, admitted: &mut HashSet<usize>, counts: &mut HashMap<i64, usize>| {
if admitted.insert(index)
&& let Some(cluster) = candidates[index].cluster
{
*counts.entry(cluster).or_default() += 1;
}
};
for index in &sorted {
if protected.contains(index) || candidates[*index].auto_include {
admit(*index, &mut admitted, &mut admitted_per_cluster);
}
}
for index in sorted
.iter()
.filter(|index| candidates[**index].exploration)
.take(3)
{
if admitted.len() >= ranking.shortlist_keep {
break;
}
admit(*index, &mut admitted, &mut admitted_per_cluster);
}
let target = ranking.shortlist_keep.max(admitted.len());
let mut suppressed_at_base_cap = HashSet::new();
admit_under_cap(
candidates,
&sorted,
target,
ranking.diversity.per_cluster_cap,
&mut admitted,
&mut admitted_per_cluster,
Some(&mut suppressed_at_base_cap),
);
if admitted.len() < target {
admit_under_cap(
candidates,
&sorted,
target,
3,
&mut admitted,
&mut admitted_per_cluster,
None,
);
}
if admitted.len() < target {
for index in &sorted {
if admitted.len() >= target {
break;
}
admit(*index, &mut admitted, &mut admitted_per_cluster);
}
}
for index in deep {
if admitted.contains(&index) {
candidates[index].stage = "shortlisted".into();
candidates[index].excluded_reason = None;
} else {
// The stage stays where the article stopped (`admitted` when the
// deep assessment never happened, else `assessed`).
candidates[index].excluded_reason = Some(
if suppressed_at_base_cap.contains(&index) {
"cluster_suppressed"
} else {
"shortlist_cap"
}
.into(),
);
}
}
RankSummary {
shortlisted: admitted.len(),
clusters: clusters.len(),
}
}
#[allow(clippy::too_many_arguments)]
fn admit_under_cap(
candidates: &[Candidate],
sorted: &[usize],
target: usize,
cap: usize,
admitted: &mut HashSet<usize>,
admitted_per_cluster: &mut HashMap<i64, usize>,
mut suppressed: Option<&mut HashSet<usize>>,
) {
for index in sorted {
if admitted.len() >= target || admitted.contains(index) {
continue;
}
let Some(cluster) = candidates[*index].cluster else {
continue;
};
if admitted_per_cluster.get(&cluster).copied().unwrap_or(0) >= cap {
if let Some(suppressed) = suppressed.as_deref_mut() {
suppressed.insert(*index);
}
continue;
}
admitted.insert(*index);
*admitted_per_cluster.entry(cluster).or_default() += 1;
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::DiversityConfig;
use crate::curate::prefilter::tests::article;
use crate::curate::signals::Signals;
use crate::types::{Deep, Facets};
fn candidate(id: i64, utility_hint: f64) -> Candidate {
let mut candidate = Candidate::new(article(id, &format!("article {id}"), 800), false);
candidate.stage = "assessed".into();
candidate.signals = Signals {
heuristic: Some(utility_hint),
..Signals::default()
};
candidate.assessment.deep = Some(Deep {
quality: utility_hint,
fit: utility_hint,
category: Some("Top Stories".into()),
rationale: "specific".into(),
paywalled_guess: false,
facets: Facets::default(),
model: "mock".into(),
prompt_version: 1,
assessed_at: "2026-09-02T00:00:00Z".parse().expect("timestamp"),
});
candidate
}
fn vector(angle: f32) -> Vec<f32> {
vec![angle.cos(), angle.sin()]
}
fn ranking(keep: usize, protected: usize, cap: usize, threshold: f64) -> RankingConfig {
RankingConfig {
shortlist_keep: keep,
diversity: DiversityConfig {
cluster_threshold: threshold,
per_cluster_cap: cap,
utility_protected: protected,
},
..RankingConfig::default()
}
}
#[test]
fn utility_renormalizes_present_signals_and_gates_learned_ones() {
let mut a = candidate(1, 8.0);
let mut b = candidate(2, 4.0);
a.signals.interest = Some(1.0);
b.signals.interest = None;
a.signals.knn = Some(0.9);
a.signals.knn_gate = 0.5;
let mut values = vec![a, b];
calculate_utility(&mut values, &UtilityWeights::default());
let [a, b] = values.as_slice() else {
panic!("two values")
};
for candidate in [&a, &b] {
assert!((candidate.signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
assert!(candidate.utility.is_some());
}
assert!(!b.signals.weights.contains_key("interest"));
assert!(
(a.signals.weights["knn"] / a.signals.weights["quality"] - (0.15 * 0.5) / 0.40).abs()
< 1e-9
);
}
#[test]
fn duplicates_cluster_and_the_third_is_suppressed() {
let ranking = ranking(3, 0, 2, 0.85);
let mut candidates = vec![
candidate(1, 9.0),
candidate(2, 8.0),
candidate(3, 7.0),
candidate(4, 6.0),
];
let embeddings = HashMap::from([
(1, vector(0.0)),
(2, vector(0.1)),
(3, vector(0.2)),
(4, vector(2.0)),
]);
let summary = shortlist(&mut candidates, &embeddings, &ranking);
assert_eq!(summary.shortlisted, 3);
assert_eq!(candidates[0].cluster, candidates[1].cluster);
assert_eq!(candidates[1].cluster, candidates[2].cluster);
assert_eq!(
candidates[2].excluded_reason.as_deref(),
Some("cluster_suppressed")
);
}
#[test]
fn protected_items_survive_and_count_toward_the_cap() {
let ranking = ranking(3, 2, 1, 0.85);
let mut candidates = vec![
candidate(1, 9.0),
candidate(2, 8.0),
candidate(3, 7.0),
candidate(4, 6.0),
];
let embeddings = HashMap::from([
(1, vector(0.0)),
(2, vector(0.05)),
(3, vector(0.1)),
(4, vector(2.0)),
]);
shortlist(&mut candidates, &embeddings, &ranking);
assert_eq!(candidates[0].stage, "shortlisted");
assert_eq!(candidates[1].stage, "shortlisted");
assert_ne!(candidates[2].stage, "shortlisted");
}
#[test]
fn bridge_case_uses_leaders_not_transitive_components() {
let ranking = ranking(3, 0, 2, 0.80);
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
// A at 0°, B at 72°, C at 36°: A~C and B~C, but A not~B.
let embeddings =
HashMap::from([(1, vector(0.0)), (2, vector(1.2566)), (3, vector(0.6283))]);
let summary = shortlist(&mut candidates, &embeddings, &ranking);
assert_eq!(summary.clusters, 2);
assert_eq!(candidates[0].cluster, candidates[2].cluster);
assert_ne!(candidates[0].cluster, candidates[1].cluster);
}
#[test]
fn missing_embeddings_are_singletons_and_never_suppressed() {
let ranking = ranking(3, 0, 1, 0.85);
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
shortlist(&mut candidates, &HashMap::new(), &ranking);
assert!(
candidates
.iter()
.all(|candidate| candidate.stage == "shortlisted")
);
assert_eq!(
candidates
.iter()
.filter_map(|candidate| candidate.cluster)
.collect::<HashSet<_>>()
.len(),
3
);
}
#[test]
fn caps_relax_to_three_then_uncapped_when_the_shortlist_is_short() {
// Six near-duplicates, keep 5: cap 2 admits two, cap 3 admits a third,
// and the uncapped pass fills the remaining two slots in utility order.
let ranking = ranking(5, 0, 2, 0.85);
let mut candidates = (1..=6)
.map(|id| candidate(id, 10.0 - id as f64))
.collect::<Vec<_>>();
let embeddings = (1..=6)
.map(|id| (id, vector(0.01 * id as f32)))
.collect::<HashMap<_, _>>();
let summary = shortlist(&mut candidates, &embeddings, &ranking);
assert_eq!(summary.clusters, 1);
assert_eq!(summary.shortlisted, 5);
let shortlisted = candidates
.iter()
.filter(|candidate| candidate.stage == "shortlisted")
.map(|candidate| candidate.article.id)
.collect::<Vec<_>>();
assert_eq!(shortlisted, vec![1, 2, 3, 4, 5], "filled in utility order");
assert_eq!(candidates[5].stage, "assessed");
assert_eq!(
candidates[5].excluded_reason.as_deref(),
Some("cluster_suppressed")
);
assert_eq!(
candidates
.iter()
.map(|c| c.rank_utility)
.collect::<Vec<_>>(),
(1..=6).map(Some).collect::<Vec<_>>()
);
assert_eq!(
candidates
.iter()
.map(|c| c.cluster_rank)
.collect::<Vec<_>>(),
(1..=6).map(Some).collect::<Vec<_>>()
);
}
#[test]
fn shortlist_cap_is_the_reason_beyond_the_keep() {
let ranking = ranking(2, 0, 2, 0.85);
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
shortlist(&mut candidates, &HashMap::new(), &ranking);
assert_eq!(candidates[0].stage, "shortlisted");
assert_eq!(candidates[1].stage, "shortlisted");
assert_eq!(candidates[2].stage, "assessed");
assert_eq!(
candidates[2].excluded_reason.as_deref(),
Some("shortlist_cap")
);
}
#[test]
fn exploration_picks_get_up_to_three_reserved_slots() {
// Keep 4 with the four best by utility being ordinary articles: three
// exploration picks are still reserved seats, the fourth is not.
let ranking = ranking(4, 0, 2, 0.85);
let mut candidates = (1..=8)
.map(|id| candidate(id, 10.0 - id as f64))
.collect::<Vec<_>>();
for candidate in candidates.iter_mut().skip(4) {
candidate.exploration = true;
}
let summary = shortlist(&mut candidates, &HashMap::new(), &ranking);
assert_eq!(summary.shortlisted, 4);
let shortlisted = candidates
.iter()
.filter(|candidate| candidate.stage == "shortlisted")
.map(|candidate| candidate.article.id)
.collect::<Vec<_>>();
assert_eq!(shortlisted, vec![1, 5, 6, 7]);
assert_eq!(
candidates[7].excluded_reason.as_deref(),
Some("shortlist_cap")
);
}
#[test]
fn auto_includes_are_admitted_regardless_and_count_toward_their_cluster() {
let ranking = ranking(2, 0, 1, 0.85);
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 1.0)];
candidates[2].auto_include = true;
let embeddings = HashMap::from([(1, vector(0.0)), (2, vector(2.0)), (3, vector(0.05))]);
let summary = shortlist(&mut candidates, &embeddings, &ranking);
assert_eq!(summary.shortlisted, 2);
assert_eq!(candidates[2].stage, "shortlisted", "auto-include survives");
// The auto-include filled its cluster's single seat, so the stronger
// near-duplicate is suppressed and the unrelated article gets the slot.
assert_eq!(candidates[0].cluster, candidates[2].cluster);
assert_eq!(candidates[0].stage, "assessed");
assert_eq!(
candidates[0].excluded_reason.as_deref(),
Some("cluster_suppressed")
);
assert_eq!(candidates[1].stage, "shortlisted");
}
#[test]
fn unassessed_articles_rank_on_present_signals_and_keep_their_stage() {
// DeepSeek down: no quality/fit anywhere, utility comes from what is present.
let ranking = ranking(2, 0, 2, 0.85);
let mut candidates = vec![candidate(1, 3.0), candidate(2, 6.0), candidate(3, 9.0)];
for candidate in &mut candidates {
candidate.assessment.deep = None;
candidate.stage = "admitted".into();
}
candidates[0].assessment.triage = Some(crate::types::Triage {
interest: 9.0,
kind: "essay".into(),
why: "promising".into(),
model: "mock".into(),
prompt_version: 1,
assessed_at: "2026-09-02T00:00:00Z".parse().expect("timestamp"),
});
let summary = shortlist(&mut candidates, &HashMap::new(), &ranking);
assert_eq!(summary.shortlisted, 2);
for candidate in &candidates {
assert!(candidate.utility.is_some(), "scored on present signals");
assert!(!candidate.signals.weights.contains_key("quality"));
assert!(!candidate.signals.weights.contains_key("fit"));
assert!((candidate.signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
}
// Triage (0.05) outweighs heuristic (0.02): the triaged article with
// the weakest heuristic overtakes the middle one.
assert_eq!(candidates[2].rank_utility, Some(1));
assert_eq!(candidates[0].rank_utility, Some(2));
assert_eq!(candidates[1].rank_utility, Some(3));
assert_eq!(
candidates[1].stage, "admitted",
"never assessed, so not `assessed`"
);
assert_eq!(
candidates[1].excluded_reason.as_deref(),
Some("shortlist_cap")
);
}
#[test]
fn percentiles_are_taken_over_the_deep_set_only() {
// The eligible-but-not-admitted article has the strongest heuristic;
// it must not shift the deep set's percentiles or receive a utility.
let ranking = ranking(10, 0, 2, 0.85);
let mut candidates = vec![candidate(1, 5.0), candidate(2, 5.0), candidate(3, 5.0)];
candidates[2].stage = "triaged".into();
candidates[2].excluded_reason = Some("not_admitted".into());
candidates[2].signals.heuristic = Some(99.0);
candidates[1].signals.heuristic = Some(5.0);
shortlist(&mut candidates, &HashMap::new(), &ranking);
assert_eq!(
candidates[0].signals.norm["heuristic"], 0.5,
"ties share a percentile"
);
assert_eq!(candidates[1].signals.norm["heuristic"], 0.5);
assert!(candidates[2].utility.is_none());
assert!(candidates[2].rank_utility.is_none());
}
}
-592
View File
@@ -1,592 +0,0 @@
//! Stage A — batched LLM scoring (spec §3.6).
//!
//! Batches of `deepseek.score_batch_size` articles per request. Per article we
//! send title, source feed, author, word count, social stats, sources list and a
//! ~200-word excerpt; the model returns one JSON object per article.
//!
//! Parsing is deliberately forgiving: one malformed item must not cost us the
//! other eleven, and a failed batch must not fail the run.
use std::collections::HashMap;
use std::fmt::Write as _;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use super::llm::{LlmClient, LlmError, strip_code_fence};
use super::{prompt_text, truncate_words};
use crate::types::{ArticleId, LlmScore, ScoredArticle, SourceKind};
/// Words of article text sent per candidate in stage A (§3.6).
pub const EXCERPT_WORDS: usize = 200;
/// One element of the stage-A JSON response (§3.6).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ScoreItem {
pub id: ArticleId,
/// 0–10.
pub score: f64,
pub category: String,
/// ≤ 20 words.
#[serde(default)]
pub rationale: String,
#[serde(default)]
pub is_paywalled_guess: bool,
}
impl From<ScoreItem> for LlmScore {
fn from(i: ScoreItem) -> Self {
LlmScore {
score: i.score,
category: i.category,
rationale: i.rationale,
is_paywalled_guess: i.is_paywalled_guess,
}
}
}
/// Envelope the model is asked to return (`{"articles": [...]}`).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ScoreResponse {
#[serde(default)]
pub articles: Vec<ScoreItem>,
}
/// The invariant instruction block for stage A. Everything article-specific goes
/// in the per-batch tail so this prefix stays cacheable (§3.6).
pub const SCORE_INSTRUCTIONS: &str = "\
TASK: score a batch of candidate articles for today's issue of The Daily EPUB.
Judge each article against the reader profile in your system prompt — not against \
a general audience, and not against what is objectively newsworthy.
Return one object per input article with these fields:
\"id\" integer, copied exactly from the input
\"score\" number 0-10, the rubric below
\"category\" one short label from the palette below
\"rationale\" at most 20 words, concrete, no hedging, no restating the title
\"is_paywalled_guess\" true when the text looks truncated, teaser-like or paywalled
SCORING RUBRIC — calibrate hard; a normal day averages about 4, and a 9 should \
appear a couple of times a week, not a couple of times a day:
9-10 Exceptional. Original reporting, a deep technical dive, or an essay he \
will still be thinking about next week. Evident effort and a real point of view.
7-8 Strong. A well-made long-form piece squarely in his interests, or an \
outstanding piece outside them.
5-6 Worth a slot on a thin day. Solid, useful, a little thin or a little \
familiar.
3-4 Marginal. Competent news-of-the-day, short posts, incremental updates, \
good writing about an over-covered story.
1-2 Weak. Announcements, changelogs and release notes, link roundups, \
listicles, rewrites of a story available at the source, thin AI-industry churn.
0 Unusable. Press releases, sponsored content, engagement bait, spam, \
pure crypto promotion, or an entry with no readable body.
CALIBRATION NOTES
- Length alone is not quality; padding scores worse than a tight short piece. But \
between two equally good pieces, prefer the one with more substance.
- Social proof is evidence, not a verdict: hundreds of HN points mean a critical \
audience read it; a quiet post from a good blog can still outrank it.
- \"came via scour\" means the story already matched one of his standing \
interests. \"came via hn_frontpage\" means it cleared HN's front page.
- Boston/New England local stories and ultra-niche community news get a genuine \
lift — this paper wants them.
- Wire-service world/US news should score low here: the World Briefing section \
covers that separately.
- Excerpt-only or paywalled text is a real cost to the reader; score it lower \
unless the piece is clearly excellent.
Return JSON exactly in this shape, with one entry per input article and nothing \
else:
{\"articles\": [{\"id\": 123, \"score\": 7.5, \"category\": \"Tech & Engineering\", \
\"rationale\": \"first-hand account of migrating 40TB off Postgres\", \
\"is_paywalled_guess\": false}]}";
/// Render the user prompt for one batch (§3.6).
pub fn build_batch_prompt(batch: &[ScoredArticle], sections: &[String]) -> String {
let mut prompt = String::with_capacity(4096 + batch.len() * 1500);
prompt.push_str(SCORE_INSTRUCTIONS);
let _ = write!(
prompt,
"\n\nCATEGORY PALETTE (use one of these exact strings): {}\n\nARTICLES ({} in this batch)\n",
sections.join(" | "),
batch.len()
);
for candidate in batch {
prompt.push('\n');
prompt.push_str(&render_candidate(candidate));
}
prompt
}
/// One article's block in the stage-A prompt (§3.6).
fn render_candidate(candidate: &ScoredArticle) -> String {
let a = &candidate.article;
let mut block = String::with_capacity(1500);
let _ = writeln!(block, "--- id: {}", a.id);
let _ = writeln!(block, "title: {}", a.title.trim());
let _ = writeln!(
block,
"feed: {}{}",
if a.feed_title.is_empty() {
"unknown"
} else {
a.feed_title.trim()
},
a.category
.as_deref()
.filter(|c| !c.is_empty())
.map(|c| format!(" (category: {c})"))
.unwrap_or_default()
);
if let Some(author) = a.author.as_deref().filter(|s| !s.trim().is_empty()) {
let _ = writeln!(block, "author: {}", author.trim());
}
let _ = writeln!(
block,
"length: {} words (~{} min read){}",
a.word_count,
a.reading_minutes(),
if a.excerpt_only {
" [EXCERPT ONLY — full text unavailable]"
} else {
""
}
);
let _ = writeln!(block, "social: {}", social_line(candidate));
let _ = writeln!(block, "came via: {}", sources_line(candidate));
let excerpt = truncate_words(&prompt_text(&a.content_html), EXCERPT_WORDS);
let _ = writeln!(
block,
"excerpt: {}",
if excerpt.is_empty() {
"(no body text extracted)"
} else {
&excerpt
}
);
block
}
fn social_line(candidate: &ScoredArticle) -> String {
if candidate.article.social.is_empty() {
return "none found".into();
}
let mut parts: Vec<String> = candidate
.article
.social
.iter()
.map(|s| {
format!(
"{} {} points / {} comments",
s.source.display_name(),
s.score,
s.num_comments
)
})
.collect();
parts.push(format!("composite {:.2}", candidate.social_score));
parts.join("; ")
}
fn sources_line(candidate: &ScoredArticle) -> String {
let mut kinds: Vec<&str> = candidate
.article
.sources
.iter()
.map(|s| match s.kind {
SourceKind::Scour => "scour",
SourceKind::HnFrontpage => "hn_frontpage",
SourceKind::Lobsters => "lobsters",
SourceKind::Reddit => "reddit",
SourceKind::Feed => "feed",
})
.collect();
kinds.sort_unstable();
kinds.dedup();
if candidate.auto_include {
kinds.push("always-include feed (cannot be dropped)");
}
if kinds.is_empty() {
"feed".into()
} else {
kinds.join(", ")
}
}
// ---------------------------------------------------------------------------
// Response parsing (§3.6: tolerate anything the model does to us)
// ---------------------------------------------------------------------------
/// Keys the model might wrap the array in, in preference order.
const ARRAY_KEYS: &[&str] = &["articles", "scores", "results", "items", "data"];
/// Parse a stage-A response leniently: missing optional fields default, scores
/// are clamped to 0–10, and malformed items are skipped with a warning (§3.6).
pub fn parse_score_response(raw: &str) -> Vec<ScoreItem> {
let cleaned = strip_code_fence(raw);
let value: Value = match serde_json::from_str(cleaned) {
Ok(v) => v,
Err(e) => {
tracing::warn!(error = %e, "stage A response was not JSON at all");
return Vec::new();
}
};
let array = match &value {
Value::Array(items) => Some(items),
Value::Object(map) => ARRAY_KEYS
.iter()
.find_map(|k| map.get(*k).and_then(Value::as_array))
// Some models return {"1234": {...}} or a single bare object.
.or_else(|| map.values().find_map(Value::as_array)),
_ => None,
};
let Some(array) = array else {
tracing::warn!("stage A response contained no array of scores");
return Vec::new();
};
let mut out = Vec::with_capacity(array.len());
let mut skipped = 0usize;
for item in array {
match parse_item(item) {
Some(parsed) => out.push(parsed),
None => {
skipped += 1;
tracing::warn!(item = %truncate_debug(item), "skipping malformed stage A item");
}
}
}
if skipped > 0 {
tracing::warn!(skipped, kept = out.len(), "stage A items were dropped");
}
out
}
fn parse_item(item: &Value) -> Option<ScoreItem> {
let obj = item.as_object()?;
let id = obj.get("id").and_then(as_i64_lenient)?;
let score = obj
.get("score")
.and_then(as_f64_lenient)
.or_else(|| obj.get("rating").and_then(as_f64_lenient))?;
Some(ScoreItem {
id,
score: score.clamp(0.0, 10.0),
category: obj
.get("category")
.and_then(Value::as_str)
.unwrap_or_default()
.trim()
.to_string(),
rationale: obj
.get("rationale")
.or_else(|| obj.get("reason"))
.and_then(Value::as_str)
.unwrap_or_default()
.trim()
.to_string(),
is_paywalled_guess: obj
.get("is_paywalled_guess")
.or_else(|| obj.get("paywalled"))
.and_then(as_bool_lenient)
.unwrap_or(false),
})
}
fn as_i64_lenient(v: &Value) -> Option<i64> {
v.as_i64()
.or_else(|| v.as_f64().map(|f| f as i64))
.or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
}
fn as_f64_lenient(v: &Value) -> Option<f64> {
v.as_f64()
.or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
.filter(|f| f.is_finite())
}
fn as_bool_lenient(v: &Value) -> Option<bool> {
v.as_bool().or_else(|| match v.as_str()?.trim() {
"true" | "yes" => Some(true),
"false" | "no" => Some(false),
_ => None,
})
}
fn truncate_debug(v: &Value) -> String {
v.to_string().chars().take(160).collect()
}
// ---------------------------------------------------------------------------
// Stage driver
// ---------------------------------------------------------------------------
/// Score every candidate, filling in [`ScoredArticle::llm`] (§3.6).
///
/// Batches that fail are logged and left unscored rather than aborting the run.
/// Returns how many candidates came back with a score.
pub async fn score_all(
llm: &LlmClient,
candidates: &mut [ScoredArticle],
batch_size: usize,
sections: &[String],
temperature: f32,
) -> Result<usize, LlmError> {
if candidates.is_empty() {
return Ok(0);
}
let batch_size = batch_size.max(1);
let batches = candidates.len().div_ceil(batch_size);
let mut scores: HashMap<ArticleId, LlmScore> = HashMap::with_capacity(candidates.len());
for (n, batch) in candidates.chunks(batch_size).enumerate() {
if let Err(e) = llm.meter.check_budget() {
tracing::error!(
error = %e,
batch = n + 1,
of = batches,
unscored = candidates.len() - scores.len(),
"COST CEILING HIT during stage A scoring — remaining batches skipped; \
the lineup will fall back to heuristic ranking for them"
);
break;
}
let prompt = build_batch_prompt(batch, sections);
tracing::debug!(
batch = n + 1,
of = batches,
articles = batch.len(),
approx_tokens = super::approx_tokens(&prompt),
"stage A request"
);
match llm.complete(&prompt, temperature, true).await {
Ok(raw) => {
let items = parse_score_response(&raw);
if items.is_empty() {
tracing::warn!(
batch = n + 1,
of = batches,
"stage A batch returned no scores"
);
}
for item in items {
scores.insert(item.id, item.into());
}
}
Err(e) => {
tracing::warn!(batch = n + 1, of = batches, error = %e,
"stage A batch failed; its articles stay unscored");
}
}
}
let mut applied = 0usize;
for candidate in candidates.iter_mut() {
if let Some(score) = scores.remove(&candidate.article.id) {
candidate.llm = Some(score);
applied += 1;
}
}
if !scores.is_empty() {
tracing::warn!(
unknown_ids = scores.len(),
"stage A returned scores for ids that were not in the batch"
);
}
Ok(applied)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::DeepseekConfig;
use crate::curate::llm::{MockBackend, UsageMeter};
use crate::curate::prefilter::tests::{article, via, with_social};
use crate::types::TokenUsage;
use std::sync::Arc;
const BATCH_FIXTURE: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/deepseek_score_batch.json"
));
const MESSY_FIXTURE: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/deepseek_score_batch_messy.json"
));
fn sections() -> Vec<String> {
crate::config::CurationConfig::default().sections
}
fn candidate(id: i64, title: &str, words: i64) -> ScoredArticle {
ScoredArticle {
article: article(id, title, words),
prefilter_score: 50.0,
social_score: 0.0,
llm: None,
auto_include: false,
}
}
#[test]
fn batch_prompt_carries_every_documented_signal() {
let mut c = candidate(12, "Migrating 40TB off Postgres", 3200);
c.article = via(
with_social(c.article, 342, 210),
SourceKind::HnFrontpage,
9001,
);
c.social_score = c.article.social_score();
c.auto_include = true;
let prompt = build_batch_prompt(&[c], &sections());
assert!(prompt.starts_with(SCORE_INSTRUCTIONS));
assert!(prompt.contains("--- id: 12"));
assert!(prompt.contains("title: Migrating 40TB off Postgres"));
assert!(prompt.contains("feed: Some Blog (category: Tech)"));
assert!(prompt.contains("author: A. Writer"));
assert!(prompt.contains("length: 3200 words"));
assert!(prompt.contains("HN 342 points / 210 comments"));
assert!(prompt.contains("hn_frontpage"));
assert!(prompt.contains("always-include feed"));
assert!(prompt.contains("excerpt: word word"));
assert!(prompt.contains("Tech & Engineering"));
// The excerpt is capped.
let excerpt_line = prompt
.lines()
.find(|l| l.starts_with("excerpt:"))
.expect("excerpt line");
assert!(excerpt_line.split_whitespace().count() <= EXCERPT_WORDS + 2);
}
#[test]
fn parses_a_realistic_deepseek_batch() {
let items = parse_score_response(BATCH_FIXTURE);
assert_eq!(items.len(), 4);
assert_eq!(items[0].id, 101);
assert!((items[0].score - 8.5).abs() < 1e-9);
assert_eq!(items[0].category, "Tech & Engineering");
assert!(items[0].rationale.split_whitespace().count() <= 20);
assert!(!items[0].is_paywalled_guess);
assert!(items[3].is_paywalled_guess);
let score: LlmScore = items[0].clone().into();
assert_eq!(score.category, "Tech & Engineering");
}
#[test]
fn parsing_survives_everything_a_model_might_do() {
let items = parse_score_response(MESSY_FIXTURE);
let ids: Vec<ArticleId> = items.iter().map(|i| i.id).collect();
// 201 fine; 202 string score clamped; 203 missing rationale/category;
// 204 out-of-range clamped; the two malformed entries are dropped.
assert_eq!(ids, vec![201, 202, 203, 204]);
assert!((items[1].score - 6.0).abs() < 1e-9);
assert_eq!(items[2].rationale, "");
assert_eq!(items[2].category, "");
assert!(
(items[3].score - 10.0).abs() < 1e-9,
"clamped to the 0-10 range"
);
assert!(items.iter().all(|i| (0.0..=10.0).contains(&i.score)));
}
#[test]
fn parsing_tolerates_fences_arrays_and_junk() {
assert_eq!(
parse_score_response("```json\n{\"articles\":[{\"id\":1,\"score\":5}]}\n```").len(),
1
);
assert_eq!(parse_score_response("[{\"id\": 2, \"score\": 3}]").len(), 1);
assert_eq!(
parse_score_response("{\"results\":[{\"id\":3,\"score\":\"4.5\"}]}")[0].score,
4.5
);
assert!(parse_score_response("I'm sorry, I can't do that").is_empty());
assert!(parse_score_response("{\"articles\": {}}").is_empty());
}
fn client(backend: Arc<MockBackend>, limit_usd: f64) -> LlmClient {
LlmClient::with_backend(
"deepseek-v4-flash",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), limit_usd),
backend,
)
}
#[tokio::test]
async fn scores_are_applied_batch_by_batch() {
let backend = Arc::new(MockBackend::new());
backend.push(
r#"{"articles":[{"id":1,"score":8,"category":"Tech & Engineering","rationale":"good"},
{"id":2,"score":2,"category":"Niche Corner","rationale":"thin"}]}"#,
TokenUsage::default(),
);
backend.push(
r#"{"articles":[{"id":3,"score":6.5,"category":"Culture & Essays","rationale":"solid"}]}"#,
TokenUsage::default(),
);
let llm = client(Arc::clone(&backend), 2.0);
let mut candidates = vec![
candidate(1, "One", 1000),
candidate(2, "Two", 1000),
candidate(3, "Three", 1000),
];
let scored = score_all(&llm, &mut candidates, 2, &sections(), 0.3)
.await
.expect("scoring");
assert_eq!(scored, 3);
assert_eq!(backend.calls(), 2, "batched by score_batch_size");
assert_eq!(candidates[0].llm.as_ref().map(|l| l.score), Some(8.0));
assert_eq!(candidates[2].llm.as_ref().map(|l| l.score), Some(6.5));
// combined_score now reflects the LLM verdict.
assert!(candidates[0].combined_score() > candidates[1].combined_score());
}
#[tokio::test]
async fn a_failed_batch_does_not_sink_the_run() {
let backend = Arc::new(MockBackend::new());
backend.push_error("500 upstream exploded");
backend.push(
r#"{"articles":[{"id":2,"score":7,"category":"Top Stories","rationale":"ok"}]}"#,
TokenUsage::default(),
);
let llm = client(Arc::clone(&backend), 2.0);
let mut candidates = vec![candidate(1, "One", 900), candidate(2, "Two", 900)];
let scored = score_all(&llm, &mut candidates, 1, &sections(), 0.3)
.await
.expect("scoring must not abort");
assert_eq!(scored, 1);
assert!(candidates[0].llm.is_none());
assert!(candidates[1].llm.is_some());
}
#[tokio::test]
async fn scoring_stops_when_the_budget_is_gone() {
let backend = Arc::new(MockBackend::new());
// First batch alone blows a $0.05 ceiling ($0.14 per 1M input tokens).
backend.push(
r#"{"articles":[{"id":1,"score":9,"category":"Top Stories","rationale":"great"}]}"#,
TokenUsage {
input_tokens: 1_000_000,
cached_tokens: 0,
output_tokens: 0,
},
);
backend.push(
r#"{"articles":[{"id":2,"score":9,"category":"Top Stories","rationale":"great"}]}"#,
TokenUsage::default(),
);
let llm = client(Arc::clone(&backend), 0.05);
let mut candidates = vec![candidate(1, "One", 900), candidate(2, "Two", 900)];
let scored = score_all(&llm, &mut candidates, 1, &sections(), 0.3)
.await
.expect("scoring");
assert_eq!(scored, 1, "only the first batch ran");
assert_eq!(backend.calls(), 1);
assert!(llm.meter.budget_exceeded());
}
}
-998
View File
@@ -1,998 +0,0 @@
//! Stage B — lineup selection (spec §3.6).
//!
//! One call: send the top ~40 candidates by [`ScoredArticle::combined_score`] with
//! their rationales; the model returns the final 15–25 picks, each with a section
//! from the configured palette, an ordering, and exactly one `lead_story`.
//!
//! The model's answer is treated as a proposal, never as gospel: sections are
//! validated against the palette, the lead is forced to be unique, auto-include
//! feeds are re-inserted if they were dropped, and the size is clamped to
//! `target_article_count ± 5`.
use std::collections::{BTreeMap, HashMap, HashSet};
use std::fmt::Write as _;
use jiff::civil::Date;
use serde::{Deserialize, Serialize};
use super::llm::{LlmClient, LlmError, strip_code_fence};
use super::{prompt_text, truncate_words};
use crate::types::{ArticleId, Lineup, Pick, ScoredArticle, SourceKind, WORLD_BRIEFING_SECTION};
/// How many candidates are offered to stage B (§3.6).
pub const SHORTLIST_SIZE: usize = 40;
/// How far the final count may drift from `target_article_count` (§3.6: 15–25
/// around a default target of 20).
pub const TARGET_TOLERANCE: usize = 5;
/// Words of lead-in text shown per candidate in the stage-B prompt.
const BLURB_WORDS: usize = 45;
/// One element of the stage-B JSON response (§3.6).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SelectionItem {
pub id: ArticleId,
/// Must be one of `curation.sections`.
pub section: String,
pub position: i64,
#[serde(default)]
pub lead_story: bool,
}
/// Envelope the model is asked to return.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SelectionResponse {
#[serde(default)]
pub picks: Vec<SelectionItem>,
}
/// The invariant instruction block for stage B (§3.6).
pub const SELECT_INSTRUCTIONS: &str = "\
TASK: assemble today's issue of The Daily EPUB from the shortlist below.
You are choosing what one specific reader — the profile in your system prompt — \
will actually read on an e-ink screen over breakfast. Build a paper, not a \
ranking: it should have a shape, a range of subjects, and a clear front page.
RULES
1. Pick articles by id from the shortlist only. Never invent an id.
2. Give every pick a section from the palette below, spelled exactly as given.
3. Number picks within each section from 1 upward, best first.
4. Flag exactly one pick as \"lead_story\": true — the day's strongest, most \
substantial piece. It must sit in the first section you use.
5. Any candidate marked \"always-include\" MUST appear; place it in \"From the \
Blogroll\" unless it clearly belongs elsewhere.
6. Do not select two articles that tell the same story; keep the better one.
EDITORIAL JUDGEMENT
- Favour depth over coverage: a slim issue of excellent pieces beats a full one \
padded with filler. Drop anything you would not defend.
- Mix the day up. Several long technical dives in a row is a bad breakfast; \
alternate register and subject across sections.
- Keep the local and ultra-niche picks — a Boston story and a small-scene story \
are worth more here than a third AI-industry item.
- Score is evidence, not an instruction: overrule it when the paper reads better \
for it, and say so through your placement.
- Leave a section out entirely rather than padding it; empty sections are dropped.
Return JSON exactly in this shape and nothing else:
{\"picks\": [{\"id\": 123, \"section\": \"Top Stories\", \"position\": 1, \
\"lead_story\": true}]}";
/// Render the stage-B user prompt (§3.6).
pub fn build_prompt(shortlist: &[ScoredArticle], sections: &[String], target: usize) -> String {
let (min, max) = size_bounds(target);
let mut prompt = String::with_capacity(2048 + shortlist.len() * 400);
prompt.push_str(SELECT_INSTRUCTIONS);
let _ = write!(
prompt,
"\n\nSECTION PALETTE (exact strings, use only these): {}\n\
Reserved and unavailable: \"{WORLD_BRIEFING_SECTION}\" is compiled separately.\n\n\
SIZE: choose {target} articles; never fewer than {min} and never more than {max}.\n\n\
SHORTLIST ({} candidates, best-ranked first)\n",
sections.join(" | "),
shortlist.len()
);
for candidate in shortlist {
prompt.push('\n');
prompt.push_str(&render_candidate(candidate));
}
prompt
}
fn render_candidate(candidate: &ScoredArticle) -> String {
let a = &candidate.article;
let mut block = String::with_capacity(400);
let _ = writeln!(block, "--- id: {}", a.id);
let _ = writeln!(block, "title: {}", a.title.trim());
let _ = writeln!(
block,
"feed: {} · {} words (~{} min){}",
if a.feed_title.is_empty() {
"unknown"
} else {
a.feed_title.trim()
},
a.word_count,
a.reading_minutes(),
if a.excerpt_only {
" · EXCERPT ONLY"
} else {
""
}
);
match candidate.llm.as_ref() {
Some(llm) => {
let _ = writeln!(
block,
"score: {:.1} ({}) — {}",
llm.score,
if llm.category.is_empty() {
"uncategorized"
} else {
llm.category.as_str()
},
llm.rationale.trim()
);
}
None => {
let _ = writeln!(
block,
"score: unscored (heuristic rank {:.0}/100)",
candidate.prefilter_score
);
}
}
let _ = writeln!(
block,
"signals: social {:.2}; via {}{}",
candidate.social_score,
source_kinds(candidate),
if candidate.auto_include {
"; ALWAYS-INCLUDE"
} else {
""
}
);
let blurb = truncate_words(&prompt_text(&a.content_html), BLURB_WORDS);
if !blurb.is_empty() {
let _ = writeln!(block, "opening: {blurb}");
}
block
}
fn source_kinds(candidate: &ScoredArticle) -> String {
let mut kinds: Vec<&str> = candidate
.article
.sources
.iter()
.map(|s| match s.kind {
SourceKind::Scour => "scour",
SourceKind::HnFrontpage => "hn_frontpage",
SourceKind::Lobsters => "lobsters",
SourceKind::Reddit => "reddit",
SourceKind::Feed => "feed",
})
.collect();
kinds.sort_unstable();
kinds.dedup();
if kinds.is_empty() {
"feed".into()
} else {
kinds.join("+")
}
}
/// `target ± TARGET_TOLERANCE`, floored at one article (§3.6).
pub fn size_bounds(target: usize) -> (usize, usize) {
let target = target.max(1);
(
target.saturating_sub(TARGET_TOLERANCE).max(1),
target + TARGET_TOLERANCE,
)
}
// ---------------------------------------------------------------------------
// Section validation (§3.6: the model may only use the configured palette)
// ---------------------------------------------------------------------------
/// The section unrecognized labels fall back to.
pub fn default_section(sections: &[String]) -> String {
sections
.iter()
.find(|s| s.as_str() == "Top Stories")
.or_else(|| sections.first())
.cloned()
.unwrap_or_else(|| "Top Stories".to_string())
}
/// Map whatever the model said onto the configured palette (§3.6).
///
/// Exact match → case-insensitive match → best word-overlap match → default.
pub fn resolve_section(raw: &str, sections: &[String]) -> String {
let candidate = raw.trim();
if candidate.is_empty() || candidate.eq_ignore_ascii_case(WORLD_BRIEFING_SECTION) {
return default_section(sections);
}
if let Some(exact) = sections.iter().find(|s| s.as_str() == candidate) {
return exact.clone();
}
if let Some(ci) = sections.iter().find(|s| s.eq_ignore_ascii_case(candidate)) {
return ci.clone();
}
let wanted = words_of(candidate);
let best = sections
.iter()
.map(|s| (s, words_of(s).intersection(&wanted).count()))
.filter(|(_, overlap)| *overlap > 0)
.max_by_key(|(_, overlap)| *overlap);
match best {
Some((section, _)) => {
tracing::debug!(raw = candidate, mapped = %section, "mapped an off-palette section");
section.clone()
}
None => {
tracing::warn!(raw = candidate, "unknown section; using the default");
default_section(sections)
}
}
}
fn words_of(s: &str) -> HashSet<String> {
s.split(|c: char| !c.is_alphanumeric())
.filter(|w| w.len() > 2 && !w.eq_ignore_ascii_case("and"))
.map(str::to_lowercase)
.collect()
}
/// Section guess from feed metadata, used by `--skip-llm` and by top-ups (§3.6).
pub fn heuristic_section(candidate: &ScoredArticle, sections: &[String]) -> String {
if candidate.auto_include {
return resolve_section("From the Blogroll", sections);
}
let haystack = format!(
"{} {} {}",
candidate.article.category.as_deref().unwrap_or_default(),
candidate.article.feed_title,
candidate.article.title
)
.to_lowercase();
const RULES: &[(&str, &[&str])] = &[
(
"Boston & Local",
&[
"boston",
"massachusetts",
"cambridge",
"mbta",
"new england",
],
),
(
"AI & Machine Learning",
&[
" ai ",
"ai:",
"llm",
"machine learning",
"neural",
"openai",
"anthropic",
"gpt",
"diffusion",
],
),
(
"Science & Space",
&[
"science",
"space",
"nasa",
"astronom",
"physics",
"biology",
"climate",
"aerospace",
],
),
(
"Culture & Essays",
&[
"essay", "culture", "book", "fiction", "poetry", "film", "music", "art", "review",
],
),
(
"Niche Corner",
&[
"hobby",
"retro",
"keyboard",
"board game",
"coffee",
"e-ink",
"eink",
],
),
(
"Tech & Engineering",
&[
"tech",
"programming",
"engineering",
"software",
"developer",
"rust",
"linux",
"database",
"systems",
"web",
],
),
];
for (section, needles) in RULES {
if needles.iter().any(|n| haystack.contains(n))
&& let Some(found) = sections.iter().find(|s| s.as_str() == *section)
{
return found.clone();
}
}
default_section(sections)
}
// ---------------------------------------------------------------------------
// Response parsing
// ---------------------------------------------------------------------------
/// Keys the model might wrap the array in.
const ARRAY_KEYS: &[&str] = &["picks", "lineup", "articles", "selection", "items"];
/// Lenient parse of the stage-B response (§3.6).
pub fn parse_selection_response(raw: &str) -> Vec<SelectionItem> {
let cleaned = strip_code_fence(raw);
let value: serde_json::Value = match serde_json::from_str(cleaned) {
Ok(v) => v,
Err(e) => {
tracing::warn!(error = %e, "stage B response was not JSON");
return Vec::new();
}
};
let array = match &value {
serde_json::Value::Array(items) => Some(items),
serde_json::Value::Object(map) => ARRAY_KEYS
.iter()
.find_map(|k| map.get(*k).and_then(serde_json::Value::as_array))
.or_else(|| map.values().find_map(serde_json::Value::as_array)),
_ => None,
};
let Some(array) = array else {
tracing::warn!("stage B response contained no array of picks");
return Vec::new();
};
let mut out = Vec::with_capacity(array.len());
for (idx, item) in array.iter().enumerate() {
let Some(obj) = item.as_object() else {
tracing::warn!("skipping a non-object stage B pick");
continue;
};
let Some(id) = obj.get("id").and_then(|v| {
v.as_i64()
.or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
}) else {
tracing::warn!("skipping a stage B pick without an id");
continue;
};
out.push(SelectionItem {
id,
section: obj
.get("section")
.and_then(serde_json::Value::as_str)
.unwrap_or_default()
.trim()
.to_string(),
position: obj
.get("position")
.and_then(|v| {
v.as_i64()
.or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
})
.unwrap_or(idx as i64 + 1),
lead_story: obj
.get("lead_story")
.or_else(|| obj.get("is_lead"))
.and_then(|v| {
v.as_bool()
.or_else(|| v.as_str().map(|s| s.eq_ignore_ascii_case("true")))
})
.unwrap_or(false),
});
}
out
}
// ---------------------------------------------------------------------------
// Stage driver
// ---------------------------------------------------------------------------
/// Ask the model for the day's lineup, validating that every section is from the
/// configured palette and exactly one pick is the lead (§3.6).
pub async fn select(
llm: &LlmClient,
candidates: Vec<ScoredArticle>,
sections: &[String],
target: usize,
date: Date,
) -> Result<Lineup, LlmError> {
if candidates.is_empty() {
tracing::warn!("stage B had no candidates");
return Ok(Lineup {
date,
picks: Vec::new(),
section_order: Vec::new(),
});
}
if let Err(e) = llm.meter.check_budget() {
tracing::error!(error = %e,
"COST CEILING HIT before stage B selection — falling back to heuristic ranking");
return Ok(select_without_llm(candidates, sections, target, date));
}
let shortlist = shortlist(&candidates, target);
let prompt = build_prompt(&shortlist, sections, target);
tracing::debug!(
shortlist = shortlist.len(),
approx_tokens = super::approx_tokens(&prompt),
"stage B request"
);
let raw = llm.complete(&prompt, llm_temperature(), true).await?;
let items = parse_selection_response(&raw);
if items.is_empty() {
tracing::error!("stage B returned no usable picks; falling back to heuristic ranking");
return Ok(select_without_llm(candidates, sections, target, date));
}
let by_id: HashMap<ArticleId, &ScoredArticle> =
candidates.iter().map(|c| (c.article.id, c)).collect();
let mut chosen: Vec<(SelectionItem, ScoredArticle)> = Vec::with_capacity(items.len());
let mut seen: HashSet<ArticleId> = HashSet::new();
for item in items {
if !seen.insert(item.id) {
tracing::warn!(id = item.id, "stage B picked the same article twice");
continue;
}
match by_id.get(&item.id) {
Some(candidate) => chosen.push((item, (*candidate).clone())),
None => tracing::warn!(id = item.id, "stage B invented an id that was not offered"),
}
}
// Auto-include feeds can never be dropped (§3.5).
for candidate in &candidates {
if candidate.auto_include && seen.insert(candidate.article.id) {
tracing::info!(
id = candidate.article.id,
title = %candidate.article.title,
"re-inserting an always-include article the model dropped"
);
chosen.push((
SelectionItem {
id: candidate.article.id,
section: "From the Blogroll".into(),
position: i64::MAX,
lead_story: false,
},
candidate.clone(),
));
}
}
let lineup = assemble(chosen, &candidates, sections, target, date);
tracing::info!(
picks = lineup.picks.len(),
sections = lineup.section_order.len(),
lead = lineup.lead().map(|p| p.article.id),
"stage B lineup ready"
);
Ok(lineup)
}
/// Stage B runs at the scoring temperature: this is a judgement call, not prose.
fn llm_temperature() -> f32 {
0.4
}
/// Top [`SHORTLIST_SIZE`] candidates by combined score, always including the
/// auto-includes (§3.6).
fn shortlist(candidates: &[ScoredArticle], target: usize) -> Vec<ScoredArticle> {
let mut ranked: Vec<ScoredArticle> = candidates.to_vec();
sort_by_combined(&mut ranked);
let keep = SHORTLIST_SIZE.max(target * 2);
if ranked.len() <= keep {
return ranked;
}
let (head, tail) = ranked.split_at(keep);
let mut out = head.to_vec();
out.extend(tail.iter().filter(|c| c.auto_include).cloned());
out
}
fn sort_by_combined(candidates: &mut [ScoredArticle]) {
candidates.sort_by(|a, b| {
b.combined_score()
.partial_cmp(&a.combined_score())
.unwrap_or(std::cmp::Ordering::Equal)
.then_with(|| a.article.id.cmp(&b.article.id))
});
}
/// Turn validated picks into a [`Lineup`]: clamp the size, force a single lead,
/// order the sections and renumber positions (§3.6).
fn assemble(
mut chosen: Vec<(SelectionItem, ScoredArticle)>,
all: &[ScoredArticle],
sections: &[String],
target: usize,
date: Date,
) -> Lineup {
let (min, max) = size_bounds(target);
// Too many: drop the weakest non-auto-include picks.
if chosen.len() > max {
chosen.sort_by(|a, b| {
b.1.auto_include.cmp(&a.1.auto_include).then_with(|| {
b.1.combined_score()
.partial_cmp(&a.1.combined_score())
.unwrap_or(std::cmp::Ordering::Equal)
})
});
let dropped = chosen.len() - max;
chosen.truncate(max);
tracing::info!(dropped, max, "trimmed the lineup to the size ceiling");
}
// Too few: top up from the best unpicked candidates.
if chosen.len() < min {
let taken: HashSet<ArticleId> = chosen.iter().map(|(i, _)| i.id).collect();
let mut rest: Vec<ScoredArticle> = all
.iter()
.filter(|c| !taken.contains(&c.article.id))
.cloned()
.collect();
sort_by_combined(&mut rest);
let wanted = min - chosen.len();
let added = rest.len().min(wanted);
for candidate in rest.into_iter().take(wanted) {
let section = heuristic_section(&candidate, sections);
chosen.push((
SelectionItem {
id: candidate.article.id,
section,
position: i64::MAX,
lead_story: false,
},
candidate,
));
}
tracing::info!(added, min, "topped the lineup up to the size floor");
}
// Normalize sections and pick the section order.
for (item, _) in chosen.iter_mut() {
item.section = resolve_section(&item.section, sections);
}
let used: HashSet<&str> = chosen.iter().map(|(i, _)| i.section.as_str()).collect();
let mut section_order: Vec<String> = sections
.iter()
.filter(|s| used.contains(s.as_str()))
.cloned()
.collect();
for (item, _) in &chosen {
if !section_order.contains(&item.section) {
section_order.push(item.section.clone());
}
}
let section_rank: HashMap<&str, usize> = section_order
.iter()
.enumerate()
.map(|(i, s)| (s.as_str(), i))
.collect();
// Order: section, then the model's position, then quality, then id.
chosen.sort_by(|a, b| {
section_rank
.get(a.0.section.as_str())
.cmp(&section_rank.get(b.0.section.as_str()))
.then_with(|| a.0.position.cmp(&b.0.position))
.then_with(|| {
b.1.combined_score()
.partial_cmp(&a.1.combined_score())
.unwrap_or(std::cmp::Ordering::Equal)
})
.then_with(|| a.1.article.id.cmp(&b.1.article.id))
});
// Exactly one lead, and it must live in the first section (§3.6 rule 4).
let lead_id = chosen
.iter()
.find(|(item, _)| item.lead_story)
.filter(|(item, _)| section_rank.get(item.section.as_str()) == Some(&0))
.or_else(|| chosen.first())
.map(|(item, _)| item.id);
let mut per_section: BTreeMap<String, i64> = BTreeMap::new();
let picks = chosen
.into_iter()
.map(|(item, candidate)| {
let position = per_section
.entry(item.section.clone())
.and_modify(|n| *n += 1)
.or_insert(1);
Pick {
section: item.section,
position: *position,
is_lead: Some(item.id) == lead_id,
summary: None,
llm: candidate.llm.clone(),
discussion: None,
article: candidate.article,
}
})
.collect();
Lineup {
date,
picks,
section_order,
}
}
/// `--skip-llm` fallback: take the top `target` by prefilter score and bucket them
/// into sections by feed category (notes §6).
pub fn select_without_llm(
candidates: Vec<ScoredArticle>,
sections: &[String],
target: usize,
date: Date,
) -> Lineup {
let mut ranked = candidates.clone();
super::prefilter::sort_by_prefilter(&mut ranked);
let mut chosen: Vec<(SelectionItem, ScoredArticle)> = Vec::new();
let mut seen: HashSet<ArticleId> = HashSet::new();
for candidate in ranked.into_iter() {
let auto = candidate.auto_include;
if chosen.len() >= target && !auto {
continue;
}
if !seen.insert(candidate.article.id) {
continue;
}
let section = heuristic_section(&candidate, sections);
chosen.push((
SelectionItem {
id: candidate.article.id,
section,
position: chosen.len() as i64 + 1,
lead_story: false,
},
candidate,
));
}
let lineup = assemble(chosen, &candidates, sections, target, date);
tracing::info!(
picks = lineup.picks.len(),
sections = lineup.section_order.len(),
"skip-llm lineup ready"
);
lineup
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::{CurationConfig, DeepseekConfig};
use crate::curate::llm::{MockBackend, UsageMeter};
use crate::curate::prefilter::tests::article;
use crate::types::{LlmScore, TokenUsage};
use std::sync::Arc;
const LINEUP_FIXTURE: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/deepseek_lineup.json"
));
fn sections() -> Vec<String> {
CurationConfig::default().sections
}
fn date() -> Date {
"2026-08-15".parse().expect("date")
}
fn candidate(id: i64, title: &str, words: i64, score: f64) -> ScoredArticle {
ScoredArticle {
article: article(id, title, words),
prefilter_score: 40.0 + score,
social_score: 1.0,
llm: Some(LlmScore {
score,
category: "Tech & Engineering".into(),
rationale: "solid".into(),
is_paywalled_guess: false,
}),
auto_include: false,
}
}
fn candidates(n: i64) -> Vec<ScoredArticle> {
(1..=n)
.map(|i| {
candidate(
i,
&format!("Article {i}"),
500 + i * 10,
(10.0 - i as f64 * 0.1).max(0.0),
)
})
.collect()
}
#[test]
fn section_resolution_maps_onto_the_palette() {
let s = sections();
assert_eq!(resolve_section("Top Stories", &s), "Top Stories");
assert_eq!(resolve_section(" top stories ", &s), "Top Stories");
assert_eq!(
resolve_section("Technology & Engineering", &s),
"Tech & Engineering"
);
assert_eq!(resolve_section("Science", &s), "Science & Space");
assert_eq!(resolve_section("Sports", &s), "Top Stories");
assert_eq!(resolve_section("", &s), "Top Stories");
// The reserved section is never allowed through (§3.6).
assert_eq!(resolve_section(WORLD_BRIEFING_SECTION, &s), "Top Stories");
// A palette without "Top Stories" falls back to its first entry.
let tiny = vec!["Niche Corner".to_string()];
assert_eq!(resolve_section("Whatever", &tiny), "Niche Corner");
}
#[test]
fn heuristic_sections_follow_feed_metadata() {
let s = sections();
let mut c = candidate(1, "MBTA slow zones, charted", 900, 6.0);
c.article.category = Some("News".into());
assert_eq!(heuristic_section(&c, &s), "Boston & Local");
let mut ai = candidate(2, "A new LLM benchmark", 900, 6.0);
ai.article.category = Some("Machine Learning".into());
assert_eq!(heuristic_section(&ai, &s), "AI & Machine Learning");
let mut blog = candidate(3, "Notes from my week", 900, 6.0);
blog.auto_include = true;
assert_eq!(heuristic_section(&blog, &s), "From the Blogroll");
let mut plain = candidate(4, "Untitled musing", 900, 6.0);
plain.article.category = None;
plain.article.feed_title = "A Journal".into();
assert_eq!(heuristic_section(&plain, &s), "Top Stories");
}
#[test]
fn parses_a_realistic_lineup_response() {
let items = parse_selection_response(LINEUP_FIXTURE);
assert_eq!(items.len(), 6);
assert_eq!(items[0].id, 101);
assert!(items[0].lead_story);
assert_eq!(items[0].section, "Top Stories");
assert_eq!(items.iter().filter(|i| i.lead_story).count(), 1);
// Junk entries in the fixture are dropped, not fatal.
assert!(items.iter().all(|i| i.id != 0));
}
#[test]
fn size_bounds_follow_the_spec() {
assert_eq!(size_bounds(20), (15, 25));
assert_eq!(size_bounds(6), (1, 11));
assert_eq!(size_bounds(0), (1, 6));
}
#[tokio::test]
async fn selection_builds_a_valid_lineup() {
let backend = Arc::new(MockBackend::new());
backend.push(LINEUP_FIXTURE, TokenUsage::default());
let llm = LlmClient::with_backend(
"deepseek-v4-flash",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), 2.0),
backend.clone(),
);
// ids 101..=112 so the fixture's picks resolve.
let pool: Vec<ScoredArticle> = (101..=112)
.map(|i| candidate(i, &format!("Article {i}"), 800, 7.0))
.collect();
let lineup = select(&llm, pool, &sections(), 6, date())
.await
.expect("selection");
assert_eq!(lineup.date, date());
assert_eq!(lineup.picks.len(), 6);
assert_eq!(lineup.picks.iter().filter(|p| p.is_lead).count(), 1);
assert_eq!(lineup.lead().map(|p| p.article.id), Some(101));
// Every section is from the palette and non-empty.
for section in &lineup.section_order {
assert!(sections().contains(section), "{section} is off-palette");
assert!(!lineup.section_picks(section).is_empty());
}
// Positions restart at 1 inside each section and ascend.
for section in &lineup.section_order {
let positions: Vec<i64> = lineup
.section_picks(section)
.iter()
.map(|p| p.position)
.collect();
assert_eq!(
positions,
(1..=positions.len() as i64).collect::<Vec<_>>(),
"{section} positions"
);
}
// The lead sits in the first section used.
assert_eq!(
lineup.lead().map(|p| p.section.clone()),
lineup.section_order.first().cloned()
);
// The prompt carried the shortlist and the palette.
let prompt = &backend.prompts()[0].user;
assert!(prompt.starts_with(SELECT_INSTRUCTIONS));
assert!(prompt.contains("--- id: 101"));
assert!(prompt.contains("never fewer than 1 and never more than 11"));
}
#[tokio::test]
async fn hallucinated_ids_and_missing_leads_are_repaired() {
let backend = Arc::new(MockBackend::new());
backend.push(
r#"{"picks":[{"id":9999,"section":"Top Stories","position":1,"lead_story":true},
{"id":1,"section":"Sportsball","position":2},
{"id":2,"section":"Niche Corner","position":1}]}"#,
TokenUsage::default(),
);
let llm = LlmClient::with_backend(
"deepseek-v4-flash",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), 2.0),
backend,
);
let lineup = select(&llm, candidates(6), &sections(), 2, date())
.await
.expect("selection");
assert!(lineup.picks.iter().all(|p| p.article.id != 9999));
assert_eq!(lineup.picks.iter().filter(|p| p.is_lead).count(), 1);
for pick in &lineup.picks {
assert!(sections().contains(&pick.section));
}
}
#[tokio::test]
async fn oversized_and_undersized_answers_are_clamped() {
// Undersized: the model returns one pick but the floor is 5.
let backend = Arc::new(MockBackend::new());
backend.push(
r#"{"picks":[{"id":1,"section":"Top Stories","position":1,"lead_story":true}]}"#,
TokenUsage::default(),
);
let llm = LlmClient::with_backend(
"deepseek-v4-flash",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), 2.0),
backend,
);
let lineup = select(&llm, candidates(30), &sections(), 10, date())
.await
.expect("selection");
assert!(lineup.picks.len() >= 5, "{}", lineup.picks.len());
// Oversized: 30 picks against a target of 6 (ceiling 11).
let picks: Vec<String> = (1..=30)
.map(|i| format!(r#"{{"id":{i},"section":"Top Stories","position":{i}}}"#))
.collect();
let backend = Arc::new(MockBackend::new());
backend.push(
format!(r#"{{"picks":[{}]}}"#, picks.join(",")),
TokenUsage::default(),
);
let llm = LlmClient::with_backend(
"deepseek-v4-flash",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), 2.0),
backend,
);
let lineup = select(&llm, candidates(30), &sections(), 6, date())
.await
.expect("selection");
assert_eq!(lineup.picks.len(), 11);
}
#[tokio::test]
async fn always_include_articles_are_reinserted() {
let backend = Arc::new(MockBackend::new());
backend.push(
r#"{"picks":[{"id":1,"section":"Top Stories","position":1,"lead_story":true}]}"#,
TokenUsage::default(),
);
let llm = LlmClient::with_backend(
"deepseek-v4-flash",
"SYSTEM".into(),
UsageMeter::new(&DeepseekConfig::default(), 2.0),
backend,
);
let mut pool = candidates(3);
pool[2].auto_include = true;
let lineup = select(&llm, pool, &sections(), 1, date())
.await
.expect("selection");
let ids: Vec<ArticleId> = lineup.picks.iter().map(|p| p.article.id).collect();
assert!(ids.contains(&3), "auto-include must survive: {ids:?}");
assert_eq!(
lineup
.picks
.iter()
.find(|p| p.article.id == 3)
.map(|p| p.section.as_str()),
Some("From the Blogroll")
);
}
#[tokio::test]
async fn a_tripped_budget_falls_back_without_calling_the_model() {
let backend = Arc::new(MockBackend::new());
let meter = UsageMeter::new(&DeepseekConfig::default(), 0.001);
meter.record(TokenUsage {
input_tokens: 1_000_000,
cached_tokens: 0,
output_tokens: 0,
});
let llm =
LlmClient::with_backend("deepseek-v4-flash", "SYSTEM".into(), meter, backend.clone());
let lineup = select(&llm, candidates(20), &sections(), 6, date())
.await
.expect("fallback");
assert_eq!(backend.calls(), 0);
assert_eq!(lineup.picks.len(), 6);
}
#[test]
fn skip_llm_lineup_uses_prefilter_order() {
let mut pool = candidates(10);
pool.iter_mut().for_each(|c| c.llm = None);
pool[7].prefilter_score = 99.0; // id 8 is the strongest heuristically
pool[9].auto_include = true; // id 10 is a personal blog
pool[9].prefilter_score = 1.0;
let lineup = select_without_llm(pool, &sections(), 4, date());
assert_eq!(lineup.picks.len(), 5, "4 picks + the auto-include");
assert_eq!(lineup.lead().map(|p| p.article.id), Some(8));
assert_eq!(lineup.picks.iter().filter(|p| p.is_lead).count(), 1);
assert!(
lineup
.picks
.iter()
.any(|p| p.article.id == 10 && p.section == "From the Blogroll")
);
for pick in &lineup.picks {
assert!(sections().contains(&pick.section));
assert!(pick.summary.is_none());
}
assert!(!lineup.section_order.is_empty());
}
#[test]
fn empty_input_yields_an_empty_lineup() {
let lineup = select_without_llm(Vec::new(), &sections(), 20, date());
assert!(lineup.picks.is_empty());
assert!(lineup.section_order.is_empty());
assert!(lineup.lead().is_none());
}
}
+981
View File
@@ -0,0 +1,981 @@
//! Cheap per-article ranking signals, the mid-rank percentile normalizer and the
//! preliminary blend (plan §9, §12.2, §12.4).
//!
//! Every signal is an `Option<f64>`: `None` means *absent*, which is never a
//! numeric zero. Absent signals are left out of the percentile computation and
//! of the blend, whose remaining weights are renormalized (§12.2, §12.4).
use std::collections::{BTreeMap, HashMap, HashSet};
use jiff::Timestamp;
use serde::{Deserialize, Serialize};
use crate::config::{PreliminaryWeights, RankingConfig, VoyageConfig};
use crate::curate::embedding::{dot, load_article_embeddings};
use crate::curate::prefilter;
use crate::db::Db;
use crate::types::{Article, ArticleId, FeedId, SourceKind};
/// Below this many embedded eligible articles the z-score is too noisy, so the
/// interest signal falls back to the raw top-1 cosine (§9.1).
pub const INTEREST_ZSCORE_MIN_ARTICLES: usize = 30;
/// Standard-deviation floor for the per-interest z-score (§9.1).
const ZSCORE_STD_FLOOR: f64 = 1e-3;
/// How many interests and rated neighbours `signals_json` records (§7.5).
const RECORDED_TOP: usize = 3;
/// The signal names that go through the percentile normalizer, in the order
/// they are rendered (§12.2). LLM scores (`triage`, `quality`, `fit`) are
/// absolute and arrive in steps 4–5.
pub const PERCENTILE_SIGNALS: [&str; 5] = ["interest", "knn", "feed", "social", "heuristic"];
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct TopInterest {
pub name: String,
pub z: f64,
pub cos: f64,
}
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Neighbour {
pub article_id: ArticleId,
pub label: String,
pub cos: f64,
pub title: String,
}
/// Every cheap signal for one article, plus what the normalizer and the blend
/// derived from them (§9, §12.2, §12.4).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Signals {
pub interest: Option<f64>,
/// Raw top-1 cosine behind `interest`, recorded for `explain` (§7.5).
pub interest_top1_cos: Option<f64>,
pub knn: Option<f64>,
pub feed: Option<f64>,
pub social: Option<f64>,
pub heuristic: Option<f64>,
/// Mid-rank percentiles of the present signals (§12.2).
#[serde(default)]
pub norm: BTreeMap<String, f64>,
/// Effective preliminary weights after gating and renormalization (§12.4).
#[serde(default)]
pub weights: BTreeMap<String, f64>,
#[serde(default)]
pub top_interests: Vec<TopInterest>,
#[serde(default)]
pub neighbours: Vec<Neighbour>,
#[serde(default)]
pub notes: Vec<String>,
/// Preliminary blend on a 0–100 scale; `None` when nothing is present.
pub preliminary: Option<f64>,
/// Gate ramps applied to the learned signals' weights (§9.2, §9.3).
#[serde(skip)]
pub knn_gate: f64,
#[serde(skip)]
pub feed_gate: f64,
}
impl Signals {
/// The raw value of a named signal, `None` when absent or unknown.
pub fn raw(&self, name: &str) -> Option<f64> {
match name {
"interest" => self.interest,
"interest_top1_cos" => self.interest_top1_cos,
"knn" => self.knn,
"feed" => self.feed,
"social" => self.social,
"heuristic" => self.heuristic,
_ => None,
}
}
pub fn present(&self, name: &str) -> bool {
self.raw(name).is_some()
}
/// The signals every eligible article gets without embeddings or ratings.
pub fn baseline(article: &Article) -> Self {
Self {
social: (!article.social.is_empty()).then(|| article.social_score()),
heuristic: Some(prefilter::text_heuristic(article)),
..Self::default()
}
}
}
/// What the run log and the report say about the learned signals (§9.2, §15.4).
#[derive(Debug, Clone, Copy, Default, PartialEq)]
pub struct PreferenceSummary {
pub rated_with_embeddings: usize,
pub attributable_feed_ratings: usize,
pub knn_gate: f64,
pub feed_gate: f64,
}
/// One rated article with an embedding: the unit of the preference state (§9.2).
#[derive(Debug, Clone, PartialEq)]
pub struct RatedExample {
pub article_id: ArticleId,
pub label: String,
pub title: String,
/// The vote's value (`loved` 1.0, `good` 0.35, `not_for_me` −1.0).
pub value: f64,
/// `0.5 ^ (age_days / half_life_days)` at the time of the run.
pub decay: f64,
pub embedding: Vec<f32>,
/// Distinct direct feeds that carried the rated article (§9.3).
pub feeds: Vec<FeedId>,
}
impl RatedExample {
/// `weight_i = value_i × decay_i` (§9.2).
pub fn weight(&self) -> f64 {
self.value * self.decay
}
}
#[derive(Debug, Clone, Copy, Default, PartialEq)]
struct FeedRate {
up: f64,
down: f64,
}
impl FeedRate {
/// Beta-smoothed rate `(up + 1) / (up + down + 2)` (§9.3).
fn rate(self) -> f64 {
(self.up + 1.0) / (self.up + self.down + 2.0)
}
}
/// Rated-neighbour and feed-affinity state, built once per run (§9.2, §9.3).
#[derive(Debug, Clone, Default)]
pub struct PreferenceState {
pub examples: Vec<RatedExample>,
feed_rates: HashMap<FeedId, FeedRate>,
pub attributable_feed_ratings: usize,
pub knn_gate: f64,
pub feed_gate: f64,
}
impl PreferenceState {
/// Build the state from already-loaded examples (pure; tests use this).
pub fn build(examples: Vec<RatedExample>, ranking: &RankingConfig) -> Self {
let (feed_rates, attributable_feed_ratings) = feed_rates(&examples);
Self {
knn_gate: gate(examples.len(), ranking.knn_floor, ranking.knn_full),
feed_gate: gate(
attributable_feed_ratings,
ranking.feed_floor,
ranking.feed_full,
),
examples,
feed_rates,
attributable_feed_ratings,
}
}
/// Load `db::current_ratings(rating_lookback_days)` joined to
/// `article_embeddings`; ratings without an embedding are skipped (§9.2).
pub async fn load(
db: &Db,
voyage: &VoyageConfig,
ranking: &RankingConfig,
now: Timestamp,
) -> anyhow::Result<Self> {
let ratings = db.current_ratings(ranking.rating_lookback_days).await?;
let ids = ratings
.iter()
.map(|rating| rating.article_id)
.collect::<Vec<_>>();
let embeddings = load_article_embeddings(db, voyage, &ids).await?;
let mut examples = Vec::new();
for rating in ratings {
let Some(embedding) = embeddings.get(&rating.article_id).cloned() else {
continue;
};
let feeds = db
.get_article(rating.article_id)
.await?
.as_ref()
.map(direct_feeds)
.unwrap_or_default();
let age_days = (now.as_second() - rating.event_at.as_second()).max(0) as f64 / 86_400.0;
examples.push(RatedExample {
article_id: rating.article_id,
label: rating.label,
title: rating.title,
value: rating.value,
decay: decay(age_days, ranking.rating_half_life_days),
embedding,
feeds,
});
}
Ok(Self::build(examples, ranking))
}
pub fn summary(&self) -> PreferenceSummary {
PreferenceSummary {
rated_with_embeddings: self.examples.len(),
attributable_feed_ratings: self.attributable_feed_ratings,
knn_gate: self.knn_gate,
feed_gate: self.feed_gate,
}
}
/// The once-per-run log line of §9.2.
pub fn log(&self, ranking: &RankingConfig) {
let feed_detail = if self.feed_gate > 0.0 {
format!("(n={})", self.attributable_feed_ratings)
} else {
format!(
"(n={} < {})",
self.attributable_feed_ratings, ranking.feed_floor
)
};
tracing::info!(
rated_with_embeddings = self.examples.len(),
knn_gate = self.knn_gate,
feed_gate = self.feed_gate,
"preference: {} rated articles with embeddings → knn gate {:.2}; feed gate {:.1} {}",
self.examples.len(),
self.knn_gate,
self.feed_gate,
feed_detail
);
}
/// Signed rated-neighbour preference and the three nearest rated articles
/// (§9.2). Absent when the gate is closed or there are no examples.
pub fn knn(&self, candidate: &[f32], ranking: &RankingConfig) -> (Option<f64>, Vec<Neighbour>) {
if self.knn_gate <= 0.0 || self.examples.is_empty() {
return (None, Vec::new());
}
let mut scored = self
.examples
.iter()
.filter_map(|example| {
dot(candidate, &example.embedding)
.ok()
.map(|s| (s, example))
})
.collect::<Vec<_>>();
scored.sort_by(|left, right| right.0.total_cmp(&left.0));
let side = |positive: bool| -> Option<f64> {
let chosen = scored
.iter()
.filter(|(_, example)| (example.weight() > 0.0) == positive)
.take(ranking.neighbour_k.max(1))
.collect::<Vec<_>>();
let denominator = chosen
.iter()
.map(|(_, example)| example.weight().abs())
.sum::<f64>();
(denominator > 0.0).then(|| {
chosen
.iter()
.map(|(similarity, example)| example.weight().abs() * similarity)
.sum::<f64>()
/ denominator
})
};
let positive = side(true);
let negative = side(false);
let knn = (positive.is_some() || negative.is_some()).then(|| {
positive.unwrap_or(0.0) - ranking.negative_coefficient * negative.unwrap_or(0.0)
});
let neighbours = scored
.iter()
.take(RECORDED_TOP)
.map(|(cos, example)| Neighbour {
article_id: example.article_id,
label: example.label.clone(),
cos: *cos,
title: example.title.clone(),
})
.collect();
(knn, neighbours)
}
/// Mean Beta-smoothed rate over the article's rated direct feeds (§9.3).
pub fn feed(&self, article: &Article) -> Option<f64> {
if self.feed_gate <= 0.0 {
return None;
}
let rates = direct_feeds(article)
.into_iter()
.filter_map(|feed| self.feed_rates.get(&feed))
.map(|rate| rate.rate())
.collect::<Vec<_>>();
(!rates.is_empty()).then(|| rates.iter().sum::<f64>() / rates.len() as f64)
}
/// Per-feed `(up, down)` credit, exposed for tests of §9.3.
pub fn feed_credit(&self, feed: FeedId) -> Option<(f64, f64)> {
self.feed_rates.get(&feed).map(|rate| (rate.up, rate.down))
}
}
/// `0.5 ^ (age_days / half_life_days)` (§9.2).
pub fn decay(age_days: f64, half_life_days: f64) -> f64 {
if half_life_days <= 0.0 {
return 1.0;
}
0.5f64.powf(age_days.max(0.0) / half_life_days)
}
/// `clamp((n − floor) / (full − floor), 0, 1)` (§9.2).
pub fn gate(n: usize, floor: usize, full: usize) -> f64 {
if n <= floor {
0.0
} else if n >= full || full <= floor {
1.0
} else {
(n - floor) as f64 / (full - floor) as f64
}
}
/// Distinct `SourceKind::Feed` feeds that carried the article; the best entry's
/// feed when there are none (§9.3).
pub fn direct_feeds(article: &Article) -> Vec<FeedId> {
let mut feeds = article
.sources
.iter()
.filter(|source| source.kind == SourceKind::Feed)
.map(|source| source.feed_id)
.collect::<HashSet<_>>()
.into_iter()
.collect::<Vec<_>>();
if feeds.is_empty() && article.feed_id != 0 {
feeds.push(article.feed_id);
}
feeds.sort_unstable();
feeds
}
fn feed_rates(examples: &[RatedExample]) -> (HashMap<FeedId, FeedRate>, usize) {
let mut rates: HashMap<FeedId, FeedRate> = HashMap::new();
let mut attributable = 0;
for example in examples {
if example.feeds.is_empty() {
continue;
}
attributable += 1;
let credit = example.weight() / example.feeds.len() as f64;
for feed in &example.feeds {
let rate = rates.entry(*feed).or_default();
rate.up += credit.max(0.0);
rate.down += (-credit).max(0.0);
}
}
(rates, attributable)
}
/// The interest match of §9.1 for one article.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct InterestMatch {
pub score: f64,
pub top1_cos: f64,
pub top_interests: Vec<TopInterest>,
}
/// Z-scored standing-interest match for every embedded article (§9.1).
///
/// Below [`INTEREST_ZSCORE_MIN_ARTICLES`] embedded articles the score is the raw
/// top-1 cosine instead, and that is logged.
pub fn interest_matches(
articles: &HashMap<ArticleId, Vec<f32>>,
interests: &HashMap<String, Vec<f32>>,
) -> HashMap<ArticleId, InterestMatch> {
if articles.is_empty() || interests.is_empty() {
return HashMap::new();
}
let fallback = articles.len() < INTEREST_ZSCORE_MIN_ARTICLES;
if fallback {
tracing::info!(
embedded = articles.len(),
"fewer than {INTEREST_ZSCORE_MIN_ARTICLES} embedded articles; interest uses the raw top-1 cosine"
);
}
let mut matches: HashMap<ArticleId, Vec<TopInterest>> = HashMap::new();
for (name, interest) in interests {
let similarities = articles
.iter()
.filter_map(|(article_id, article)| {
dot(interest, article).ok().map(|cos| (*article_id, cos))
})
.collect::<Vec<_>>();
if similarities.is_empty() {
continue;
}
let n = similarities.len() as f64;
let mean = similarities.iter().map(|(_, cos)| cos).sum::<f64>() / n;
let variance = similarities
.iter()
.map(|(_, cos)| (cos - mean).powi(2))
.sum::<f64>()
/ n;
let std = variance.sqrt().max(ZSCORE_STD_FLOOR);
for (article_id, cos) in similarities {
matches.entry(article_id).or_default().push(TopInterest {
name: name.clone(),
z: (cos - mean) / std,
cos,
});
}
}
matches
.into_iter()
.map(|(article_id, mut all)| {
all.sort_by(|left, right| {
right
.z
.total_cmp(&left.z)
.then_with(|| left.name.cmp(&right.name))
});
let top1_cos = all
.iter()
.map(|item| item.cos)
.fold(f64::NEG_INFINITY, f64::max);
all.truncate(RECORDED_TOP);
let score = if fallback {
top1_cos
} else {
let top_mean = all.iter().map(|item| item.z).sum::<f64>() / all.len() as f64;
0.7 * all[0].z + 0.3 * top_mean
};
(
article_id,
InterestMatch {
score,
top1_cos,
top_interests: all,
},
)
})
.collect()
}
/// Every cheap signal for the eligible set, normalized and blended (§9, §12.2,
/// §12.4). Pure: the preference state is already loaded.
pub fn compute(
articles: &[Article],
article_embeddings: &HashMap<ArticleId, Vec<f32>>,
interest_embeddings: &HashMap<String, Vec<f32>>,
preference: &PreferenceState,
ranking: &RankingConfig,
) -> HashMap<ArticleId, Signals> {
let interests = interest_matches(article_embeddings, interest_embeddings);
let mut all = articles
.iter()
.map(|article| {
let mut signals = Signals::baseline(article);
signals.knn_gate = preference.knn_gate;
signals.feed_gate = preference.feed_gate;
if let Some(matched) = interests.get(&article.id) {
signals.interest = Some(matched.score);
signals.interest_top1_cos = Some(matched.top1_cos);
signals.top_interests = matched.top_interests.clone();
}
if let Some(embedding) = article_embeddings.get(&article.id) {
let (knn, neighbours) = preference.knn(embedding, ranking);
signals.knn = knn;
signals.neighbours = neighbours;
}
signals.feed = preference.feed(article);
if preference.knn_gate > 0.0 {
signals.notes.push(format!(
"knn gate {:.2} (n={} rated with embeddings)",
preference.knn_gate,
preference.examples.len()
));
}
signals
})
.collect::<Vec<_>>();
normalize(&mut all.iter_mut().collect::<Vec<_>>());
for signals in &mut all {
preliminary_blend(signals, &ranking.weights.preliminary);
}
articles.iter().map(|article| article.id).zip(all).collect()
}
/// [`compute`] with the preference state loaded from the database.
pub async fn compute_all(
db: &Db,
articles: &[Article],
article_embeddings: &HashMap<ArticleId, Vec<f32>>,
interest_embeddings: &HashMap<String, Vec<f32>>,
voyage: &VoyageConfig,
ranking: &RankingConfig,
now: Timestamp,
) -> anyhow::Result<(HashMap<ArticleId, Signals>, PreferenceSummary)> {
let preference = PreferenceState::load(db, voyage, ranking, now).await?;
preference.log(ranking);
Ok((
compute(
articles,
article_embeddings,
interest_embeddings,
&preference,
ranking,
),
preference.summary(),
))
}
/// Mid-rank percentiles over the present values of each signal (§12.2).
///
/// `p(x) = (count_below + (count_equal + 1) / 2) / n_present`; fewer than two
/// present values or all-equal values give 0.5. Article id never breaks ties.
pub fn normalize(signals: &mut [&mut Signals]) {
for name in PERCENTILE_SIGNALS {
let mut values = signals
.iter()
.filter_map(|signal| signal.raw(name))
.collect::<Vec<_>>();
if values.is_empty() {
continue;
}
values.sort_by(f64::total_cmp);
let n = values.len() as f64;
let constant = values.len() < 2 || values.first() == values.last();
for signal in signals.iter_mut() {
let Some(value) = signal.raw(name) else {
continue;
};
let percentile = if constant {
0.5
} else {
let below = values.partition_point(|other| *other < value);
let equal = values.partition_point(|other| *other <= value) - below;
(below as f64 + (equal as f64 + 1.0) / 2.0) / n
};
signal.norm.insert(name.to_string(), percentile);
}
}
}
/// The preliminary blend of §12.4 on a 0–100 scale: present-and-active
/// signals only, learned weights multiplied by their gate, renormalized to 1.
pub fn preliminary_blend(signals: &mut Signals, configured: &PreliminaryWeights) -> Option<f64> {
let candidates = [
("interest", configured.interest, 1.0),
("knn", configured.knn, signals.knn_gate),
("heuristic", configured.heuristic, 1.0),
("feed", configured.feed, signals.feed_gate),
("social", configured.social, 1.0),
];
let active = candidates
.into_iter()
.filter_map(|(name, weight, gate)| {
let norm = *signals.norm.get(name)?;
let effective = weight * gate;
(effective > 0.0).then_some((name, effective, norm))
})
.collect::<Vec<_>>();
let total = active.iter().map(|(_, weight, _)| weight).sum::<f64>();
if total <= 0.0 {
signals.weights.clear();
signals.preliminary = None;
return None;
}
signals.weights = active
.iter()
.map(|(name, weight, _)| ((*name).to_string(), weight / total))
.collect();
let blend = active
.iter()
.map(|(_, weight, norm)| weight / total * norm)
.sum::<f64>()
* 100.0;
signals.preliminary = Some(blend);
Some(blend)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::types::{ExtractMethod, SourceRef};
fn ranking() -> RankingConfig {
RankingConfig::default()
}
fn unit(values: &[f32]) -> Vec<f32> {
let norm = values.iter().map(|v| v * v).sum::<f32>().sqrt();
values.iter().map(|v| v / norm).collect()
}
fn example(id: ArticleId, label: &str, value: f64, embedding: &[f32]) -> RatedExample {
RatedExample {
article_id: id,
label: label.into(),
title: format!("rated {id}"),
value,
decay: 1.0,
embedding: unit(embedding),
feeds: vec![id],
}
}
fn article(id: ArticleId, feeds: &[FeedId]) -> Article {
Article {
id,
canonical_url: format!("https://example.com/{id}"),
title: format!("Article {id}"),
best_entry_id: id,
content_html: String::new(),
word_count: 1000,
excerpt_only: false,
image_count: 0,
sources: feeds
.iter()
.map(|feed| SourceRef {
entry_id: id,
feed_id: *feed,
feed_title: format!("feed {feed}"),
category: None,
kind: SourceKind::Feed,
})
.collect(),
first_seen: "2026-08-15T00:00:00Z".parse().unwrap(),
url: format!("https://example.com/{id}"),
author: None,
feed_id: feeds.first().copied().unwrap_or(0),
feed_title: String::new(),
category: None,
published_at: None,
comments_url: None,
image_urls: vec![],
social: vec![],
extract_method: ExtractMethod::Miniflux,
}
}
fn with_heuristic(value: Option<f64>) -> Signals {
Signals {
heuristic: value,
..Signals::default()
}
}
// --- §9.1 interest z-scores ---
fn interest_fixture(n: usize) -> (HashMap<ArticleId, Vec<f32>>, HashMap<String, Vec<f32>>) {
// Article 1 sits on axis x; the rest sit near axis y with a tiny spread.
let mut articles = HashMap::new();
articles.insert(1, unit(&[1.0, 0.0, 0.0]));
for id in 2..=n as ArticleId {
articles.insert(id, unit(&[0.0, 1.0, 0.001 * id as f32]));
}
// "Broad" is about equally close to everything; "Specific" matches only article 1.
let mut interests = HashMap::new();
interests.insert("Broad".to_string(), unit(&[1.0, 1.0, 0.0]));
interests.insert("Specific".to_string(), unit(&[1.0, 0.0, 0.0]));
(articles, interests)
}
#[test]
fn specific_interest_with_one_strong_match_beats_a_broad_one() {
let (articles, interests) = interest_fixture(40);
let matched = interest_matches(&articles, &interests);
let strong = &matched[&1];
assert_eq!(strong.top_interests[0].name, "Specific");
assert!(
strong.top_interests[0].z > 3.0,
"z = {}",
strong.top_interests[0].z
);
let others = (2..=40)
.map(|id| matched[&id].score)
.fold(f64::NEG_INFINITY, f64::max);
assert!(strong.score > others + 2.0, "{} vs {others}", strong.score);
// Raw cosine would have called Broad a near-tie everywhere (≈0.707).
assert!((matched[&2].top1_cos - 0.707).abs() < 0.01);
}
#[test]
fn interest_falls_back_to_raw_cosine_under_thirty_articles() {
let (articles, interests) = interest_fixture(10);
let matched = interest_matches(&articles, &interests);
for (id, m) in &matched {
assert!(
(m.score - m.top1_cos).abs() < 1e-9,
"article {id} should use raw top-1"
);
}
assert!((matched[&2].score - 0.707).abs() < 0.01);
}
// --- §9.2 preference ---
#[test]
fn one_loved_article_gives_a_positive_knn_to_a_near_neighbour() {
let mut ranking = ranking();
ranking.knn_floor = 0;
ranking.knn_full = 1;
let state = PreferenceState::build(vec![example(1, "loved", 1.0, &[1.0, 0.0])], &ranking);
let (knn, neighbours) = state.knn(&unit(&[0.9, 0.1]), &ranking);
assert!(knn.unwrap() > 0.9);
assert_eq!(neighbours.len(), 1);
assert_eq!(neighbours[0].label, "loved");
let (far, _) = state.knn(&unit(&[0.0, 1.0]), &ranking);
assert!(far.unwrap().abs() < 1e-6);
}
#[test]
fn two_unrelated_loved_clusters_both_score_high() {
let mut ranking = ranking();
ranking.knn_floor = 0;
ranking.knn_full = 1;
ranking.neighbour_k = 2;
let state = PreferenceState::build(
vec![
example(1, "loved", 1.0, &[1.0, 0.0, 0.0]),
example(2, "loved", 1.0, &[0.98, 0.02, 0.0]),
example(3, "loved", 1.0, &[0.0, 1.0, 0.0]),
example(4, "loved", 1.0, &[0.0, 0.98, 0.02]),
],
&ranking,
);
let (near_a, _) = state.knn(&unit(&[1.0, 0.0, 0.0]), &ranking);
let (near_b, _) = state.knn(&unit(&[0.0, 1.0, 0.0]), &ranking);
assert!(near_a.unwrap() > 0.95, "{near_a:?}");
assert!(near_b.unwrap() > 0.95, "{near_b:?}");
// A centroid would have put both at ~0.7.
}
#[test]
fn good_carries_a_third_of_loved() {
let loved = example(1, "loved", 1.0, &[1.0, 0.0]);
let good = example(2, "good", 0.35, &[1.0, 0.0]);
assert!((good.weight() / loved.weight() - 0.35).abs() < 1e-9);
// A mixed neighbourhood: the far example pulls the mean down by 0.35× as
// much weight when it is merely "good" as when it is "loved".
let mut ranking = ranking();
ranking.knn_floor = 0;
ranking.knn_full = 1;
let near = example(1, "loved", 1.0, &[1.0, 0.0]);
let candidate = unit(&[1.0, 0.0]);
let both_loved = PreferenceState::build(
vec![near.clone(), example(2, "loved", 1.0, &[0.0, 1.0])],
&ranking,
);
let one_good =
PreferenceState::build(vec![near, example(2, "good", 0.35, &[0.0, 1.0])], &ranking);
let pull_loved = 1.0 - both_loved.knn(&candidate, &ranking).0.unwrap();
let pull_good = 1.0 - one_good.knn(&candidate, &ranking).0.unwrap();
assert!(pull_good < pull_loved);
// Weighted means: 0.5 vs 1/1.35 → pulls 0.5 vs 0.35/1.35.
assert!((pull_good / pull_loved - 0.35 / 1.35 / 0.5).abs() < 1e-9);
}
#[test]
fn negatives_subtract_with_the_negative_coefficient() {
let mut ranking = ranking();
ranking.knn_floor = 0;
ranking.knn_full = 1;
let state =
PreferenceState::build(vec![example(1, "not_for_me", -1.0, &[1.0, 0.0])], &ranking);
let (knn, _) = state.knn(&unit(&[1.0, 0.0]), &ranking);
assert!((knn.unwrap() + ranking.negative_coefficient).abs() < 1e-9);
}
#[test]
fn decay_halves_at_the_half_life() {
assert!((decay(60.0, 60.0) - 0.5).abs() < 1e-12);
assert!((decay(0.0, 60.0) - 1.0).abs() < 1e-12);
assert!((decay(120.0, 60.0) - 0.25).abs() < 1e-12);
}
#[test]
fn gate_is_zero_below_floor_one_at_full_and_linear_between() {
assert_eq!(gate(0, 8, 25), 0.0);
assert_eq!(gate(8, 8, 25), 0.0);
assert_eq!(gate(25, 8, 25), 1.0);
assert_eq!(gate(100, 8, 25), 1.0);
assert!((gate(16, 8, 24) - 0.5).abs() < 1e-9);
assert!((gate(9, 8, 25) - 1.0 / 17.0).abs() < 1e-9);
}
#[test]
fn knn_is_absent_when_the_gate_is_closed() {
let ranking = ranking(); // knn_floor 8
let state = PreferenceState::build(vec![example(1, "loved", 1.0, &[1.0, 0.0])], &ranking);
assert_eq!(state.knn_gate, 0.0);
assert_eq!(state.knn(&unit(&[1.0, 0.0]), &ranking), (None, Vec::new()));
}
// --- §9.3 feed affinity ---
#[test]
fn feed_credit_sums_to_one_across_direct_feeds() {
let mut rated = example(1, "loved", 1.0, &[1.0, 0.0]);
rated.feeds = vec![10, 20, 30];
let state = PreferenceState::build(vec![rated], &ranking());
let total: f64 = [10, 20, 30]
.iter()
.map(|feed| state.feed_credit(*feed).unwrap().0)
.sum();
assert!((total - 1.0).abs() < 1e-9);
assert!((state.feed_credit(10).unwrap().0 - 1.0 / 3.0).abs() < 1e-9);
}
#[test]
fn feed_affinity_is_the_mean_over_rated_feeds() {
let mut ranking = ranking();
ranking.feed_floor = 0;
ranking.feed_full = 1;
let mut loved = example(1, "loved", 1.0, &[1.0, 0.0]);
loved.feeds = vec![10];
let mut down = example(2, "not_for_me", -1.0, &[1.0, 0.0]);
down.feeds = vec![20];
let state = PreferenceState::build(vec![loved, down], &ranking);
// feed 10: (1+1)/(1+0+2) = 2/3; feed 20: (0+1)/(0+1+2) = 1/3; unrated 99 ignored.
let both = state.feed(&article(7, &[10, 20, 99])).unwrap();
assert!((both - 0.5).abs() < 1e-9, "{both}");
let best = state.feed(&article(8, &[10])).unwrap();
assert!((best - 2.0 / 3.0).abs() < 1e-9);
assert_eq!(state.feed(&article(9, &[99])), None);
}
#[test]
fn feed_is_absent_when_the_gate_is_closed() {
let ranking = ranking(); // feed_floor 15
let mut loved = example(1, "loved", 1.0, &[1.0, 0.0]);
loved.feeds = vec![10];
let state = PreferenceState::build(vec![loved], &ranking);
assert_eq!(state.feed_gate, 0.0);
assert_eq!(state.feed(&article(7, &[10])), None);
}
#[test]
fn direct_feeds_fall_back_to_the_best_entry_feed() {
let mut a = article(1, &[]);
a.feed_id = 42;
assert_eq!(direct_feeds(&a), vec![42]);
assert_eq!(direct_feeds(&article(2, &[5, 3, 5])), vec![3, 5]);
}
// --- §12.2 normalization, §12.4 blend ---
#[test]
fn constant_signal_normalizes_to_half_for_everyone() {
let mut values = [with_heuristic(Some(7.0)), with_heuristic(Some(7.0))];
let mut refs = values.iter_mut().collect::<Vec<_>>();
normalize(&mut refs);
assert!(values.iter().all(|v| v.norm["heuristic"] == 0.5));
let mut single = [with_heuristic(Some(3.0))];
let mut refs = single.iter_mut().collect::<Vec<_>>();
normalize(&mut refs);
assert_eq!(single[0].norm["heuristic"], 0.5);
}
#[test]
fn ties_get_equal_percentiles_without_an_id_ramp() {
let mut values = (0..400)
.map(|_| with_heuristic(Some(0.0)))
.collect::<Vec<_>>();
let mut refs = values.iter_mut().collect::<Vec<_>>();
normalize(&mut refs);
assert!(values.iter().all(|v| v.norm["heuristic"] == 0.5));
let mut mixed = [
with_heuristic(Some(1.0)),
with_heuristic(Some(2.0)),
with_heuristic(Some(2.0)),
with_heuristic(Some(3.0)),
];
let mut refs = mixed.iter_mut().collect::<Vec<_>>();
normalize(&mut refs);
assert_eq!(mixed[0].norm["heuristic"], 0.25);
assert_eq!(mixed[1].norm["heuristic"], 0.625);
assert_eq!(mixed[2].norm["heuristic"], 0.625);
assert_eq!(mixed[3].norm["heuristic"], 1.0);
}
#[test]
fn absent_values_do_not_shift_present_values() {
let mut values = [
with_heuristic(Some(1.0)),
with_heuristic(Some(2.0)),
with_heuristic(None),
];
let mut refs = values.iter_mut().collect::<Vec<_>>();
normalize(&mut refs);
assert!(!values[2].norm.contains_key("heuristic"));
// n_present = 2: the absent third value does not widen the scale.
assert_eq!(values[0].norm["heuristic"], 0.5);
assert_eq!(values[1].norm["heuristic"], 1.0);
}
#[test]
fn effective_weights_sum_to_one_and_missing_signals_are_skipped() {
let mut signals = Signals {
interest: Some(1.0),
heuristic: Some(2.0),
norm: BTreeMap::from([("interest".into(), 0.8), ("heuristic".into(), 0.4)]),
..Signals::default()
};
let blend = preliminary_blend(&mut signals, &PreliminaryWeights::default()).unwrap();
assert!((signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
assert!(!signals.weights.contains_key("knn"));
assert!(!signals.weights.contains_key("social"));
// 0.35/0.55 × 0.8 + 0.20/0.55 × 0.4 = 0.6545…
assert!((blend - 65.4545).abs() < 0.01, "{blend}");
let mut only_heuristic = Signals {
heuristic: Some(2.0),
norm: BTreeMap::from([("heuristic".into(), 0.4)]),
..Signals::default()
};
let blend = preliminary_blend(&mut only_heuristic, &PreliminaryWeights::default());
assert!((blend.unwrap() - 40.0).abs() < 1e-9);
assert_eq!(only_heuristic.weights["heuristic"], 1.0);
}
#[test]
fn learned_weights_are_multiplied_by_their_gate() {
let mut signals = Signals {
knn: Some(0.5),
heuristic: Some(2.0),
knn_gate: 0.5,
norm: BTreeMap::from([("knn".into(), 1.0), ("heuristic".into(), 0.0)]),
..Signals::default()
};
preliminary_blend(&mut signals, &PreliminaryWeights::default());
// knn 0.25 × 0.5 = 0.125 against heuristic 0.20.
assert!((signals.weights["knn"] - 0.125 / 0.325).abs() < 1e-9);
}
#[test]
fn compute_scores_every_article_and_leaves_ungated_signals_absent() {
let articles = vec![article(1, &[10]), article(2, &[20]), article(3, &[30])];
let mut embeddings = HashMap::new();
embeddings.insert(1, unit(&[1.0, 0.0]));
embeddings.insert(2, unit(&[0.0, 1.0]));
let mut interests = HashMap::new();
interests.insert("Axis".to_string(), unit(&[1.0, 0.0]));
let state = PreferenceState::build(vec![example(9, "loved", 1.0, &[1.0, 0.0])], &ranking());
let signals = compute(&articles, &embeddings, &interests, &state, &ranking());
assert_eq!(signals.len(), 3);
assert!(signals[&1].interest.is_some());
assert!(signals[&3].interest.is_none(), "no embedding → absent");
assert!(
signals
.values()
.all(|s| s.knn.is_none() && s.feed.is_none())
);
assert!(
signals
.values()
.all(|s| s.heuristic.is_some() && s.preliminary.is_some())
);
}
}
File diff suppressed because it is too large. Load diff
+1018
View File
File diff suppressed because it is too large. Load diff
+176 -63
View File
@@ -5,6 +5,7 @@
//! (implementation notes §2). Pipeline writes are idempotent upserts so that
//! `generate --date X` can be re-run safely; feedback events are append-only.
use std::collections::BTreeMap;
use std::path::Path;
use std::str::FromStr;
use std::time::Duration;
@@ -15,8 +16,8 @@ use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePoolOptions, S
use sqlx::{Row, SqlitePool};
use crate::types::{
Article, ArticleId, Entry, EntryId, Facets, LlmScore, Pick, RatedArticle, RatingEvent,
SocialRef, SocialSource, SourceRef,
Article, ArticleId, Entry, EntryId, Facets, Pick, RatedArticle, RatingEvent, SocialRef,
SocialSource, SourceRef,
};
/// Embedded migrations from `./migrations` (implementation notes §1).
@@ -24,9 +25,9 @@ pub static MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations");
/// `kv` key holding the ingest watermark (§3.1).
pub const KV_WATERMARK: &str = "ingest_watermark";
/// `kv` key holding the current taste profile document (§3.6).
/// `kv` key holding the current system-prompt profile document (§8.4).
pub const KV_TASTE_PROFILE: &str = "taste_profile";
/// `kv` key holding the taste profile version/build time (§3.6).
/// `kv` key holding the profile version/build time (§8.2).
pub const KV_PROFILE_VERSION: &str = "profile_version";
#[derive(Debug, thiserror::Error)]
@@ -43,6 +44,12 @@ pub enum DbError {
},
#[error("malformed value in column `{column}`: {value}")]
Decode { column: &'static str, value: String },
#[error("malformed JSON in column `{column}`: {source}")]
Json {
column: &'static str,
#[source]
source: serde_json::Error,
},
}
type Result<T> = std::result::Result<T, DbError>;
@@ -100,7 +107,7 @@ impl Db {
}
// -----------------------------------------------------------------
// kv (§3.1 watermark, §3.6 taste profile)
// kv (§3.1 watermark, §8 profile)
// -----------------------------------------------------------------
pub async fn kv_get(&self, key: &str) -> Result<Option<String>> {
@@ -321,21 +328,43 @@ impl Db {
Ok(rows.iter().map(|r| r.get::<i64, _>("article_id")).collect())
}
/// Articles the LLM scored below `threshold` within the last `days` (§3.5).
pub async fn recently_low_scored_ids(
&self,
threshold: f64,
since: Date,
) -> Result<Vec<ArticleId>> {
let rows = sqlx::query(
"SELECT DISTINCT article_id FROM scores
WHERE llm_score IS NOT NULL AND llm_score < ? AND run_date >= ?",
)
.bind(threshold)
.bind(since.to_string())
/// Article ids published before this issue date; same-date regeneration is allowed (§8.1).
pub async fn previously_published_ids_before(&self, date: Date) -> Result<Vec<ArticleId>> {
let rows =
sqlx::query("SELECT DISTINCT article_id FROM issue_articles WHERE issue_date < ?")
.bind(date.to_string())
.fetch_all(&self.pool)
.await?;
Ok(rows.iter().map(|r| r.get::<i64, _>("article_id")).collect())
Ok(rows
.iter()
.map(|row| row.get::<i64, _>("article_id"))
.collect())
}
/// Published article ids first seen at or after `since` (`features backfill`).
pub async fn published_article_ids_since(&self, since: Timestamp) -> Result<Vec<ArticleId>> {
let rows = sqlx::query(
"SELECT DISTINCT ia.article_id FROM issue_articles ia
JOIN articles a ON a.id = ia.article_id
WHERE a.first_seen >= ?
ORDER BY ia.article_id",
)
.bind(fmt_ts(since))
.fetch_all(&self.pool)
.await?;
Ok(rows
.iter()
.map(|row| row.get::<i64, _>("article_id"))
.collect())
}
/// Every article id first seen at or after `since` (`features backfill --all`).
pub async fn article_ids_since(&self, since: Timestamp) -> Result<Vec<ArticleId>> {
let rows = sqlx::query("SELECT id FROM articles WHERE first_seen >= ? ORDER BY id")
.bind(fmt_ts(since))
.fetch_all(&self.pool)
.await?;
Ok(rows.iter().map(|row| row.get::<i64, _>("id")).collect())
}
// -----------------------------------------------------------------
@@ -376,37 +405,6 @@ impl Db {
rows.iter().map(social_from_row).collect()
}
// -----------------------------------------------------------------
// scores (§3.5, §3.6)
// -----------------------------------------------------------------
pub async fn upsert_score(
&self,
article_id: ArticleId,
run_date: Date,
prefilter_score: Option<f64>,
llm: Option<&LlmScore>,
) -> Result<()> {
sqlx::query(
"INSERT INTO scores (article_id, run_date, prefilter_score, llm_score, llm_category, rationale)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(article_id, run_date) DO UPDATE SET
prefilter_score = COALESCE(excluded.prefilter_score, scores.prefilter_score),
llm_score = COALESCE(excluded.llm_score, scores.llm_score),
llm_category = COALESCE(excluded.llm_category, scores.llm_category),
rationale = COALESCE(excluded.rationale, scores.rationale)",
)
.bind(article_id)
.bind(run_date.to_string())
.bind(prefilter_score)
.bind(llm.map(|l| l.score))
.bind(llm.map(|l| l.category.as_str()))
.bind(llm.map(|l| l.rationale.as_str()))
.execute(&self.pool)
.await?;
Ok(())
}
// -----------------------------------------------------------------
// issues + lineup (§3.10)
// -----------------------------------------------------------------
@@ -480,8 +478,8 @@ impl Db {
.await?;
for pick in picks {
sqlx::query(
"INSERT INTO issue_articles (issue_date, article_id, section, position, is_lead, summary)
VALUES (?, ?, ?, ?, ?, ?)",
"INSERT INTO issue_articles (issue_date, article_id, section, position, is_lead, summary, why)
VALUES (?, ?, ?, ?, ?, ?, ?)",
)
.bind(date.to_string())
.bind(pick.article.id)
@@ -489,6 +487,7 @@ impl Db {
.bind(pick.position)
.bind(pick.is_lead)
.bind(pick.summary.as_deref())
.bind(pick.why.as_deref())
.execute(&mut *tx)
.await?;
}
@@ -636,7 +635,7 @@ impl Db {
}
// -----------------------------------------------------------------
// runs (§3.6 cost guardrail, §3.13)
// runs (§3.13, plan §7.6)
// -----------------------------------------------------------------
/// Insert a `running` row at the top of `generate`; returns `runs.id`.
@@ -656,7 +655,7 @@ impl Db {
sqlx::query(
"UPDATE runs SET finished_at = ?, entries_fetched = ?, candidates = ?, selected = ?,
input_tokens = ?, cached_tokens = ?, output_tokens = ?, cost_usd = ?,
status = ?, error = ?
status = ?, error = ?, provider_costs_json = ?, config_json = ?
WHERE id = ?",
)
.bind(report.finished_at.map(fmt_ts))
@@ -669,20 +668,55 @@ impl Db {
.bind(report.cost_usd)
.bind(report.status.as_str())
.bind(report.error.as_deref())
.bind(
serde_json::to_string(&report.provider_costs).map_err(|source| DbError::Json {
column: "runs.provider_costs_json",
source,
})?,
)
.bind(
serde_json::to_string(&report.config_json).map_err(|source| DbError::Json {
column: "runs.config_json",
source,
})?,
)
.bind(id)
.execute(&self.pool)
.await?;
Ok(())
}
/// Total spend recorded for a date, for the `max_daily_usd` guardrail (§3.6).
pub async fn spend_for_date(&self, date: Date) -> Result<f64> {
let row =
sqlx::query("SELECT COALESCE(SUM(cost_usd), 0.0) AS total FROM runs WHERE date = ?")
.bind(date.to_string())
.fetch_one(&self.pool)
/// Earlier provider spend on the UTC date containing this run's start (§5).
pub async fn provider_spend_for_utc_day(
&self,
started_at: Timestamp,
) -> Result<BTreeMap<String, f64>> {
let utc_date = started_at
.to_zoned(jiff::tz::TimeZone::UTC)
.date()
.to_string();
let rows = sqlx::query(
"SELECT provider_costs_json FROM runs
WHERE substr(started_at, 1, 10) = ? AND started_at < ?
AND provider_costs_json IS NOT NULL",
)
.bind(utc_date)
.bind(fmt_ts(started_at))
.fetch_all(&self.pool)
.await?;
Ok(row.get::<f64, _>("total"))
let mut totals = BTreeMap::new();
for row in rows {
let raw = row.get::<String, _>("provider_costs_json");
let providers: BTreeMap<String, crate::report::ProviderUsage> =
serde_json::from_str(&raw).map_err(|source| DbError::Json {
column: "runs.provider_costs_json",
source,
})?;
for (provider, usage) in providers {
*totals.entry(provider).or_insert(0.0) += usage.cost_usd;
}
}
Ok(totals)
}
}
@@ -988,7 +1022,7 @@ mod tests {
report.counts.entries_fetched = 412;
report.counts.candidates = 120;
report.counts.selected = 20;
report.finish(ts("2026-08-15T05:36:00Z"), 0.14, 0.0028, 0.28);
report.finish(ts("2026-08-15T05:36:00Z"));
db.finish_run(run_id, &report).await.unwrap();
db.upsert_issue(
@@ -1006,7 +1040,82 @@ mod tests {
let next: Date = "2026-08-16".parse().unwrap();
assert_eq!(db.next_issue_number(next).await.unwrap(), 2);
assert_eq!(db.recent_reports(5).await.unwrap().len(), 1);
assert_eq!(db.spend_for_date(date).await.unwrap(), 0.0);
// No provider spend was recorded, so the budget-day preload is empty.
let spend = db
.provider_spend_for_utc_day(ts("2026-08-15T23:00:00Z"))
.await
.unwrap();
assert!(spend.values().all(|usd| *usd == 0.0));
}
async fn record_run(db: &Db, date: Date, started: &str, deepseek: f64, anthropic: f64) {
use crate::report::{ProviderUsage, RunReport};
let started_at = ts(started);
let run_id = db.start_run(date, started_at).await.unwrap();
let mut report = RunReport::new(date, started_at);
report.provider_costs.insert(
"deepseek".into(),
ProviderUsage {
usage: crate::types::TokenUsage {
input_tokens: 10,
..Default::default()
},
cost_usd: deepseek,
},
);
report.provider_costs.insert(
"anthropic".into(),
ProviderUsage {
usage: crate::types::TokenUsage::default(),
cost_usd: anthropic,
},
);
report.config_json = serde_json::json!({"models": {"editor": "claude-opus-5"}});
report.finish(started_at);
db.finish_run(run_id, &report).await.unwrap();
}
#[tokio::test]
async fn provider_spend_is_summed_by_the_utc_day_of_started_at() {
let (_dir, db) = temp_db().await;
let date: Date = "2026-08-15".parse().unwrap();
record_run(&db, date, "2026-08-15T03:00:00Z", 0.10, 0.50).await;
record_run(&db, date, "2026-08-15T23:30:00Z", 0.05, 0.0).await;
// Nominal issue date 08-15 in New York, but already 08-16 in UTC: a
// different budget day (§5).
record_run(&db, date, "2026-08-16T01:00:00Z", 1.0, 1.0).await;
let spend = db
.provider_spend_for_utc_day(ts("2026-08-15T23:45:00Z"))
.await
.unwrap();
assert!((spend["deepseek"] - 0.15).abs() < 1e-9);
assert!((spend["anthropic"] - 0.5).abs() < 1e-9);
// Only runs that started earlier than this one count.
let spend = db
.provider_spend_for_utc_day(ts("2026-08-15T12:00:00Z"))
.await
.unwrap();
assert!((spend["deepseek"] - 0.10).abs() < 1e-9);
let spend = db
.provider_spend_for_utc_day(ts("2026-08-17T12:00:00Z"))
.await
.unwrap();
assert!(spend.is_empty());
// `provider_costs_json` and `config_json` were written and round-trip.
let row =
sqlx::query("SELECT provider_costs_json, config_json FROM runs ORDER BY id LIMIT 1")
.fetch_one(&db.pool)
.await
.unwrap();
let costs: BTreeMap<String, crate::report::ProviderUsage> =
serde_json::from_str(&row.get::<String, _>("provider_costs_json")).unwrap();
assert_eq!(costs["deepseek"].usage.input_tokens, 10);
assert!((costs["anthropic"].cost_usd - 0.5).abs() < 1e-9);
let config: serde_json::Value =
serde_json::from_str(&row.get::<String, _>("config_json")).unwrap();
assert_eq!(config["models"]["editor"], "claude-opus-5");
}
#[tokio::test]
@@ -1155,6 +1264,10 @@ mod tests {
.execute(&pool)
.await
.unwrap();
sqlx::raw_sql(include_str!("../migrations/0003_drop_scores.sql"))
.execute(&pool)
.await
.unwrap();
let rows = sqlx::query(
"SELECT article_id, issue_date, kind, source, label, value, event_at
@@ -1182,7 +1295,7 @@ mod tests {
.unwrap();
assert!(!tables.iter().any(|table| table == "ratings"));
assert!(!tables.iter().any(|table| table == "feed_priors"));
assert!(tables.iter().any(|table| table == "scores"));
assert!(!tables.iter().any(|table| table == "scores"));
for expected in [
"rating_events",
"article_embeddings",
+9 -11
View File
@@ -1,7 +1,8 @@
//! Chapter ordering and `epub-builder` assembly (spec §3.10).
//!
//! Structure: cover → From the Editor → In This Issue → sections (title page,
//! article chapters, discussion chapters) → World Briefing → colophon. The
//! Structure: cover → The Brief → In This Issue → sections (title page,
//! article chapters, discussion chapters) → World Briefing → Behind the paper
//! → colophon. The
//! chapters themselves are rendered by [`super::chapters`] and the cover by
//! [`super::cover`]; this module decides the order and zips the result.
@@ -14,8 +15,8 @@ use crate::types::{Edition, ImageAsset, Issue};
use super::EpubError;
use super::chapters::{
render_colophon, render_front_page, render_in_this_issue, render_section_page,
render_world_briefing, section_names,
render_behind_the_paper, render_colophon, render_front_page, render_in_this_issue,
render_section_page, render_world_briefing, section_names,
};
use super::cover::{CoverAsset, render_cover_page};
use super::x4;
@@ -66,12 +67,7 @@ pub fn render_all(
];
for name in section_names(issue) {
let intro = issue
.editorial
.section_intros
.get(&name)
.map(|s| s.as_str());
chapters.push(render_section_page(&name, intro)?);
chapters.push(render_section_page(&name)?);
for pick in issue.lineup.section_picks(&name) {
chapters.push(render_article(
issue,
@@ -90,6 +86,7 @@ pub fn render_all(
if let Some(world) = render_world_briefing(issue)? {
chapters.push(world);
}
chapters.push(render_behind_the_paper(issue)?);
chapters.push(render_colophon(issue)?);
if edition == Edition::X4 {
@@ -277,11 +274,12 @@ mod tests {
"sec-Niche Corner",
"art-1002",
"world",
"behind",
"colophon",
]
);
let levels: Vec<u8> = chapters.iter().map(|c| c.toc_level).collect();
assert_eq!(levels, vec![1, 1, 1, 1, 2, 3, 1, 2, 1, 1]);
assert_eq!(levels, vec![1, 1, 1, 1, 2, 3, 1, 2, 1, 1, 1]);
for chapter in &chapters {
assert_xml_ok(&chapter.xhtml);
}
+235 -17
View File
@@ -10,7 +10,10 @@ use askama::Template;
use crate::comments;
use crate::html::{text_escape, to_xhtml};
use crate::images;
use crate::types::{Edition, ImageAsset, Issue, Pick, SocialRef, Vote, WORLD_BRIEFING_SECTION};
use crate::types::{
BehindThePaper, Edition, ImageAsset, Issue, NearMiss, Pick, SocialRef, Vote,
WORLD_BRIEFING_SECTION,
};
use crate::world;
use super::EpubError;
@@ -39,6 +42,7 @@ struct IndexEntry {
source: String,
reading_minutes: i64,
summary: String,
why: Option<String>,
}
struct IndexSection {
@@ -59,7 +63,6 @@ struct InThisIssue {
struct SectionPage {
title: String,
name: String,
intro: Option<String>,
}
struct RatingLinks {
@@ -76,6 +79,7 @@ struct ArticleChapter {
byline: Option<String>,
meta_line: String,
social_line: Option<String>,
why: Option<String>,
summary: Option<String>,
excerpt_only: bool,
body_html: String,
@@ -101,6 +105,17 @@ struct WorldBriefingChapter {
body_html: String,
}
#[derive(Template)]
#[template(path = "behind.xhtml", escape = "html")]
struct BehindChapter {
title: String,
summary_line: String,
admitted_line: String,
learned_line: String,
near_misses: Vec<String>,
models_line: String,
}
#[derive(Template)]
#[template(path = "colophon.xhtml", escape = "html")]
struct ColophonChapter {
@@ -108,7 +123,10 @@ struct ColophonChapter {
issue_number: i64,
display_date: String,
generated_at: String,
model: String,
bulk_model: String,
editor_model: String,
summaries_model: String,
provider_costs: Vec<ProviderCostLine>,
entries_fetched: i64,
feeds_seen: i64,
candidates: i64,
@@ -120,6 +138,11 @@ struct ColophonChapter {
generator_version: String,
}
struct ProviderCostLine {
provider: String,
cost: String,
}
// ---------------------------------------------------------------------------
// Chapters (§3.10)
// ---------------------------------------------------------------------------
@@ -214,7 +237,7 @@ fn published_display(pick: &Pick) -> Option<String> {
.map(|ts| ts.to_zoned(jiff::tz::TimeZone::UTC).date().to_string())
}
/// "From the Editor" front page plus the issue stats line (§3.10).
/// The Brief (§14.2) under the masthead, plus the issue stats line (§3.10).
pub fn render_front_page(issue: &Issue) -> Result<Chapter, EpubError> {
let body = issue.editorial.front_page_html.trim();
let body_html = if body.is_empty() {
@@ -226,7 +249,7 @@ pub fn render_front_page(issue: &Issue) -> Result<Chapter, EpubError> {
to_xhtml(&ammonia::clean(body))
};
let tpl = FrontPage {
title: "From the Editor".into(),
title: "The Brief".into(),
display_date: issue.meta.display_date.clone(),
issue_number: issue.meta.issue_number,
stats_line: issue.meta.stats_line(),
@@ -235,7 +258,7 @@ pub fn render_front_page(issue: &Issue) -> Result<Chapter, EpubError> {
Ok(Chapter {
id: "front".into(),
href: "front.xhtml".into(),
title: "From the Editor".into(),
title: "The Brief".into(),
xhtml: tpl.render()?,
toc_level: 1,
})
@@ -274,6 +297,7 @@ pub fn render_in_this_issue(issue: &Issue) -> Result<Chapter, EpubError> {
source: pick.article.feed_title.clone(),
reading_minutes: pick.article.reading_minutes(),
summary: summary_for(issue, pick).unwrap_or_default().to_string(),
why: pick.why.clone(),
})
.collect();
sections.push(IndexSection { name, entries });
@@ -287,6 +311,7 @@ pub fn render_in_this_issue(issue: &Issue) -> Result<Chapter, EpubError> {
source: "Wikipedia Current Events".into(),
reading_minutes: 3,
summary: "The day's events, as recorded by the Current Events portal.".into(),
why: None,
}],
});
}
@@ -304,14 +329,11 @@ pub fn render_in_this_issue(issue: &Issue) -> Result<Chapter, EpubError> {
})
}
/// A section title page: name + LLM intro (§3.10).
pub fn render_section_page(name: &str, intro: Option<&str>) -> Result<Chapter, EpubError> {
/// A section title page: the name only (§14.2 removed the LLM intros).
pub fn render_section_page(name: &str) -> Result<Chapter, EpubError> {
let tpl = SectionPage {
title: name.to_string(),
name: name.to_string(),
intro: intro
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty()),
};
Ok(Chapter {
id: format!("sec-{name}"),
@@ -364,6 +386,7 @@ pub fn render_article(
byline: article.author.as_ref().map(|a| format!("By {a}")),
meta_line: meta_parts.join(" \u{00b7} "),
social_line: social_line(&article.social),
why: pick.why.clone(),
summary: summary_for(issue, pick).map(str::to_string),
excerpt_only: article.excerpt_only,
body_html: prepare_body(&article.content_html, images_),
@@ -426,19 +449,151 @@ pub fn render_world_briefing(issue: &Issue) -> Result<Option<Chapter>, EpubError
}))
}
/// "1,465" — thousands separators for the counts in Behind the paper.
fn thousands(n: i64) -> String {
let digits = n.abs().to_string();
let mut out = String::with_capacity(digits.len() + digits.len() / 3);
for (i, c) in digits.chars().enumerate() {
if i > 0 && (digits.len() - i).is_multiple_of(3) {
out.push(',');
}
out.push(c);
}
if n < 0 { format!("-{out}") } else { out }
}
/// `Considered 412 articles from 1,465 feeds · 398 eligible · …` (§15.1).
pub fn behind_summary_line(b: &BehindThePaper) -> String {
format!(
"Considered {} articles from {} feeds \u{00b7} {} eligible \u{00b7} {} triaged \u{00b7} {} read closely \u{00b7} {} shortlisted \u{00b7} {} selected.",
thousands(b.considered),
thousands(b.feeds_seen),
thousands(b.eligible),
thousands(b.triaged),
thousands(b.read_closely),
thousands(b.shortlisted),
thousands(b.selected),
)
}
/// `Admitted via: triage 60 · interests 20 · your ratings 12 · exploration 5 · blend 23.`
pub fn behind_admitted_line(b: &BehindThePaper) -> String {
let count = |name: &str| b.admitted_by.get(name).copied().unwrap_or(0);
let mut parts = vec![
format!("triage {}", count("triage")),
format!("interests {}", count("interest")),
format!("your ratings {}", count("knn")),
format!("exploration {}", count("exploration")),
format!("blend {}", count("blend")),
];
if count("auto_include") > 0 {
parts.push(format!("always-include {}", count("auto_include")));
}
format!("Admitted via: {}.", parts.join(" \u{00b7} "))
}
/// `Learned signals: 14 rated articles with embeddings (neighbour signal at 35%); feed affinity off.`
pub fn behind_learned_line(b: &BehindThePaper) -> String {
let percent = |gate: f64| (gate.clamp(0.0, 1.0) * 100.0).round() as i64;
let neighbour = if b.knn_gate > 0.0 {
format!("neighbour signal at {}%", percent(b.knn_gate))
} else {
"neighbour signal off".to_string()
};
let feed = if b.feed_gate > 0.0 {
format!("feed affinity at {}%", percent(b.feed_gate))
} else {
"feed affinity off".to_string()
};
format!(
"Learned signals: {} rated articles with embeddings ({neighbour}); {feed}.",
thousands(b.rated_with_embeddings)
)
}
/// `<title> — <feed> · quality 8.0 · fit 6.5 · shortlisted, not selected`.
pub fn behind_near_miss_line(miss: &NearMiss) -> String {
let mut parts = vec![format!("{} \u{2014} {}", miss.title, miss.feed_title)];
if let Some(quality) = miss.quality {
parts.push(format!("quality {quality:.1}"));
}
if let Some(fit) = miss.fit {
parts.push(format!("fit {fit:.1}"));
}
parts.push(match &miss.reason {
Some(reason) => format!("{}, {}", miss.stage, reason.replace('_', " ")),
None => miss.stage.clone(),
});
parts.join(" \u{00b7} ")
}
/// `Models: triage and assessment … · editor and summaries … · embeddings ….
/// Cost $0.81. Generation 23 min.`
pub fn behind_models_line(b: &BehindThePaper) -> String {
let editorial = if b.models.summaries == b.models.editor {
format!("editor and summaries {}", b.models.editor)
} else {
format!(
"editor {} \u{00b7} summaries {}",
b.models.editor, b.models.summaries
)
};
let generation = if b.generation_secs < 60 {
format!("{} s", b.generation_secs.max(0))
} else {
format!("{} min", (b.generation_secs as f64 / 60.0).round() as i64)
};
format!(
"Models: triage and assessment {} \u{00b7} {editorial} \u{00b7} embeddings {}. Cost ${:.2}. Generation {generation}.",
b.models.bulk, b.embedding_model, b.cost_usd
)
}
/// "Behind the paper": the run's counts, admission mix, learned-signal state,
/// near misses and models (§15.1). Same text in both editions; no links.
pub fn render_behind_the_paper(issue: &Issue) -> Result<Chapter, EpubError> {
let behind = &issue.behind;
let tpl = BehindChapter {
title: "Behind the paper".into(),
summary_line: behind_summary_line(behind),
admitted_line: behind_admitted_line(behind),
learned_line: behind_learned_line(behind),
near_misses: behind
.near_misses
.iter()
.map(behind_near_miss_line)
.collect(),
models_line: behind_models_line(behind),
};
Ok(Chapter {
id: "behind".into(),
href: "behind.xhtml".into(),
title: "Behind the paper".into(),
xhtml: tpl.render()?,
toc_level: 1,
})
}
/// Colophon: generation timestamp, models used, token cost, feed counts (§3.10).
pub fn render_colophon(issue: &Issue) -> Result<Chapter, EpubError> {
let colophon = &issue.colophon;
let provider_costs = colophon
.provider_costs
.iter()
.map(|(provider, cost)| ProviderCostLine {
provider: provider.clone(),
cost: format!("${cost:.4}"),
})
.collect();
let tpl = ColophonChapter {
title: "Colophon".into(),
issue_number: issue.meta.issue_number,
display_date: issue.meta.display_date.clone(),
generated_at: issue.meta.generated_at.to_string(),
model: if colophon.model.is_empty() {
"none (heuristic selection)".into()
} else {
colophon.model.clone()
},
bulk_model: colophon.models.bulk.clone(),
editor_model: colophon.models.editor.clone(),
summaries_model: colophon.models.summaries.clone(),
provider_costs,
entries_fetched: colophon.entries_fetched,
feeds_seen: colophon.feeds_seen,
candidates: colophon.candidates,
@@ -571,7 +726,7 @@ mod tests {
let issue = issue();
let chapter = render_front_page(&issue).unwrap();
assert_eq!(chapter.href, "front.xhtml");
assert!(chapter.xhtml.contains("From the Editor"));
assert!(chapter.xhtml.contains("The Brief"));
assert!(chapter.xhtml.contains("2 articles"));
assert!(chapter.xhtml.contains("both worth your coffee"));
assert!(chapter.xhtml.contains("No. 42"));
@@ -660,6 +815,69 @@ mod tests {
assert!(render_discussion(&issue.lineup.picks[1]).unwrap().is_none());
}
#[test]
fn behind_the_paper_lines_follow_the_plan_shape() {
let issue = issue();
let behind = &issue.behind;
assert_eq!(
behind_summary_line(behind),
"Considered 412 articles from 1,465 feeds \u{00b7} 398 eligible \u{00b7} 398 triaged \u{00b7} 120 read closely \u{00b7} 60 shortlisted \u{00b7} 2 selected."
);
assert_eq!(
behind_admitted_line(behind),
"Admitted via: triage 60 \u{00b7} interests 20 \u{00b7} your ratings 12 \u{00b7} exploration 5 \u{00b7} blend 23."
);
assert_eq!(
behind_learned_line(behind),
"Learned signals: 14 rated articles with embeddings (neighbour signal at 35%); feed affinity off."
);
assert_eq!(
behind_near_miss_line(&behind.near_misses[0]),
"The One That Got Away \u{2014} Example Feed \u{00b7} quality 8.0 \u{00b7} fit 6.5 \u{00b7} shortlisted, not selected"
);
assert_eq!(
behind_near_miss_line(&behind.near_misses[1]),
"Never Read Closely \u{2014} Other Feed \u{00b7} triaged, not admitted"
);
assert_eq!(
behind_models_line(behind),
"Models: triage and assessment deepseek-v4-flash \u{00b7} editor and summaries claude-opus-5 \u{00b7} embeddings voyage-4-lite. Cost $0.81. Generation 23 min."
);
let chapter = render_behind_the_paper(&issue).unwrap();
assert_eq!(chapter.id, "behind");
assert_eq!(chapter.href, "behind.xhtml");
assert!(chapter.xhtml.contains("Behind the paper"));
assert!(chapter.xhtml.contains("1,465 feeds"));
assert!(chapter.xhtml.contains("The One That Got Away"));
assert!(!chapter.xhtml.contains("<a "), "no links in the chapter");
assert_xml_ok(&chapter.xhtml);
// Auto-includes are named only when there were any; a short run is in
// seconds; a differing summaries model is listed separately.
let mut short = behind.clone();
short.admitted_by.insert("auto_include".into(), 2);
short.generation_secs = 48;
short.feed_gate = 0.6;
short.models.summaries = "deepseek-v4-flash".into();
short.near_misses.clear();
assert!(behind_admitted_line(&short).ends_with("blend 23 \u{00b7} always-include 2."));
assert!(behind_learned_line(&short).ends_with("feed affinity at 60%."));
assert!(
behind_models_line(&short)
.contains("editor claude-opus-5 \u{00b7} summaries deepseek-v4-flash")
);
assert!(behind_models_line(&short).ends_with("Generation 48 s."));
let mut issue = issue;
issue.behind = short;
let chapter = render_behind_the_paper(&issue).unwrap();
assert!(chapter.xhtml.contains("None recorded"));
assert_xml_ok(&chapter.xhtml);
assert_eq!(thousands(0), "0");
assert_eq!(thousands(999), "999");
assert_eq!(thousands(1_000), "1,000");
assert_eq!(thousands(1_234_567), "1,234,567");
}
#[test]
fn world_and_colophon_chapters_render() {
let issue = issue();
+58 -4
View File
@@ -86,6 +86,7 @@ pub fn issue() -> Issue {
section: "Top Stories".into(),
position: 0,
is_lead: true,
why: Some("The systems story with enough operational detail to matter".into()),
summary: Some("What it argues, and why it is worth the time.".into()),
llm: None,
discussion: Some(discussion(1, 1001)),
@@ -95,12 +96,11 @@ pub fn issue() -> Issue {
section: "Niche Corner".into(),
position: 0,
is_lead: false,
why: Some("A small-scene delight outside the usual technical orbit".into()),
summary: None,
llm: None,
discussion: None,
};
let mut section_intros = BTreeMap::new();
section_intros.insert("Top Stories".to_string(), "The day in brief.".to_string());
let mut summaries = BTreeMap::new();
summaries.insert(2, "A short abstract for the second piece.".to_string());
@@ -122,7 +122,6 @@ pub fn issue() -> Issue {
},
editorial: Editorial {
front_page_html: "<p>Two stories today, both worth your coffee.</p>".into(),
section_intros,
summaries,
},
world_briefing: Some(WorldBriefing {
@@ -141,13 +140,68 @@ pub fn issue() -> Issue {
}],
}),
colophon: Colophon {
model: "deepseek-v4-flash".into(),
provider_costs: BTreeMap::from([
("deepseek".into(), 0.0231),
("anthropic".into(), 0.05),
]),
models: Models {
bulk: "deepseek-v4-flash".into(),
editor: "claude-opus-5".into(),
summaries: "claude-opus-5".into(),
},
entries_fetched: 431,
feeds_seen: 92,
candidates: 120,
cost_usd: 0.0731,
generator_version: "daily-epub 0.1.0".into(),
},
behind: BehindThePaper {
considered: 412,
feeds_seen: 1465,
eligible: 398,
triaged: 398,
read_closely: 120,
shortlisted: 60,
selected: 2,
admitted_by: BTreeMap::from([
("triage".to_string(), 60),
("interest".to_string(), 20),
("knn".to_string(), 12),
("exploration".to_string(), 5),
("blend".to_string(), 23),
]),
rated_with_embeddings: 14,
knn_gate: 0.35,
feed_gate: 0.0,
near_misses: vec![
NearMiss {
article_id: 3,
title: "The One That Got Away".into(),
feed_title: "Example Feed".into(),
quality: Some(8.0),
fit: Some(6.5),
stage: "shortlisted".into(),
reason: Some("not_selected".into()),
},
NearMiss {
article_id: 4,
title: "Never Read Closely".into(),
feed_title: "Other Feed".into(),
quality: None,
fit: None,
stage: "triaged".into(),
reason: Some("not_admitted".into()),
},
],
models: Models {
bulk: "deepseek-v4-flash".into(),
editor: "claude-opus-5".into(),
summaries: "claude-opus-5".into(),
},
embedding_model: "voyage-4-lite".into(),
cost_usd: 0.81,
generation_secs: 23 * 60 + 12,
},
}
}
+1 -10
View File
@@ -16,16 +16,6 @@ use crate::config::Config;
use crate::images;
use crate::types::{Artifact, Edition, ImageAsset, Issue};
/// Chapter order inside an issue (§3.10).
pub const CHAPTER_ORDER: &[&str] = &[
"cover",
"from-the-editor",
"in-this-issue",
"sections",
"world-briefing",
"colophon",
];
#[derive(Debug, thiserror::Error)]
pub enum EpubError {
#[error("epub build failed: {0}")]
@@ -192,6 +182,7 @@ mod tests {
"OEBPS/disc-1001.xhtml",
"OEBPS/art-1002.xhtml",
"OEBPS/world.xhtml",
"OEBPS/behind.xhtml",
"OEBPS/colophon.xhtml",
] {
assert!(
+3 -2
View File
@@ -8,12 +8,13 @@ the crate root points askama here (`dirs = ["src/epub/templates"]`).
|---|---|---|
| `base.xhtml` | — | Shared XHTML skeleton (`{% block body_class %}`, `{% block content %}`) |
| `cover_page.xhtml` | `CoverPage` | Page that displays the rasterized cover image |
| `front_page.xhtml` | `FrontPage` | "From the Editor" + issue stats line |
| `front_page.xhtml` | `FrontPage` | "The Brief" + issue stats line |
| `in_this_issue.xhtml` | `InThisIssue` | Introduction chapter: per-section linked index |
| `section.xhtml` | `SectionPage` | Section title page + LLM intro |
| `section.xhtml` | `SectionPage` | Section title page (name only) |
| `chapter.xhtml` | `ArticleChapter` | Article: header, body, rating/read-online footer |
| `discussion.xhtml` | `DiscussionChapter` | Comment chapter (§3.7); body from `comments::render_xhtml` |
| `world_briefing.xhtml` | `WorldBriefingChapter` | Wikipedia Current Events (§3.8), body from `world::render_xhtml` |
| `behind.xhtml` | `BehindChapter` | "Behind the paper": run counts, admission mix, near misses, models (§15.1) |
| `colophon.xhtml` | `ColophonChapter` | Back matter: models, cost, counts |
| `cover.svg` | `CoverSvg` | Typographic cover, rasterized with resvg + tiny-skia |
| `style.css` | — | Standard-edition stylesheet, embedded as `stylesheet.css` |
+20
View File
@@ -0,0 +1,20 @@
{% extends "base.xhtml" %}
{% block body_class %}behind{% endblock %}
{% block content %}
<h1>Behind the paper</h1>
<p class="fact-line">{{ summary_line }}</p>
<p class="fact-line">{{ admitted_line }}</p>
<p class="fact-line">{{ learned_line }}</p>
<h2>Near misses</h2>
<p class="fact-line"><em>Highest utility not selected.</em></p>
{% if near_misses.is_empty() %}
<p class="fact-line">None recorded for this run.</p>
{% else %}
<ul class="near-misses">
{% for line in near_misses %}
<li>{{ line }}</li>
{% endfor %}
</ul>
{% endif %}
<p class="fact-line">{{ models_line }}</p>
{% endblock %}
+3
View File
@@ -7,6 +7,9 @@
<p class="byline">{{ line }}</p>
{% endif %}
<p class="meta">{{ meta_line }}</p>
{% if let Some(text) = why %}
<p class="why"><em>Why it&#39;s here: {{ text }}</em></p>
{% endif %}
{% if let Some(line) = social_line %}
<p class="social">{{ line }}</p>
{% endif %}
+7 -2
View File
@@ -9,12 +9,17 @@
</p>
<p class="fact-line"><strong>Issue:</strong> No. {{ issue_number }} &#183; {{ display_date }}</p>
<p class="fact-line"><strong>Generated:</strong> {{ generated_at }}</p>
<p class="fact-line"><strong>Curation model:</strong> {{ model }}</p>
<p class="fact-line"><strong>Bulk model:</strong> {{ bulk_model }}</p>
<p class="fact-line"><strong>Editor model:</strong> {{ editor_model }}</p>
<p class="fact-line"><strong>Summaries model:</strong> {{ summaries_model }}</p>
<p class="fact-line"><strong>Entries considered:</strong> {{ entries_fetched }} from {{ feeds_seen }} feeds</p>
<p class="fact-line"><strong>Candidates scored:</strong> {{ candidates }}</p>
<p class="fact-line"><strong>Articles selected:</strong> {{ article_count }} across {{ section_count }} sections</p>
<p class="fact-line"><strong>Words:</strong> {{ total_words }} &#183; {{ reading_line }}</p>
<p class="fact-line"><strong>Token cost:</strong> {{ cost_usd }}</p>
{% for line in provider_costs %}
<p class="fact-line"><strong>{{ line.provider }} cost:</strong> {{ line.cost }}</p>
{% endfor %}
<p class="fact-line"><strong>Total token cost:</strong> {{ cost_usd }}</p>
<p class="fact-line"><strong>Generator:</strong> {{ generator_version }}</p>
<p class="attribution">
Article text belongs to its authors and publications; excerpts and links are
+1 -1
View File
@@ -4,7 +4,7 @@
<h1 class="masthead">The Daily EPUB</h1>
<p class="dateline">{{ display_date }} &#183; No. {{ issue_number }}</p>
<hr class="rule"/>
<h2 class="kicker">From the Editor</h2>
<h2 class="kicker">The Brief</h2>
<div class="editorial">
{{ body_html|safe }}
</div>
+3
View File
@@ -12,6 +12,9 @@
<p class="index-meta">{{ entry.source }} &#183; {{ entry.reading_minutes }} min read</p>
{% if !entry.summary.is_empty() %}
<p class="index-summary">{{ entry.summary }}</p>
{% endif %}
{% if let Some(text) = entry.why %}
<p class="index-why"><em>Why it&#39;s here: {{ text }}</em></p>
{% endif %}
</li>
{% endfor %}
-3
View File
@@ -3,7 +3,4 @@
{% block content %}
<h1 class="section-title">{{ name }}</h1>
<hr class="rule"/>
{% if let Some(text) = intro %}
<p class="section-intro">{{ text }}</p>
{% endif %}
{% endblock %}
+2
View File
@@ -190,3 +190,5 @@ p.comment-line {
.fact-line {
margin: 0 0 0.35em 0;
}
.why, .index-why { font-size: 0.9em; font-style: italic; }
+2
View File
@@ -296,3 +296,5 @@ blockquote.comment blockquote.comment {
.fact-line {
margin: 0 0 0.35em 0;
}
.why, .index-why { font-size: 0.9em; font-style: italic; }
+1
View File
@@ -23,6 +23,7 @@ pub mod extract;
pub mod html;
pub mod http;
pub mod images;
pub mod lock;
pub mod miniflux;
pub mod pipeline;
pub mod publish;
+168
View File
@@ -0,0 +1,168 @@
//! One writer at a time (plan §5): an advisory `flock(LOCK_EX | LOCK_NB)` on
//! `<database_path>.lock`, taken in `main` for `generate`, `profile rebuild`,
//! `features backfill` and `backfill-social`. No table, no TTL: the kernel
//! releases the lock when the holder exits, however it exits.
use std::fs::{File, OpenOptions};
use std::io::{self, Write as _};
use std::os::fd::AsRawFd;
use std::path::{Path, PathBuf};
/// Held for as long as the value lives; dropping it (or dying) releases it.
#[derive(Debug)]
pub struct RunLock {
_file: File,
pub path: PathBuf,
}
#[derive(Debug, thiserror::Error)]
pub enum LockError {
/// Another process holds the lock; `holder` names its command when known.
#[error("{holder} is already running")]
Held { holder: String, path: PathBuf },
#[error("lock file {path}: {source}")]
Io {
path: PathBuf,
#[source]
source: io::Error,
},
}
/// `<database_path>.lock`, next to the database.
pub fn lock_path(database_path: &Path) -> PathBuf {
let mut name = database_path.file_name().unwrap_or_default().to_os_string();
name.push(".lock");
database_path.with_file_name(name)
}
/// Take the lock for `command`, writing the command's name into the file so a
/// second invocation can say who is holding it.
pub fn acquire(database_path: &Path, command: &str) -> Result<RunLock, LockError> {
let path = lock_path(database_path);
let io_err = |source: io::Error| LockError::Io {
path: path.clone(),
source,
};
if let Some(parent) = path.parent().filter(|p| !p.as_os_str().is_empty()) {
std::fs::create_dir_all(parent).map_err(io_err)?;
}
let mut file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(false)
.open(&path)
.map_err(io_err)?;
// SAFETY: `file` owns a valid open descriptor for the duration of the call.
let rc = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) };
if rc != 0 {
let source = io::Error::last_os_error();
if source.kind() == io::ErrorKind::WouldBlock {
let holder = std::fs::read_to_string(&path)
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.unwrap_or_else(|| "generate".to_string());
return Err(LockError::Held { holder, path });
}
return Err(io_err(source));
}
// Best effort: the name is a courtesy for the error message, never load-bearing.
let _ = file.set_len(0);
let _ = writeln!(file, "{command}");
Ok(RunLock { _file: file, path })
}
#[cfg(test)]
mod tests {
use super::*;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
const HELPER_ENV: &str = "DAILY_EPUB_LOCK_TEST_HOLDER";
#[test]
fn lock_path_sits_next_to_the_database() {
assert_eq!(
lock_path(Path::new("/var/lib/daily-epub/daily-epub.db")),
PathBuf::from("/var/lib/daily-epub/daily-epub.db.lock")
);
assert_eq!(lock_path(Path::new("x.db")), PathBuf::from("x.db.lock"));
}
#[test]
fn second_acquire_in_the_same_process_loses_and_names_the_holder() {
let dir = tempfile::tempdir().unwrap();
let db = dir.path().join("nested").join("daily-epub.db");
let first = acquire(&db, "generate").expect("first lock");
assert!(first.path.exists(), "the lock file is created on demand");
match acquire(&db, "profile rebuild") {
Err(LockError::Held { holder, .. }) => {
assert_eq!(holder, "generate");
assert_eq!(
LockError::Held {
holder,
path: first.path.clone()
}
.to_string(),
"generate is already running"
);
}
other => panic!("expected the lock to be held, got {other:?}"),
}
drop(first);
let again = acquire(&db, "backfill-social").expect("released on drop");
assert_eq!(
std::fs::read_to_string(&again.path).unwrap().trim(),
"backfill-social"
);
}
/// Not a test of its own: when `HELPER_ENV` names a database path, this
/// body takes the lock and holds it until it is killed. The parent below
/// spawns the test binary with that variable set.
#[test]
fn lock_holder_helper() {
let Ok(db) = std::env::var(HELPER_ENV) else {
return;
};
let _held = acquire(Path::new(&db), "helper").expect("helper takes the lock");
std::thread::sleep(Duration::from_secs(30));
}
#[test]
fn a_killed_holder_frees_the_lock_for_the_next_process() {
let dir = tempfile::tempdir().unwrap();
let db = dir.path().join("daily-epub.db");
let mut child = Command::new(std::env::current_exe().unwrap())
.args(["--exact", "lock::tests::lock_holder_helper", "--nocapture"])
.env(HELPER_ENV, &db)
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.expect("spawn the holder process");
// Wait (well under a second) until the child has written its name.
let path = lock_path(&db);
let deadline = Instant::now() + Duration::from_millis(900);
while std::fs::read_to_string(&path)
.map(|s| s.trim() != "helper")
.unwrap_or(true)
{
assert!(Instant::now() < deadline, "the helper never took the lock");
std::thread::sleep(Duration::from_millis(10));
}
match acquire(&db, "generate") {
Err(LockError::Held { holder, .. }) => assert_eq!(holder, "helper"),
other => panic!("another process holds the lock, got {other:?}"),
}
child.kill().expect("kill the holder");
child.wait().expect("reap the holder");
let lock = acquire(&db, "generate").expect("the kernel released the dead holder's lock");
assert_eq!(
std::fs::read_to_string(&lock.path).unwrap().trim(),
"generate"
);
}
}
+476 -8
View File
@@ -3,6 +3,7 @@
//! Everything of substance lives in the library (`src/lib.rs`); this binary only
//! parses flags, loads config, opens the database and dispatches.
use std::io::Write as _;
use std::path::PathBuf;
use anyhow::{Context, Result};
@@ -10,11 +11,13 @@ use clap::{Parser, Subcommand, ValueEnum};
use tracing_subscriber::EnvFilter;
use daily_epub::config::Config;
use daily_epub::curate::embedding::{self, BACKFILL_CONFIRM_TOKENS};
use daily_epub::curate::telemetry;
use daily_epub::db::Db;
use daily_epub::pipeline::{self, GenerateOptions, GenerateOutcome};
use daily_epub::report::RunReport;
use daily_epub::report::{RunReport, VOYAGE_PROVIDER};
use daily_epub::types::{ArticleId, RatingEvent, Vote};
use daily_epub::{curate, http, server, social};
use daily_epub::{curate, http, lock, server, social};
/// A personalized daily newspaper, delivered as an EPUB.
#[derive(Debug, Parser)]
@@ -40,11 +43,27 @@ enum Command {
/// Inspect and edit explicit article verdicts.
#[command(subcommand)]
Ratings(RatingsCommand),
/// Why an article was (not) in the paper, from persisted run telemetry.
Explain(ExplainArgs),
/// The weekly numbers: issues, ratings, retriever yield, cost, timing.
Stats(StatsArgs),
/// Embedding cache and telemetry maintenance.
#[command(subcommand)]
Features(FeaturesCommand),
/// Re-poll social scores for recent entries.
BackfillSocial(BackfillSocialArgs),
/// Database maintenance.
#[command(subcommand)]
Db(DbCommand),
/// Inspect the resolved configuration.
#[command(subcommand)]
Config(ConfigCommand),
}
#[derive(Debug, Subcommand)]
enum ConfigCommand {
/// Load and validate the config as `generate` would, then print one fact per line.
Check,
}
#[derive(Debug, clap::Args)]
@@ -61,9 +80,15 @@ struct GenerateArgs {
/// Cap the lineup size (overrides `target_article_count`).
#[arg(long, value_name = "N")]
max_articles: Option<usize>,
/// Skip every LLM call: prefilter order selects, excerpts stand in for summaries.
/// Skip every LLM call: cheap-signal admission, excerpt summaries.
#[arg(long)]
skip_llm: bool,
/// Use cached embeddings only: zero Voyage calls.
#[arg(long)]
skip_embeddings: bool,
/// Ignore reusable triage/deep assessments and ask the bulk model again.
#[arg(long)]
rescore: bool,
}
impl From<&GenerateArgs> for GenerateOptions {
@@ -74,6 +99,8 @@ impl From<&GenerateArgs> for GenerateOptions {
out: args.out.clone(),
max_articles: args.max_articles,
skip_llm: args.skip_llm,
skip_embeddings: args.skip_embeddings,
rescore: args.rescore,
}
}
}
@@ -132,32 +159,97 @@ impl RatingSetLabel {
#[derive(Debug, clap::Args)]
struct RatingsListArgs {
/// How many days of verdicts to list.
#[arg(long, default_value_t = 90)]
days: i64,
/// Only verdicts with this label.
#[arg(long, value_enum)]
label: Option<RatingListLabel>,
}
#[derive(Debug, clap::Args)]
struct RatingsSetArgs {
/// Article id, as printed by `ratings list` or `explain`.
#[arg(long, required_unless_present = "url", conflicts_with = "url")]
article: Option<ArticleId>,
/// Article URL; canonicalized before lookup.
#[arg(long, required_unless_present = "article", conflicts_with = "article")]
url: Option<String>,
/// The verdict to record.
#[arg(long, value_enum)]
label: RatingSetLabel,
/// Free-text note shown to the weekly profile rebuild.
#[arg(long)]
note: Option<String>,
}
#[derive(Debug, clap::Args)]
struct RatingsClearArgs {
/// Article id, as printed by `ratings list` or `explain`.
#[arg(long, required_unless_present = "url", conflicts_with = "url")]
article: Option<ArticleId>,
/// Article URL; canonicalized before lookup.
#[arg(long, required_unless_present = "article", conflicts_with = "article")]
url: Option<String>,
}
/// `explain --date D (--article ID | --url URL) [--run-id N]` or
/// `explain --date D --near-misses [N]` (plan §15.2).
#[derive(Debug, clap::Args)]
struct ExplainArgs {
/// Issue date whose run to read.
#[arg(long, value_name = "YYYY-MM-DD")]
date: String,
/// Article id, as printed by `ratings list` or `explain --near-misses`.
#[arg(
long,
required_unless_present_any = ["url", "near_misses"],
conflicts_with_all = ["url", "near_misses"]
)]
article: Option<ArticleId>,
/// Article URL; canonicalized before lookup.
#[arg(long, conflicts_with = "near_misses")]
url: Option<String>,
/// A specific run of that date instead of the latest non-dry one.
#[arg(long, value_name = "N")]
run_id: Option<i64>,
/// The top N articles that were considered but not selected (default 10).
#[arg(long, value_name = "N", num_args = 0..=1, default_missing_value = "10")]
near_misses: Option<usize>,
}
/// `stats [--days 14]` (plan §15.3).
#[derive(Debug, clap::Args)]
struct StatsArgs {
/// How many days back to summarize.
#[arg(long, default_value_t = 14)]
days: i64,
}
#[derive(Debug, Subcommand)]
enum FeaturesCommand {
/// Embed rated and published articles, then interests, into the cache.
Backfill(BackfillArgs),
/// Drop stale embeddings, old candidate telemetry and old assessments per the retention config.
Prune,
}
#[derive(Debug, clap::Args)]
struct BackfillArgs {
/// Window for published (and, with --all, other) articles.
#[arg(long, default_value_t = 30)]
days: i64,
/// Only the rated set.
#[arg(long, conflicts_with = "all")]
rated_only: bool,
/// Also every other article first seen inside the window.
#[arg(long)]
all: bool,
/// Skip the confirmation prompt above the token threshold.
#[arg(long)]
yes: bool,
}
#[derive(Debug, clap::Args)]
struct BackfillSocialArgs {
/// How many days back to re-poll.
@@ -178,6 +270,14 @@ async fn main() -> Result<()> {
let config = Config::load(cli.config.as_deref()).context("loading configuration")?;
tracing::debug!(?config.database_path, "configuration loaded");
// One writer at a time (§5); read-only commands never wait on it.
let _lock = match lock_holder(&cli.command) {
Some(name) => {
Some(lock::acquire(&config.database_path, name).map_err(|e| anyhow::anyhow!("{e}"))?)
}
None => None,
};
match cli.command {
Command::Generate(args) => {
let db = Db::open_and_migrate(&config.database_path).await?;
@@ -196,6 +296,19 @@ async fn main() -> Result<()> {
let db = Db::open_and_migrate(&config.database_path).await?;
cmd_ratings(&config, &db, command).await?;
}
Command::Explain(args) => {
let db = Db::open_and_migrate(&config.database_path).await?;
cmd_explain(&db, args).await?;
}
Command::Stats(args) => {
let db = Db::open_and_migrate(&config.database_path).await?;
let text = telemetry::stats(&db, args.days, jiff::Timestamp::now()).await?;
print!("{text}");
}
Command::Features(command) => {
let db = Db::open_and_migrate(&config.database_path).await?;
cmd_features(&config, &db, command).await?;
}
Command::BackfillSocial(args) => {
let db = Db::open_and_migrate(&config.database_path).await?;
cmd_backfill_social(&db, args.days).await?;
@@ -205,10 +318,37 @@ async fn main() -> Result<()> {
db.migrate().await?;
println!("migrations up to date: {}", config.database_path.display());
}
Command::Config(ConfigCommand::Check) => {
// Reaching here means `Config::load` already validated it; a bad
// config exited non-zero above. Nothing is opened, nothing locked.
let path = Config::resolve_path(cli.config.as_deref());
for line in config.check_report(path.as_deref()) {
println!("{line}");
}
}
}
Ok(())
}
/// The commands that write the database and provider budgets and so hold the
/// run lock (§5): `generate`, `profile rebuild`, `features backfill`,
/// `backfill-social`. Everything else is read-only or its own writer.
fn lock_holder(command: &Command) -> Option<&'static str> {
match command {
Command::Generate(_) => Some("generate"),
Command::Profile(ProfileCommand::Rebuild) => Some("profile rebuild"),
Command::Features(FeaturesCommand::Backfill(_)) => Some("features backfill"),
Command::BackfillSocial(_) => Some("backfill-social"),
Command::Serve
| Command::Ratings(_)
| Command::Explain(_)
| Command::Stats(_)
| Command::Features(FeaturesCommand::Prune)
| Command::Db(_)
| Command::Config(_) => None,
}
}
/// `RUST_LOG`-driven tracing, defaulting to `info` (crate table "logging").
fn init_tracing() {
let filter = EnvFilter::try_from_default_env()
@@ -270,13 +410,36 @@ fn print_report(report: &RunReport) {
report.counts.duplicates_merged,
report.counts.entries_dropped,
);
// The same four lines the run logged (§15.4).
for line in report.info_block() {
println!("{line}");
}
println!(
"tokens: {} input · {} cached · {} output = ${:.4}",
"tokens: {} input · {} cache read · {} cache write · {} output · {} voyage = ${:.4}",
report.usage.input_tokens,
report.usage.cached_tokens,
report.usage.cache_write_tokens,
report.usage.output_tokens,
report.voyage_tokens,
report.cost_usd,
);
for (provider, usage) in &report.provider_costs {
if provider == VOYAGE_PROVIDER {
continue; // embedding tokens are printed on their own line below
}
println!(
" {provider}: {} input · {} cache read · {} cache write · {} output = ${:.4}",
usage.usage.input_tokens,
usage.usage.cached_tokens,
usage.usage.cache_write_tokens,
usage.usage.output_tokens,
usage.cost_usd,
);
}
println!(
" voyage: {} tokens = ${:.4}",
report.voyage_tokens, report.voyage_cost_usd
);
for warning in &report.warnings {
println!("warning: {warning}");
}
@@ -316,8 +479,9 @@ fn print_lineup(issue: &daily_epub::types::Issue) {
// Other subcommands
// ---------------------------------------------------------------------------
/// `profile rebuild` runs on the editor when configured, else bulk (§14.3).
async fn cmd_profile_rebuild(config: &Config, db: &Db) -> Result<()> {
let meter = curate::llm::UsageMeter::new(&config.deepseek, config.max_daily_usd);
use curate::llm::{Llms, provider_meters};
let profile = curate::profile::load_or_build(
db,
&config.interests_opml,
@@ -325,10 +489,26 @@ async fn cmd_profile_rebuild(config: &Config, db: &Db) -> Result<()> {
config.curation.feedback.verdicts_in_prompt,
)
.await?;
let llm = curate::llm::LlmClient::new(&config.deepseek, profile.text, meter)?;
let llms = Llms::from_config(config, profile.text, &provider_meters(config));
let Some(llm) = llms.editor_or_bulk() else {
let keys = config
.referenced_providers()
.iter()
.map(|(name, _)| daily_epub::config::ProviderConfig::api_key_env_var(name))
.collect::<Vec<_>>();
anyhow::bail!(
"no LLM provider is available; assign [llm] roles and set {}",
if keys.is_empty() {
"a provider key".to_string()
} else {
keys.join(" or ")
}
);
};
tracing::info!(provider = llm.provider(), model = %llm.model, "rebuilding the profile");
let rebuilt = curate::profile::rebuild(
db,
&llm,
llm,
&config.interests_opml,
&config.profile_path,
config.curation.feedback.verdicts_in_prompt,
@@ -442,6 +622,105 @@ async fn cmd_ratings(config: &Config, db: &Db, command: RatingsCommand) -> Resul
Ok(())
}
async fn cmd_explain(db: &Db, args: ExplainArgs) -> Result<()> {
let date: jiff::civil::Date = args
.date
.parse()
.with_context(|| format!("invalid --date {:?}, expected YYYY-MM-DD", args.date))?;
let text = if let Some(limit) = args.near_misses {
telemetry::explain_near_misses(db, date, args.run_id, limit).await?
} else {
let target = match (args.article, args.url) {
(Some(id), _) => telemetry::ExplainTarget::Article(id),
(None, Some(url)) => telemetry::ExplainTarget::Url(url),
(None, None) => anyhow::bail!("provide --article, --url or --near-misses"),
};
telemetry::explain(db, date, args.run_id, &target).await?
};
print!("{text}");
Ok(())
}
async fn cmd_features(config: &Config, db: &Db, command: FeaturesCommand) -> Result<()> {
match command {
FeaturesCommand::Backfill(args) => {
if !config.voyage.enabled {
anyhow::bail!("voyage.enabled is false; nothing to backfill");
}
let service = embedding::EmbeddingService::real(db.clone(), config.voyage.clone())
.context("building the Voyage client")?;
let opts = embedding::BackfillOptions {
days: args.days,
rated_only: args.rated_only,
all: args.all,
};
let plan = embedding::plan_backfill(db, config, &service, &opts).await?;
println!(
"backfill: {} articles ({} learned, {} other) + {} interests to embed, {} already cached",
plan.article_count(),
plan.learned.len(),
plan.others.len(),
plan.interests.len(),
plan.cached
);
if plan.is_empty() {
println!("cache is warm; nothing to do");
return Ok(());
}
println!(
"estimate: ~{} tokens ≈ ${:.4} with {} at ${:.2}/M",
plan.estimated_tokens,
plan.estimated_cost_usd(),
config.voyage.model,
embedding::VOYAGE_PRICE_PER_MTOK
);
if plan.estimated_tokens > BACKFILL_CONFIRM_TOKENS
&& !args.yes
&& !confirm("continue?")?
{
println!("aborted");
return Ok(());
}
let outcome = embedding::run_backfill(&service, &plan).await?;
println!(
"embedded {} articles and {} interests · {} tokens · ${:.4}",
outcome.articles_embedded,
outcome.interests_embedded,
outcome.tokens,
outcome.cost_usd
);
}
FeaturesCommand::Prune => {
let ranking = &config.curation.ranking;
let pruned = telemetry::prune(
db,
ranking.embedding_retention_days,
ranking.telemetry_retention_days,
jiff::Timestamp::now(),
)
.await?;
println!(
"pruned {} embeddings older than {} days, {} candidate rows and {} assessments older than {} days",
pruned.embeddings,
ranking.embedding_retention_days,
pruned.telemetry,
pruned.assessments,
ranking.telemetry_retention_days
);
}
}
Ok(())
}
/// A y/N question on stdin; anything but a leading `y` is a no.
fn confirm(question: &str) -> Result<bool> {
print!("{question} [y/N] ");
std::io::stdout().flush()?;
let mut answer = String::new();
std::io::stdin().read_line(&mut answer)?;
Ok(answer.trim().to_lowercase().starts_with('y'))
}
async fn cmd_backfill_social(db: &Db, days: u32) -> Result<()> {
let http = http::build_client(http::DEFAULT_TIMEOUT)?;
let enricher = social::SocialEnricher::new(http, db.clone());
@@ -473,6 +752,8 @@ mod tests {
"--max-articles",
"6",
"--skip-llm",
"--skip-embeddings",
"--rescore",
])
.unwrap();
match cli.command {
@@ -482,10 +763,12 @@ mod tests {
assert_eq!(a.out, Some(PathBuf::from("./out")));
assert_eq!(a.max_articles, Some(6));
assert!(a.skip_llm);
assert!(a.skip_embeddings);
assert!(a.rescore);
let opts = GenerateOptions::from(&a);
assert_eq!(opts.date.as_deref(), Some("2026-08-15"));
assert!(opts.dry_run && opts.skip_llm);
assert!(opts.dry_run && opts.skip_llm && opts.skip_embeddings && opts.rescore);
assert_eq!(opts.max_articles, Some(6));
}
other => panic!("expected generate, got {other:?}"),
@@ -547,6 +830,191 @@ mod tests {
assert_eq!(cli.config, Some(PathBuf::from("/tmp/x.toml")));
}
#[test]
fn only_the_writing_commands_take_the_lock() {
let parse = |args: &[&str]| {
Cli::try_parse_from(std::iter::once("daily-epub").chain(args.iter().copied()))
.unwrap()
.command
};
assert_eq!(lock_holder(&parse(&["generate"])), Some("generate"));
assert_eq!(
lock_holder(&parse(&["profile", "rebuild"])),
Some("profile rebuild")
);
assert_eq!(
lock_holder(&parse(&["features", "backfill"])),
Some("features backfill")
);
assert_eq!(
lock_holder(&parse(&["backfill-social"])),
Some("backfill-social")
);
for args in [
vec!["serve"],
vec!["explain", "--date", "2026-09-02", "--near-misses"],
vec!["stats"],
vec!["ratings", "list"],
vec!["db", "migrate"],
vec!["features", "prune"],
vec!["config", "check"],
] {
assert_eq!(lock_holder(&parse(&args)), None, "{args:?}");
}
}
#[test]
fn parses_config_check() {
assert!(matches!(
Cli::try_parse_from(["daily-epub", "--config", "/etc/x.toml", "config", "check"])
.unwrap()
.command,
Command::Config(ConfigCommand::Check)
));
assert!(Cli::try_parse_from(["daily-epub", "config"]).is_err());
}
#[test]
fn parses_stats() {
match Cli::try_parse_from(["daily-epub", "stats"])
.unwrap()
.command
{
Command::Stats(args) => assert_eq!(args.days, 14),
other => panic!("expected stats, got {other:?}"),
}
match Cli::try_parse_from(["daily-epub", "stats", "--days", "7"])
.unwrap()
.command
{
Command::Stats(args) => assert_eq!(args.days, 7),
other => panic!("expected stats, got {other:?}"),
}
}
#[test]
fn parses_explain_and_features() {
match Cli::try_parse_from([
"daily-epub",
"explain",
"--date",
"2026-09-02",
"--article",
"42",
"--run-id",
"7",
])
.unwrap()
.command
{
Command::Explain(args) => {
assert_eq!(args.date, "2026-09-02");
assert_eq!(args.article, Some(42));
assert_eq!(args.run_id, Some(7));
assert_eq!(args.near_misses, None);
}
other => panic!("expected explain, got {other:?}"),
}
match Cli::try_parse_from([
"daily-epub",
"explain",
"--date",
"2026-09-02",
"--url",
"https://example.com/post",
])
.unwrap()
.command
{
Command::Explain(args) => {
assert_eq!(args.url.as_deref(), Some("https://example.com/post"))
}
other => panic!("expected explain, got {other:?}"),
}
match Cli::try_parse_from([
"daily-epub",
"explain",
"--date",
"2026-09-02",
"--near-misses",
])
.unwrap()
.command
{
Command::Explain(args) => assert_eq!(args.near_misses, Some(10)),
other => panic!("expected explain, got {other:?}"),
}
match Cli::try_parse_from([
"daily-epub",
"explain",
"--date",
"2026-09-02",
"--near-misses",
"3",
])
.unwrap()
.command
{
Command::Explain(args) => assert_eq!(args.near_misses, Some(3)),
other => panic!("expected explain, got {other:?}"),
}
assert!(Cli::try_parse_from(["daily-epub", "explain", "--date", "2026-09-02"]).is_err());
assert!(
Cli::try_parse_from([
"daily-epub",
"explain",
"--date",
"2026-09-02",
"--article",
"1",
"--near-misses"
])
.is_err()
);
match Cli::try_parse_from([
"daily-epub",
"features",
"backfill",
"--days",
"60",
"--all",
"--yes",
])
.unwrap()
.command
{
Command::Features(FeaturesCommand::Backfill(args)) => {
assert_eq!(args.days, 60);
assert!(args.all && args.yes && !args.rated_only);
}
other => panic!("expected features backfill, got {other:?}"),
}
match Cli::try_parse_from(["daily-epub", "features", "backfill"])
.unwrap()
.command
{
Command::Features(FeaturesCommand::Backfill(args)) => assert_eq!(args.days, 30),
other => panic!("expected features backfill, got {other:?}"),
}
assert!(
Cli::try_parse_from([
"daily-epub",
"features",
"backfill",
"--rated-only",
"--all"
])
.is_err()
);
assert!(matches!(
Cli::try_parse_from(["daily-epub", "features", "prune"])
.unwrap()
.command,
Command::Features(FeaturesCommand::Prune)
));
}
#[tokio::test]
async fn cli_set_and_clear_append_cli_events_with_latest_issue_date() {
use sqlx::Row as _;
+1346 -102
View File
File diff suppressed because it is too large. Load diff
+1
View File
@@ -752,6 +752,7 @@ mod tests {
editorial: Editorial::default(),
world_briefing: None,
colophon: Colophon::default(),
behind: Default::default(),
}
}
+292 -26
View File
@@ -22,7 +22,7 @@ pub enum RunStatus {
/// Everything completed.
Ok,
/// The issue was produced but a best-effort stage failed (social, XTC,
/// world briefing, images) or the cost guardrail tripped (§3.6).
/// world briefing, images) or a provider budget tripped (§5).
Degraded,
/// No issue was produced.
Failed,
@@ -49,7 +49,7 @@ impl fmt::Display for RunStatus {
}
/// Per-stage article counts as the pipeline narrows the day's feed volume (§2).
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct StageCounts {
/// Entries returned by Miniflux inside the lookback window (§3.1).
pub entries_fetched: i64,
@@ -67,11 +67,52 @@ pub struct StageCounts {
pub excerpt_only: i64,
/// Social lookups that returned a hit (§3.4).
pub social_hits: i64,
/// Articles surviving the heuristic pre-filter (§3.5).
/// Articles passing hygiene and eligible for personalized signals.
pub eligible: i64,
/// Eligible articles with a valid embedding.
pub embedded: i64,
/// Current rated articles with a valid embedding.
pub rated_with_embeddings: i64,
/// The neighbour signal's gate ramp, 0–1 (§9.2); 0 means the signal is absent.
#[serde(default)]
pub knn_gate: f64,
/// The feed-affinity gate ramp, 0–1 (§9.3).
#[serde(default)]
pub feed_gate: f64,
/// Explicit verdicts rendered into the system prompt (§8.4).
#[serde(default)]
pub verdicts_in_prompt: i64,
/// Articles with a reusable or newly produced triage assessment.
pub triaged: i64,
/// Triage assessments served from cached rows.
#[serde(default)]
pub triage_reused: i64,
/// Pool articles without a triage because a provider rejected them
/// (this run's unrecovered rejections plus fresh `provider_rejected` rows).
#[serde(default)]
pub triage_rejected: i64,
/// Articles admitted to close reading.
pub admitted: i64,
/// First admitting retriever counts.
pub admitted_by: BTreeMap<String, i64>,
pub exploration_admitted: i64,
pub exploration_selected: i64,
/// Deep assessments, whether reused or newly produced.
pub assessed: i64,
/// Deep assessments served from cached rows.
#[serde(default)]
pub deep_reused: i64,
/// Admitted articles without a deep assessment because a provider
/// rejected them.
#[serde(default)]
pub deep_rejected: i64,
/// Candidates shown to the editor after diversification.
pub shortlisted: i64,
/// Leader clusters formed over the deep set.
pub clusters: i64,
/// Admitted deep-set count retained for the colophon and runs table.
pub candidates: i64,
/// Articles scored by the LLM (§3.6 stage A).
pub llm_scored: i64,
/// Articles in the final lineup (§3.6 stage B).
/// Articles in the final lineup (§13).
pub selected: i64,
/// Discussion chapters rendered (§3.7).
pub discussions: i64,
@@ -79,6 +120,10 @@ pub struct StageCounts {
pub images_embedded: i64,
}
/// Key under which Voyage usage sits in `provider_costs` (§7.6). Its token count
/// is embedding input, so it stays out of the LLM `usage` aggregate.
pub const VOYAGE_PROVIDER: &str = "voyage";
/// Wall-clock milliseconds per pipeline stage.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct StageTimings(pub BTreeMap<String, i64>);
@@ -93,6 +138,14 @@ impl StageTimings {
}
}
/// Usage and computed cost for one provider in this run (§5).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct ProviderUsage {
#[serde(flatten)]
pub usage: TokenUsage,
pub cost_usd: f64,
}
/// The full summary of one `generate` invocation (§3.13).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct RunReport {
@@ -101,7 +154,15 @@ pub struct RunReport {
pub finished_at: Option<Timestamp>,
pub status: RunStatus,
pub counts: StageCounts,
/// Aggregate usage retained for the legacy `runs` columns.
pub usage: TokenUsage,
/// LLM provider-keyed usage and cost written to `runs.provider_costs_json`.
pub provider_costs: BTreeMap<String, ProviderUsage>,
/// Resolved curation/editorial/model settings for this run.
pub config_json: serde_json::Value,
/// Voyage document/query tokens and cost for this run.
pub voyage_tokens: i64,
pub voyage_cost_usd: f64,
pub cost_usd: f64,
pub timings: StageTimings,
/// Ingest window actually used, RFC3339 (§3.1).
@@ -124,6 +185,10 @@ impl RunReport {
status: RunStatus::Running,
counts: StageCounts::default(),
usage: TokenUsage::default(),
provider_costs: BTreeMap::new(),
config_json: serde_json::Value::Null,
voyage_tokens: 0,
voyage_cost_usd: 0.0,
cost_usd: 0.0,
timings: StageTimings::default(),
window_start: None,
@@ -146,16 +211,24 @@ impl RunReport {
self.error = Some(err.to_string());
}
/// Stamp the end time, compute cost from [`TokenUsage`] and settle the status.
pub fn finish(
&mut self,
finished_at: Timestamp,
price_input: f64,
price_cached: f64,
price_output: f64,
) {
/// Stamp the end time, total provider costs (LLM providers plus Voyage) and
/// settle the status.
///
/// Voyage is counted once: through its `provider_costs` entry when the
/// pipeline recorded one, else through `voyage_cost_usd`.
pub fn finish(&mut self, finished_at: Timestamp) {
self.finished_at = Some(finished_at);
self.cost_usd = self.usage.cost_usd(price_input, price_cached, price_output);
self.usage = TokenUsage::default();
self.cost_usd = 0.0;
for (provider, usage) in &self.provider_costs {
self.cost_usd += usage.cost_usd;
if provider != VOYAGE_PROVIDER {
self.usage.add(usage.usage);
}
}
if !self.provider_costs.contains_key(VOYAGE_PROVIDER) {
self.cost_usd += self.voyage_cost_usd;
}
if self.status == RunStatus::Running {
self.status = if self.warnings.is_empty() {
RunStatus::Ok
@@ -182,11 +255,12 @@ impl RunReport {
/// Compact human-readable summary printed at the end of `generate`.
pub fn summary_line(&self) -> String {
format!(
"{} [{}] {} entries → {} articles → {} candidates → {} selected · ${:.4} · {}s",
"{} [{}] {} entries → {} articles → {} eligible → {} candidates → {} selected · ${:.4} · {}s",
self.date,
self.status,
self.counts.entries_fetched,
self.counts.articles,
self.counts.eligible,
self.counts.candidates,
self.counts.selected,
self.cost_usd,
@@ -194,6 +268,82 @@ impl RunReport {
)
}
/// "23m12s" / "48s" for the log block and `stats`.
pub fn format_duration(secs: i64) -> String {
let secs = secs.max(0);
if secs >= 60 {
format!("{}m{:02}s", secs / 60, secs % 60)
} else {
format!("{secs}s")
}
}
/// The four-line info block of §15.4 (`curation:`, `admission:`,
/// `preference:`, `providers:`), logged once per run and printed by the CLI.
pub fn info_block(&self) -> [String; 4] {
let c = &self.counts;
let admitted = |name: &str| c.admitted_by.get(name).copied().unwrap_or(0);
let gate = |value: f64| {
if value > 0.0 {
format!("{value:.2}")
} else {
"off".to_string()
}
};
let providers = self
.provider_costs
.iter()
.map(|(provider, usage)| format!("{provider} ${:.2}", usage.cost_usd))
.collect::<Vec<_>>()
.join(" · ");
let providers = if providers.is_empty() {
"none".to_string()
} else {
providers
};
let rejected = |count: i64| {
if count > 0 {
format!(" · {count} rejected")
} else {
String::new()
}
};
[
format!(
"curation: {} considered → {} eligible → {} triaged{} → {} assessed{} → {} shortlisted → {} selected",
c.articles,
c.eligible,
c.triaged,
rejected(c.triage_rejected),
c.assessed,
rejected(c.deep_rejected),
c.shortlisted,
c.selected
),
format!(
"admission: triage {} · interest {} · knn {} · exploration {} · blend {} · auto {}",
admitted("triage"),
admitted("interest"),
admitted("knn"),
admitted("exploration"),
admitted("blend"),
admitted("auto_include"),
),
format!(
"preference: {} rated w/ embeddings → knn {} · feed {} · {} verdicts in prompt",
c.rated_with_embeddings,
gate(c.knn_gate),
gate(c.feed_gate),
c.verdicts_in_prompt
),
format!(
"providers: {providers} · total ${:.2} · {}",
self.cost_usd,
Self::format_duration(self.duration_secs().unwrap_or(0))
),
]
}
/// Feeds ordered by entry count, descending — the M1 dry-run breakdown.
pub fn top_feeds(&self, limit: usize) -> Vec<(&str, i64)> {
let mut v: Vec<(&str, i64)> = self
@@ -215,25 +365,126 @@ mod tests {
s.parse().unwrap()
}
fn usage(input: i64, cached: i64, cache_write: i64, output: i64) -> TokenUsage {
TokenUsage {
input_tokens: input,
cached_tokens: cached,
cache_write_tokens: cache_write,
output_tokens: output,
}
}
#[test]
fn finish_computes_cost_and_status() {
fn finish_totals_provider_costs_and_settles_status() {
let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
r.usage.add(TokenUsage {
input_tokens: 1_000_000,
cached_tokens: 1_000_000,
output_tokens: 1_000_000,
});
r.finish(ts("2026-08-15T05:36:00Z"), 0.14, 0.0028, 0.28);
r.provider_costs.insert(
"deepseek".into(),
ProviderUsage {
usage: usage(1_000_000, 1_000_000, 0, 1_000_000),
cost_usd: 0.4228,
},
);
r.provider_costs.insert(
"anthropic".into(),
ProviderUsage {
usage: usage(100, 3_000, 2_000, 800),
cost_usd: 0.05,
},
);
r.voyage_tokens = 250_000;
r.voyage_cost_usd = 0.005;
r.finish(ts("2026-08-15T05:36:00Z"));
assert_eq!(r.status, RunStatus::Ok);
assert!((r.cost_usd - 0.4228).abs() < 1e-9);
// LLM providers plus Voyage; Voyage tokens stay out of the LLM aggregate.
assert!((r.cost_usd - 0.4778).abs() < 1e-9);
// The legacy aggregate columns are the sum across providers.
assert_eq!(r.usage, usage(1_000_100, 1_003_000, 2_000, 1_000_800));
assert_eq!(r.duration_secs(), Some(360));
// Once the pipeline records Voyage as a provider (§7.6) it is counted
// there, not twice, and its tokens still stay out of the LLM aggregate.
r.provider_costs.insert(
VOYAGE_PROVIDER.into(),
ProviderUsage {
usage: usage(250_000, 0, 0, 0),
cost_usd: 0.005,
},
);
r.finish(ts("2026-08-15T05:36:00Z"));
assert!((r.cost_usd - 0.4778).abs() < 1e-9);
assert_eq!(r.usage, usage(1_000_100, 1_003_000, 2_000, 1_000_800));
}
#[test]
fn info_block_has_the_four_lines_of_the_plan() {
let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
r.counts.articles = 412;
r.counts.eligible = 398;
r.counts.triaged = 398;
r.counts.assessed = 120;
r.counts.shortlisted = 60;
r.counts.selected = 17;
r.counts.admitted_by = BTreeMap::from([
("triage".to_string(), 60),
("interest".to_string(), 20),
("knn".to_string(), 12),
("exploration".to_string(), 5),
("blend".to_string(), 23),
]);
r.counts.rated_with_embeddings = 14;
r.counts.knn_gate = 0.35;
r.counts.verdicts_in_prompt = 41;
for (provider, cost) in [("deepseek", 0.11), ("anthropic", 0.62), ("voyage", 0.02)] {
r.provider_costs.insert(
provider.into(),
ProviderUsage {
usage: usage(1, 0, 0, 1),
cost_usd: cost,
},
);
}
r.finish(ts("2026-08-15T05:53:12Z"));
let [curation, admission, preference, providers] = r.info_block();
assert_eq!(
curation,
"curation: 412 considered → 398 eligible → 398 triaged → 120 assessed → 60 shortlisted → 17 selected"
);
assert_eq!(
admission,
"admission: triage 60 · interest 20 · knn 12 · exploration 5 · blend 23 · auto 0"
);
assert_eq!(
preference,
"preference: 14 rated w/ embeddings → knn 0.35 · feed off · 41 verdicts in prompt"
);
assert_eq!(
providers,
"providers: anthropic $0.62 · deepseek $0.11 · voyage $0.02 · total $0.75 · 23m12s"
);
assert_eq!(RunReport::format_duration(48), "48s");
assert_eq!(RunReport::format_duration(3600), "60m00s");
// Provider rejections show up only when there were any.
r.counts.triaged = 395;
r.counts.triage_rejected = 3;
r.counts.deep_rejected = 0;
assert_eq!(
r.info_block()[0],
"curation: 412 considered → 398 eligible → 395 triaged · 3 rejected → 120 assessed → 60 shortlisted → 17 selected"
);
r.counts.assessed = 119;
r.counts.deep_rejected = 1;
assert_eq!(
r.info_block()[0],
"curation: 412 considered → 398 eligible → 395 triaged · 3 rejected → 119 assessed · 1 rejected → 60 shortlisted → 17 selected"
);
}
#[test]
fn warnings_degrade_the_run() {
let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
r.warn("xtc converter missing");
r.finish(ts("2026-08-15T05:31:00Z"), 0.14, 0.0028, 0.28);
r.finish(ts("2026-08-15T05:31:00Z"));
assert_eq!(r.status, RunStatus::Degraded);
assert_eq!(r.warnings.len(), 1);
}
@@ -245,12 +496,27 @@ mod tests {
r.per_feed_counts.insert("Hacker News".into(), 30);
r.per_feed_counts.insert("Lobsters".into(), 12);
r.timings.record("ingest", 1500);
r.finish(ts("2026-08-15T05:31:00Z"), 0.14, 0.0028, 0.28);
r.config_json = serde_json::json!({"models": {"editor": "claude-opus-5"}});
r.provider_costs.insert(
"anthropic".into(),
ProviderUsage {
usage: usage(1, 2, 3, 4),
cost_usd: 0.01,
},
);
r.finish(ts("2026-08-15T05:31:00Z"));
let json = r.to_json();
let back: RunReport = serde_json::from_str(&json).unwrap();
assert_eq!(back, r);
assert_eq!(back.top_feeds(1), vec![("Hacker News", 30)]);
assert_eq!(back.timings.total_ms(), 1500);
assert!(back.summary_line().contains("412 entries"));
// `ProviderUsage` flattens the token counts next to the cost.
let value: serde_json::Value = serde_json::from_str(&json).unwrap();
assert_eq!(
value["provider_costs"]["anthropic"]["cache_write_tokens"],
3
);
assert_eq!(value["provider_costs"]["anthropic"]["cost_usd"], 0.01);
}
}
+153 -43
View File
@@ -246,44 +246,83 @@ pub fn composite_social_score(refs: &[SocialRef]) -> f64 {
}
// ---------------------------------------------------------------------------
// Curation (§3.5, §3.6)
// Curation (plan §10–§13)
// ---------------------------------------------------------------------------
/// DeepSeek stage-A output for one article (§3.6).
/// DeepSeek's close read of one article (§12.1).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct LlmScore {
/// 0–10.
pub score: f64,
pub category: String,
/// ≤ 20 words.
pub struct Deep {
/// Editorial quality on the article's own terms, clamped to 0–10.
pub quality: f64,
/// Fit for this reader, clamped to 0–10.
pub fit: f64,
pub category: Option<String>,
pub rationale: String,
#[serde(default)]
pub is_paywalled_guess: bool,
pub paywalled_guess: bool,
pub facets: Facets,
pub model: String,
pub prompt_version: i64,
pub assessed_at: Timestamp,
}
/// An article carrying every ranking signal computed so far (§3.5, §3.6).
/// Personalized first-pass judgment cached in `article_assessments` (§10).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ScoredArticle {
pub article: Article,
/// Heuristic pre-filter score, 0–100 (§3.5).
pub prefilter_score: f64,
/// Cached [`composite_social_score`] for the article.
pub social_score: f64,
/// `None` until stage A has run (or when `--skip-llm`).
pub llm: Option<LlmScore>,
/// From `curation.always_include_feeds`: may be scored but never dropped (§3.5).
pub auto_include: bool,
pub struct Triage {
/// Reader interest, clamped to 0–10.
pub interest: f64,
pub kind: String,
/// At most twelve words in a conforming response.
pub why: String,
pub model: String,
pub prompt_version: i64,
pub assessed_at: Timestamp,
}
impl ScoredArticle {
/// Ranking key for stage B: LLM score weighted with social proof (§3.6).
pub fn combined_score(&self) -> f64 {
let llm = self.llm.as_ref().map(|l| l.score).unwrap_or(0.0);
llm * 10.0 + self.social_score * 4.0 + self.prefilter_score * 0.1
/// Cached LLM judgments accumulated for an article (§7.3).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Assessment {
pub triage: Option<Triage>,
pub deep: Option<Deep>,
}
/// An article carrying the state of the personalized curation pipeline (§18).
#[derive(Debug, Clone, PartialEq)]
pub struct Candidate {
pub article: Article,
pub auto_include: bool,
pub exploration: bool,
pub signals: crate::curate::signals::Signals,
pub assessment: Assessment,
pub utility: Option<f64>,
pub rank_utility: Option<i64>,
pub cluster: Option<i64>,
pub cluster_rank: Option<i64>,
pub admitted_by: Vec<String>,
pub stage: String,
pub excluded_reason: Option<String>,
}
impl Candidate {
pub fn new(article: Article, auto_include: bool) -> Self {
let signals = crate::curate::signals::Signals::baseline(&article);
Self {
article,
auto_include,
exploration: false,
signals,
assessment: Assessment::default(),
utility: None,
rank_utility: None,
cluster: None,
cluster_rank: None,
admitted_by: Vec::new(),
stage: "eligible".into(),
excluded_reason: None,
}
}
}
/// One selected article with its section placement (§3.6 stage B).
/// One selected article with its section placement (§13).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Pick {
pub article: Article,
@@ -292,20 +331,23 @@ pub struct Pick {
/// Order within the section, ascending.
pub position: i64,
pub is_lead: bool,
/// Editor-written reason, at most 14 words (§13).
pub why: Option<String>,
/// Newspaper-abstract summary from stage C; `None` until editorial runs.
pub summary: Option<String>,
pub llm: Option<LlmScore>,
pub llm: Option<Deep>,
/// Rendered comment chapter, when the article had social refs (§3.7).
pub discussion: Option<Discussion>,
}
/// The day's final lineup: 15–25 picks grouped into sections (§3.6 stage B).
/// The day's final lineup grouped into sections (§13): no minimum size,
/// `curation.max_article_count` (or `--max-articles`) as the ceiling.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Lineup {
pub date: Date,
/// Sorted by (section order, position).
pub picks: Vec<Pick>,
/// Section names in issue order; empty sections are omitted (§3.6).
/// Section names in issue order; empty sections are omitted.
pub section_order: Vec<String>,
}
@@ -326,24 +368,26 @@ impl Lineup {
}
}
/// Stage-C editorial output (§3.6).
/// Editorial output: the Brief and the per-article summaries (§14).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Editorial {
/// "From the Editor", 250–400 words, already sanitized XHTML.
/// The Brief (§14.2), 120–200 words, already sanitized XHTML.
pub front_page_html: String,
/// Section name → 2–3 sentence intro.
pub section_intros: BTreeMap<String, String>,
/// Article id → 2–3 sentence newspaper abstract.
/// Article id → 2–3 sentence newspaper abstract (§14.1).
pub summaries: BTreeMap<ArticleId, String>,
}
/// The taste profile that forms the DeepSeek system prompt (§3.6, `kv`).
/// The reader profile that forms the system prompt shared by every LLM call
/// (§8.4); the current text lives in `kv`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct TasteProfile {
/// Full ~600-word prompt document.
pub text: String,
pub version: i64,
pub built_at: Timestamp,
/// Explicit verdicts rendered into the prompt's "Recent verdicts" block (§8.4).
#[serde(default)]
pub verdicts: usize,
}
// ---------------------------------------------------------------------------
@@ -523,12 +567,24 @@ pub struct Issue {
pub world_briefing: Option<WorldBriefing>,
/// Colophon facts: models used, token cost, feed counts (§3.10).
pub colophon: Colophon,
/// The "Behind the paper" chapter's facts (§15.1); filled by the pipeline.
#[serde(default)]
pub behind: BehindThePaper,
}
/// Resolved model names printed in the colophon (§15.1).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Models {
pub bulk: String,
pub editor: String,
pub summaries: String,
}
/// Back-matter facts printed in the colophon chapter (§3.10).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Colophon {
pub model: String,
pub provider_costs: BTreeMap<String, f64>,
pub models: Models,
pub entries_fetched: i64,
pub feeds_seen: i64,
pub candidates: i64,
@@ -536,6 +592,49 @@ pub struct Colophon {
pub generator_version: String,
}
/// One of the highest-utility articles that did not make the paper (§15.1).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct NearMiss {
pub article_id: ArticleId,
pub title: String,
pub feed_title: String,
pub quality: Option<f64>,
pub fit: Option<f64>,
/// `candidate_runs.stage` reached.
pub stage: String,
/// `candidate_runs.excluded_reason`, when the row carries one.
pub reason: Option<String>,
}
/// Facts for the "Behind the paper" chapter (§15.1), derived from the run's
/// report and its `candidate_runs` rows. Templates render it verbatim.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct BehindThePaper {
/// Deduplicated articles the run looked at.
pub considered: i64,
pub feeds_seen: i64,
pub eligible: i64,
pub triaged: i64,
/// Deep-assessed ("read closely").
pub read_closely: i64,
pub shortlisted: i64,
pub selected: i64,
/// First admitting retriever → count (§11).
pub admitted_by: BTreeMap<String, i64>,
pub rated_with_embeddings: i64,
/// The neighbour signal's gate ramp, 0–1 (§9.2).
pub knn_gate: f64,
/// The feed-affinity gate ramp, 0–1 (§9.3).
pub feed_gate: f64,
pub near_misses: Vec<NearMiss>,
pub models: Models,
pub embedding_model: String,
/// Every provider, Voyage included.
pub cost_usd: f64,
/// Wall clock from the run's start to issue assembly.
pub generation_secs: i64,
}
/// A downloaded, re-encoded image embedded in an edition (§3.10 images).
#[derive(Debug, Clone, PartialEq)]
pub struct ImageAsset {
@@ -616,7 +715,8 @@ pub struct RatingEvent {
pub event_at: Timestamp,
}
/// Descriptive deep-assessment facets (§12.1), populated beginning in step 5.
/// Descriptive deep-assessment facets (§12.1); shown to the editor, the profile
/// rebuild and `explain`, never a numeric ranking signal.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Facets {
pub format: Option<String>,
@@ -645,16 +745,18 @@ pub struct RatedArticle {
}
// ---------------------------------------------------------------------------
// LLM accounting (§3.6 cost guardrail)
// LLM accounting (§5 per-provider budgets)
// ---------------------------------------------------------------------------
/// Token counters accumulated across every DeepSeek call in a run (§3.6).
/// Token counters accumulated across one provider's calls in a run (§5).
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct TokenUsage {
/// Cache-miss input tokens (billed at the full input rate).
pub input_tokens: i64,
/// Prefix-cache hits (billed at the cached rate).
/// Prefix-cache reads (billed at the provider's cache-read rate).
pub cached_tokens: i64,
/// Tokens written into a prompt cache (Anthropic only).
pub cache_write_tokens: i64,
pub output_tokens: i64,
}
@@ -662,13 +764,21 @@ impl TokenUsage {
pub fn add(&mut self, other: TokenUsage) {
self.input_tokens += other.input_tokens;
self.cached_tokens += other.cached_tokens;
self.cache_write_tokens += other.cache_write_tokens;
self.output_tokens += other.output_tokens;
}
/// USD cost given the per-1M-token prices from `[deepseek]` config (§3.6).
pub fn cost_usd(&self, price_input: f64, price_cached: f64, price_output: f64) -> f64 {
/// USD cost given a provider's per-1M-token prices (§4.1–§4.2).
pub fn cost_usd(
&self,
price_input: f64,
price_cache_write: f64,
price_cache_read: f64,
price_output: f64,
) -> f64 {
(self.input_tokens as f64 * price_input
+ self.cached_tokens as f64 * price_cached
+ self.cache_write_tokens as f64 * price_cache_write
+ self.cached_tokens as f64 * price_cache_read
+ self.output_tokens as f64 * price_output)
/ 1_000_000.0
}
+8 -5
View File
@@ -160,11 +160,11 @@ fn dropped(element: &scraper::node::Element) -> bool {
fn node_text(node: NodeRef<'_>, skip_lists: bool, out: &mut String) {
match node.value() {
Node::Text(text) => {
out.push_str(&text);
out.push_str(text);
out.push(' ');
}
Node::Element(element) => {
if dropped(&element) || (skip_lists && matches!(element.name(), "ul" | "ol")) {
if dropped(element) || (skip_lists && matches!(element.name(), "ul" | "ol")) {
return;
}
for child in node.children() {
@@ -208,7 +208,7 @@ fn links_without_child_lists(element: ElementRef<'_>, base: &Url) -> Vec<String>
let Node::Element(element) = node.value() else {
return;
};
if dropped(&element) || matches!(element.name(), "ul" | "ol") {
if dropped(element) || matches!(element.name(), "ul" | "ol") {
return;
}
if element.name() == "a"
@@ -718,11 +718,14 @@ mod tests {
backend: std::sync::Arc<crate::curate::llm::MockBackend>,
limit: f64,
) -> LlmClient {
let config = crate::config::DeepseekConfig::default();
let config = crate::config::ProviderConfig::deepseek();
LlmClient::with_backend(
"mock",
"World Briefing test".into(),
crate::curate::llm::UsageMeter::new(&config, limit),
crate::curate::llm::UsageMeter::with_prices(
crate::curate::llm::PriceTable::from(&config),
limit,
),
backend,
)
}
+69
View File
@@ -0,0 +1,69 @@
//! `daily-epub config check` end to end: the built binary, a temp config, no
//! environment. It must validate like `generate`, print the provider table
//! with `key MISSING` warnings, exit 0, and exit non-zero on an invalid file —
//! all without a database or the run lock.
use std::path::Path;
use std::process::Command;
fn run(config_body: &str) -> (i32, String, String) {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("config.toml");
std::fs::write(&path, config_body).expect("write config");
let output = Command::new(env!("CARGO_BIN_EXE_daily-epub"))
.env_clear()
.env("PATH", std::env::var_os("PATH").unwrap_or_default())
.args(["--config"])
.arg(&path)
.args(["config", "check"])
.output()
.expect("run daily-epub");
(
output.status.code().unwrap_or(-1),
String::from_utf8_lossy(&output.stdout).into_owned(),
String::from_utf8_lossy(&output.stderr).into_owned(),
)
}
#[test]
fn config_check_prints_the_facts_and_exits_zero_without_keys() {
let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml");
let body = std::fs::read_to_string(example).expect("example config");
let (code, stdout, stderr) = run(&body);
assert_eq!(code, 0, "stdout:\n{stdout}\nstderr:\n{stderr}");
for needle in [
"config: ",
"database_path: /var/lib/daily-epub/daily-epub.db",
"profile_path: data/profile.md",
"interests_opml: data/scour-interests.opml",
"llm.bulk: deepseek · openai · deepseek-v4-flash",
"key MISSING (set DAILY_EPUB_PROVIDERS__DEEPSEEK__API_KEY)",
"llm.editor: anthropic · anthropic · claude-opus-5 · effort high · max_daily_usd $3.00",
"key MISSING (set DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY)",
"providers.gemini: unreferenced",
"voyage: voyage-4-lite · enabled · max_daily_usd $0.50 · key MISSING (set DAILY_EPUB_VOYAGE__API_KEY)",
"editorial.summary_model: editor",
"publish.epub_dir: /srv/bookorbit/libraries/daily-epub",
"publish.xtc_dir: /var/lib/daily-epub/xtc",
] {
assert!(stdout.contains(needle), "missing {needle:?} in:\n{stdout}");
}
assert!(
stdout.lines().any(|line| line.starts_with("! ")),
"missing keys are flagged with a `!` prefix:\n{stdout}"
);
// No lock file, no database: the command is read-only.
assert!(!Path::new("/var/lib/daily-epub/daily-epub.db.lock").exists());
}
#[test]
fn config_check_exits_non_zero_on_an_invalid_config() {
let (code, stdout, stderr) = run("[llm]\nbulk = \"nope\"\n");
assert_ne!(code, 0);
assert!(stdout.is_empty(), "{stdout}");
assert!(stderr.contains("providers.nope"), "{stderr}");
let (code, _, stderr) = run("[deepseek]\nmodel = \"x\"\n");
assert_ne!(code, 0);
assert!(stderr.contains("[providers.deepseek]"), "{stderr}");
}
+164 -45
View File
@@ -2,14 +2,14 @@
//! (spec §2, §5).
//!
//! ```text
//! synthetic entries → dedupe → extract (offline) → persist → prefilter
//! synthetic entries → dedupe → extract (offline) → persist → admission
//! → select → editorial → issue → both EPUB editions → publish → OPDS + rows
//! ```
//!
//! Two passes over the same machinery:
//!
//! * [`skip_llm_pipeline_produces_a_published_issue`] takes the `--skip-llm`
//! route (prefilter order selects, feed excerpts stand in for summaries);
//! route (preliminary blend selects, feed excerpts stand in for summaries);
//! * [`llm_pipeline_runs_against_a_mock_backend`] takes the DeepSeek route with
//! [`MockBackend`] standing in for the API, so stages A, B and C are all
//! exercised — prompts, parsers, budget accounting and all — offline.
@@ -18,18 +18,19 @@
//! no article in the fixtures carries an image, so the EPUB builder's image
//! downloader has nothing to fetch.
use std::collections::{BTreeMap, HashMap};
use std::path::Path;
use jiff::Timestamp;
use jiff::civil::Date;
use daily_epub::config::{Config, PublishConfig, ServerConfig, XtcConfig};
use daily_epub::curate::llm::{LlmClient, MockBackend, UsageMeter};
use daily_epub::curate::{Curator, editorial, prefilter};
use daily_epub::curate::llm::{LlmClient, Llms, MockBackend, UsageMeter};
use daily_epub::curate::{Curator, admit, editorial, rank};
use daily_epub::db::Db;
use daily_epub::extract::Extractor;
use daily_epub::types::{
Article, Colophon, Edition, Entry, Issue, Lineup, ScoredArticle, SourceKind, Vote,
Article, Candidate, Colophon, Edition, Entry, Issue, Lineup, Models, SourceKind, Vote,
};
use daily_epub::{auth, dedupe, epub, miniflux, pipeline, publish};
@@ -56,7 +57,6 @@ fn test_config(root: &Path) -> Config {
database_path: root.join("db").join("daily-epub.db"),
out_dir: root.join("out"),
target_article_count: 6,
prefilter_keep: 20,
world_briefing: false,
publish: PublishConfig {
epub_dir: root.join("bookorbit"),
@@ -400,24 +400,32 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
let articles = ingest_dedupe_extract_persist(&db).await;
// --- Stages 6–7 with no LLM at all (notes §6) ---
let curator = Curator::new(cfg.clone(), db.clone(), None);
let candidates = curator
.prefilter(articles, date())
.await
.expect("prefilter runs");
let curator = Curator::new(cfg.clone(), db.clone(), Llms::default());
let mut personalized = articles
.into_iter()
.map(|article| Candidate::new(article, false))
.collect::<Vec<_>>();
for candidate in &mut personalized {
candidate.signals.preliminary = candidate.signals.heuristic;
}
admit::admit(&mut personalized, date(), &cfg.curation.ranking);
let candidates = personalized
.into_iter()
.filter(|candidate| candidate.stage == "admitted")
.collect::<Vec<_>>();
assert_eq!(candidates.len(), 5, "nothing is dropped at this volume");
assert!(
candidates
.windows(2)
.all(|w| w[0].prefilter_score >= w[1].prefilter_score),
"candidates come back in prefilter order"
);
// The excerpt-only story is penalized (§3.5).
let allocator = candidates
.iter()
.find(|c| c.article.excerpt_only)
.expect("the allocator teaser survived");
assert!(allocator.prefilter_score < candidates[0].prefilter_score);
assert!(
allocator.signals.heuristic.unwrap_or_default()
< candidates
.iter()
.filter_map(|candidate| candidate.signals.heuristic)
.fold(f64::NEG_INFINITY, f64::max)
);
let lineup = curator.select(candidates, date()).await.expect("select");
assert_eq!(lineup.picks.len(), 5, "target 6, only 5 candidates exist");
@@ -432,7 +440,12 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
);
let colophon = Colophon {
model: "none (--skip-llm)".into(),
provider_costs: BTreeMap::new(),
models: Models {
bulk: "none".into(),
editor: "none".into(),
summaries: "none".into(),
},
entries_fetched: 8,
feeds_seen: 8,
candidates: 5,
@@ -491,19 +504,28 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
.expect("open db");
let articles = ingest_dedupe_extract_persist(&db).await;
let ctx = prefilter::PrefilterContext::load(&db, date())
.await
.expect("prefilter context");
let candidates: Vec<ScoredArticle> = prefilter::run(articles, &ctx, &cfg);
let mut personalized = articles
.into_iter()
.map(|article| Candidate::new(article, false))
.collect::<Vec<_>>();
for candidate in &mut personalized {
candidate.signals.preliminary = candidate.signals.heuristic;
}
admit::admit(&mut personalized, date(), &cfg.curation.ranking);
let candidates = personalized
.into_iter()
.filter(|candidate| candidate.stage == "admitted")
.collect::<Vec<_>>();
let ids: Vec<i64> = candidates.iter().map(|c| c.article.id).collect();
assert_eq!(ids.len(), 5);
// --- Script DeepSeek: one stage-A batch, one stage-B call, five stage-C
// summaries and one front page (§3.6). ---
// --- Script DeepSeek: one deep-assessment batch, one editor call, five
// summaries and one brief. ---
let backend = std::sync::Arc::new(MockBackend::new());
let usage = daily_epub::types::TokenUsage {
input_tokens: 1000,
cached_tokens: 500,
cache_write_tokens: 0,
output_tokens: 200,
};
let scores: Vec<String> = ids
@@ -511,8 +533,8 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
.enumerate()
.map(|(i, id)| {
format!(
r#"{{"id": {id}, "score": {}, "category": "Tech & Engineering",
"rationale": "solid systems writeup", "is_paywalled_guess": false}}"#,
r#"{{"id": {id}, "quality": {}, "fit": 7, "category": "Tech & Engineering",
"rationale": "solid systems writeup", "paywalled_guess": false, "facets": {{"format":"analysis_essay"}}}}"#,
9 - i
)
})
@@ -544,31 +566,37 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
);
}
backend.push(
r#"{"from_the_editor": "Today's issue leans on storage internals.\n\nRead on.",
"section_intros": {"Top Stories": "The day in one place."}}"#,
r#"{"brief": "Today's issue leans on storage internals.\n\nRead on."}"#,
usage,
);
let meter = UsageMeter::new(&cfg.deepseek, cfg.max_daily_usd);
let meter = UsageMeter::for_provider(&cfg.providers["deepseek"]);
let llm = LlmClient::with_backend(
&cfg.deepseek.model,
&cfg.providers["deepseek"].model,
"You are the editor of The Daily EPUB.".into(),
meter.clone(),
backend.clone(),
);
let curator = Curator::new(cfg.clone(), db.clone(), Some(llm));
let curator = Curator::new(
cfg.clone(),
db.clone(),
Llms {
bulk: Some(llm),
editor: None,
},
);
let mut candidates = candidates;
curator
.score(&mut candidates, date())
.assess(&mut candidates, true, None, jiff::Timestamp::now())
.await
.expect("stage A");
.expect("deep assessment");
assert!(
candidates.iter().all(|c| c.llm.is_some()),
"every candidate came back scored"
candidates.iter().all(|c| c.assessment.deep.is_some()),
"every candidate came back assessed"
);
let lineup = curator.select(candidates, date()).await.expect("stage B");
let lineup = curator.select(candidates, date()).await.expect("editor");
assert_eq!(lineup.picks.len(), 5);
assert_eq!(
lineup.lead().map(|p| p.article.id),
@@ -590,12 +618,9 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
"the model's summaries were used, not excerpts"
);
assert!(editorial_doc.front_page_html.contains("storage internals"));
assert_eq!(
editorial_doc
.section_intros
.get("Top Stories")
.map(String::as_str),
Some("The day in one place.")
assert!(
lineup.picks.iter().all(|p| p.why.is_none()),
"the scripted editor gave no why lines"
);
// Every scripted response was consumed, and the meter priced them (§3.6).
@@ -615,7 +640,12 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
// And it all assembles, builds and publishes like the skip-llm route does.
let colophon = Colophon {
model: cfg.deepseek.model.clone(),
provider_costs: BTreeMap::from([("deepseek".to_string(), meter.cost_usd())]),
models: Models {
bulk: cfg.providers["deepseek"].model.clone(),
editor: format!("{} (bulk fallback)", cfg.providers["deepseek"].model),
summaries: cfg.providers["deepseek"].model.clone(),
},
entries_fetched: 8,
feeds_seen: 8,
candidates: 5,
@@ -625,6 +655,95 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
let mut lineup = lineup;
pipeline::apply_summaries(&mut lineup, &editorial_doc);
let issue = assemble_build_publish(&db, &cfg, lineup, colophon).await;
assert_eq!(issue.colophon.model, cfg.deepseek.model);
assert_eq!(issue.colophon.models.bulk, cfg.providers["deepseek"].model);
assert!(issue.colophon.cost_usd > 0.0);
}
#[tokio::test]
async fn failing_deepseek_still_publishes_with_heuristic_fallbacks() {
let root = tempfile::tempdir().expect("tempdir");
let cfg = test_config(root.path());
let db = Db::open_and_migrate(&cfg.database_path)
.await
.expect("open db");
let articles = ingest_dedupe_extract_persist(&db).await;
let mut personalized = articles
.into_iter()
.map(|article| Candidate::new(article, false))
.collect::<Vec<_>>();
for candidate in &mut personalized {
candidate.signals.preliminary = candidate.signals.heuristic;
}
admit::admit(&mut personalized, date(), &cfg.curation.ranking);
let mut candidates = personalized
.into_iter()
.filter(|candidate| candidate.stage == "admitted")
.collect::<Vec<_>>();
let backend = std::sync::Arc::new(MockBackend::new());
let client = LlmClient::with_backend(
&cfg.providers["deepseek"].model,
"reader profile".into(),
UsageMeter::for_provider(&cfg.providers["deepseek"]),
backend.clone(),
);
let curator = Curator::new(
cfg.clone(),
db.clone(),
Llms {
bulk: Some(client),
editor: None,
},
);
curator
.assess(&mut candidates, true, None, jiff::Timestamp::now())
.await
.expect("failed batches degrade, not abort");
assert!(
candidates
.iter()
.all(|candidate| candidate.assessment.deep.is_none())
);
// Utility over the present signals, clustered as singletons without
// embeddings: every admitted article is shortlisted with a cluster id.
let ranked = rank::shortlist(&mut candidates, &HashMap::new(), &cfg.curation.ranking);
assert_eq!(ranked.shortlisted, candidates.len());
assert_eq!(ranked.clusters, candidates.len());
for candidate in &candidates {
assert_eq!(candidate.stage, "shortlisted");
assert!(candidate.utility.is_some() && candidate.cluster.is_some());
assert!(candidate.rank_utility.is_some());
}
let mut lineup = curator
.select(candidates, date())
.await
.expect("fallback lineup");
let editorial = curator
.editorial(&lineup)
.await
.expect("fallback editorial");
pipeline::apply_summaries(&mut lineup, &editorial);
assert!(!lineup.picks.is_empty());
assert!(backend.calls() > 0, "the failing backend was exercised");
let issue = assemble_build_publish(
&db,
&cfg,
lineup,
Colophon {
provider_costs: BTreeMap::new(),
models: Models {
bulk: cfg.providers["deepseek"].model.clone(),
editor: format!("{} (bulk fallback)", cfg.providers["deepseek"].model),
summaries: cfg.providers["deepseek"].model.clone(),
},
entries_fetched: 8,
feeds_seen: 8,
candidates: 5,
cost_usd: 0.0,
generator_version: format!("daily-epub {}", daily_epub::VERSION),
},
)
.await;
assert!(!issue.lineup.picks.is_empty());
assert_eq!(std::fs::read_dir(&cfg.publish.epub_dir).unwrap().count(), 2);
}
+3
View File
@@ -0,0 +1,3 @@
{
"brief": "The lead, \"Migrating 40TB off Postgres\", is the rare migration write-up that keeps its failures in: two aborted cutovers, the rollback that took longer than the move, and a bill at the end. Read it first while the coffee is hot; it rewards attention and it is long.\n\nThe local desk answers with \"The MBTA slow-zone dataset\", which finally puts the T's own numbers into a shape a rider can argue with, and the charts do more persuading than a year of press releases. \"A failover story you'd argue with\" rounds out the engineering pages with a Postgres HA design that disagrees with the lead on almost every point, which is exactly why the two belong in the same issue. The issue is shorter than usual because a thin Friday is a good excuse to finish the long one properly rather than skim six."
}
+76
View File
@@ -0,0 +1,76 @@
{
"articles": [
{
"id": 101,
"quality": 8.5,
"fit": 7.0,
"category": "Tech & Engineering",
"rationale": "First-hand 40TB Postgres migration with failure timeline, numbers and a rollback plan",
"paywalled_guess": false,
"facets": {
"format": "first_hand_account",
"depth": "deep",
"evidence": "first_hand",
"commerciality": "none",
"topic_group": "software_engineering",
"technicality": "advanced",
"locality": "not_applicable",
"specific_topics": ["Postgres migration", "storage failover", "rollback planning"]
}
},
{
"id": 102,
"quality": 3.0,
"fit": 2.5,
"category": "AI & Machine Learning",
"rationale": "Model release announcement, benchmark table from the vendor, no independent evaluation",
"paywalled_guess": false,
"facets": {
"format": "announcement_roundup",
"depth": "brief",
"evidence": "speculative",
"commerciality": "promotional",
"topic_group": "ai_ml",
"technicality": "light",
"locality": "international",
"specific_topics": ["model launch"]
}
},
{
"id": 103,
"quality": 6.5,
"fit": 8.0,
"category": "Boston & Local",
"rationale": "MBTA slow-zone data pulled from the tracker and charted by line with original analysis",
"paywalled_guess": false,
"facets": {
"format": "analysis_essay",
"depth": "standard",
"evidence": "data_or_experiment",
"commerciality": "none",
"topic_group": "boston_new_england",
"technicality": "intermediate",
"locality": "boston_new_england",
"specific_topics": ["MBTA slow zones", "transit data"]
}
},
{
"id": 104,
"quality": 5.0,
"fit": 6.0,
"category": "Culture & Essays",
"rationale": "Promising essay on typesetting history; the body reads cut off after the second section",
"paywalled_guess": true,
"facets": {
"format": "analysis_essay",
"depth": "standard",
"evidence": "synthesis",
"commerciality": "none",
"topic_group": "books_writing",
"technicality": "nontechnical",
"locality": "not_applicable",
"specific_topics": ["typesetting", "print history"]
}
}
]
}
+67
View File
@@ -0,0 +1,67 @@
{
"articles": [
{
"id": 201,
"quality": 7.0,
"fit": 6.5,
"category": "Science & Space",
"rationale": "careful write-up of an amateur radio occultation measurement",
"paywalled_guess": false,
"facets": {
"format": "first_hand_account",
"depth": "standard",
"evidence": "data_or_experiment",
"commerciality": "none",
"topic_group": "science_space",
"technicality": "intermediate",
"locality": "us",
"specific_topics": ["radio occultation"]
}
},
{
"id": "202",
"quality": "6",
"fit": "5.5",
"category": "Niche Corner",
"rationale": "mailing-list argument about tape drives, oddly gripping",
"paywalled_guess": "false",
"facets": {
"format": "discussion_thread",
"depth": "standard",
"evidence": "anecdote",
"commerciality": "none",
"topic_group": "retro_computing",
"technicality": "intermediate",
"locality": "not_applicable",
"specific_topics": ["LTO tape", "backups", "archival", "vendors", "pricing"]
}
},
{
"id": 203,
"quality": 4,
"fit": 4
},
{
"id": 204,
"quality": 12.5,
"fit": -1,
"category": "Sports",
"rationale": "model ignored the rubric ceiling and invented a section here",
"paywalled_guess": false,
"facets": "not an object"
},
{
"id": 205,
"quality": 8.0,
"category": "Top Stories",
"rationale": "fit is missing, so this item is unusable"
},
{
"quality": 9.0,
"fit": 9.0,
"category": "Top Stories",
"rationale": "no id at all, unusable"
},
"a bare string where an object belongs"
]
}
-8
View File
@@ -1,8 +0,0 @@
{
"from_the_editor": "Two of today's pieces are, underneath, the same story: what it costs to move data you no longer trust. The lead — a team hauling forty terabytes off Postgres, rollback plans and all — is the version with the invoices attached, and it earns the front page by refusing to tidy up its failures. Read it first, while the coffee is hot; it rewards attention and it is long.\n\nThe local desk supplies the counterpoint. Somebody has finally put the MBTA's slow-zone data into a shape a rider can argue with, and the charts do more persuading than a year of press releases. It is a short read and a satisfying one, and it pairs unreasonably well with the migration story: both are about institutions discovering what they actually have.\n\nThe rest of the issue is quieter than usual. That is not a complaint — a thin Friday is a good excuse to finish the long one properly rather than skimming six. If you only get through the lead today, you will not have missed much else.",
"section_intros": {
"Top Stories": "The day's most substantial piece: a full account of a forty-terabyte migration, with the failures left in. It is long, technical and unusually honest about what went wrong.",
"Boston & Local": "Transit data gets the treatment it deserves. A rider-built analysis of MBTA slow zones, with charts you can check yourself and a methodology section that holds up.",
"Niche Corner": "A section the model wrote an intro for even though nothing was placed in it today — a stray thread about tape-drive firmware and the people who still maintain it. The issue drops intros for sections that never ran."
}
}
+5 -5
View File
@@ -1,10 +1,10 @@
{
"picks": [
{ "id": 101, "section": "Top Stories", "position": 1, "lead_story": true },
{ "id": 102, "section": "Tech & Engineering", "position": 1, "lead_story": false },
{ "id": 103, "section": "Tech & Engineering", "position": 2, "lead_story": false },
{ "id": 104, "section": "AI & Machine Learning", "position": 1, "lead_story": false },
{ "id": 105, "section": "Boston & Local", "position": 1, "lead_story": false },
{ "id": 101, "section": "Top Stories", "position": 1, "lead_story": true, "why": "The migration post-mortem with the invoices still attached" },
{ "id": 102, "section": "Tech & Engineering", "position": 1, "lead_story": false, "why": "A failover story you'd argue with over coffee" },
{ "id": 103, "section": "Tech & Engineering", "position": 2, "lead_story": false, "why": "Rare first-hand detail on a tool you use daily" },
{ "id": 104, "section": "AI & Machine Learning", "position": 1, "lead_story": false, "why": "The one benchmark piece this week that shows its work" },
{ "id": 105, "section": "Boston & Local", "position": 1, "lead_story": false, "why": "MBTA slow zones charted by a rider, not a press office" },
{ "id": 106, "section": "Culture & Essays", "position": 1 },
{ "section": "Niche Corner", "position": 2, "lead_story": false },
"the model sometimes trails off like this"
-32
View File
@@ -1,32 +0,0 @@
{
"articles": [
{
"id": 101,
"score": 8.5,
"category": "Tech & Engineering",
"rationale": "first-hand 40TB Postgres migration with numbers, failures and rollback plan",
"is_paywalled_guess": false
},
{
"id": 102,
"score": 3.0,
"category": "AI & Machine Learning",
"rationale": "model release announcement, no independent evaluation",
"is_paywalled_guess": false
},
{
"id": 103,
"score": 6.5,
"category": "Boston & Local",
"rationale": "MBTA slow-zone data analysis with original charts",
"is_paywalled_guess": false
},
{
"id": 104,
"score": 5.0,
"category": "Culture & Essays",
"rationale": "promising essay on typesetting, body appears truncated",
"is_paywalled_guess": true
}
]
}
-35
View File
@@ -1,35 +0,0 @@
{
"articles": [
{
"id": 201,
"score": 7.0,
"category": "Science & Space",
"rationale": "careful write-up of an amateur radio occultation measurement",
"is_paywalled_guess": false
},
{
"id": "202",
"score": "6",
"category": "Niche Corner",
"rationale": "mailing-list argument about tape drives, oddly gripping",
"is_paywalled_guess": "false"
},
{
"id": 203,
"score": 4
},
{
"id": 204,
"score": 12.5,
"category": "Top Stories",
"rationale": "model ignored the rubric ceiling here",
"is_paywalled_guess": false
},
{
"score": 9.0,
"category": "Top Stories",
"rationale": "no id at all, unusable"
},
"a bare string where an object belongs"
]
}
+16
View File
@@ -0,0 +1,16 @@
{
"articles": [
{
"id": 4821,
"interest": 7.5,
"kind": "first_hand",
"why": "Specific field notes from a difficult migration"
},
{
"id": 4822,
"interest": 3.0,
"kind": "news",
"why": "Competent but familiar daily industry coverage"
}
]
}
+82 -60
View File
@@ -3,8 +3,8 @@
//! The stage logic is unit-tested inside `src/curate/*`. What this file guards is
//! the contract *between* the curation stages and everything around them:
//!
//! * the recorded DeepSeek fixtures still parse through the real
//! `score.rs` / `select.rs` / `editorial.rs` parsers into the structures the
//! * recorded DeepSeek fixtures still parse through the real
//! `assess.rs` / `editor.rs` / `editorial.rs` parsers into the structures the
//! pipeline consumes, and the lenient parsers still cope with the messy one;
//! * the shipped Scour OPML still yields the ~220 interests the taste profile is
//! assembled from (§3.6a);
@@ -17,11 +17,10 @@ use std::collections::BTreeSet;
use std::path::{Path, PathBuf};
use std::process::Command;
use daily_epub::curate::editorial::FrontPageResponse;
use daily_epub::curate::assess::parse_deep_response;
use daily_epub::curate::editor::parse_selection_response;
use daily_epub::curate::editorial::BriefResponse;
use daily_epub::curate::profile;
use daily_epub::curate::score::parse_score_response;
use daily_epub::curate::select::parse_selection_response;
use daily_epub::types::LlmScore;
fn repo(rel: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR")).join(rel)
@@ -32,71 +31,88 @@ fn fixture(name: &str) -> String {
std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("reading {}: {e}", path.display()))
}
/// Stage A responses must parse into `{id, score, category, rationale,
/// is_paywalled_guess}` per article (§3.6).
/// Deep responses must parse into `{id, quality, fit, category, rationale,
/// paywalled_guess, facets}` per article (§12.1).
#[test]
fn stage_a_fixture_parses_into_scores() {
let items = parse_score_response(&fixture("deepseek_score_batch.json"));
fn deep_fixture_parses_into_assessments() {
let sections = daily_epub::config::CurationConfig::default().sections;
let items = parse_deep_response(&fixture("deepseek_deep_batch.json"), &sections);
assert!(items.len() >= 4, "fixture should cover a realistic batch");
for item in &items {
assert!(item.id > 0, "every item carries a positive article id");
assert!((0.0..=10.0).contains(&item.quality));
assert!((0.0..=10.0).contains(&item.fit));
assert!(item.category.is_some(), "every category is on the palette");
assert!(
(0.0..=10.0).contains(&item.score),
"score {} out of range",
item.score
);
assert!(!item.category.is_empty());
assert!(
item.rationale.split_whitespace().count() <= 20,
"rationale must stay under 20 words: {:?}",
item.rationale.split_whitespace().count() <= 25,
"rationale must stay under 25 words: {:?}",
item.rationale
);
assert!(item.facets.format.is_some() && item.facets.topic_group.is_some());
}
assert!(
items.iter().any(|i| i.is_paywalled_guess),
items.iter().any(|item| item.paywalled_guess),
"the fixture should exercise the paywall flag"
);
// Quality and fit are judged separately: the fixture has an article whose
// fit exceeds its quality and one the other way round.
assert!(items.iter().any(|item| item.fit > item.quality));
assert!(items.iter().any(|item| item.quality > item.fit));
// The batch spans the rubric rather than clustering at one score.
let scores: Vec<f64> = items.iter().map(|i| i.score).collect();
let spread = scores.iter().cloned().fold(f64::MIN, f64::max)
- scores.iter().cloned().fold(f64::MAX, f64::min);
let qualities: Vec<f64> = items.iter().map(|i| i.quality).collect();
let spread = qualities.iter().cloned().fold(f64::MIN, f64::max)
- qualities.iter().cloned().fold(f64::MAX, f64::min);
assert!(spread >= 3.0, "fixture scores are too uniform to be useful");
// Every item converts into the shared curation type.
let converted: Vec<LlmScore> = items.into_iter().map(LlmScore::from).collect();
assert!(converted.iter().all(|s| (0.0..=10.0).contains(&s.score)));
}
/// The messy fixture must stay messy: it is what proves the parser is lenient
/// (string ids, string scores, out-of-range scores, junk entries).
/// (string ids and scores, out-of-range scores, unknown facet tokens, junk).
#[test]
fn stage_a_messy_fixture_is_salvaged_not_rejected() {
let raw = fixture("deepseek_score_batch_messy.json");
// The hard cases are still present in the recording…
fn deep_messy_fixture_is_salvaged_not_rejected() {
let sections = daily_epub::config::CurationConfig::default().sections;
let raw = fixture("deepseek_deep_batch_messy.json");
assert!(raw.contains("\"id\": \""), "needs a string id");
assert!(raw.contains("\"score\": \""), "needs a string score");
assert!(raw.contains("\"quality\": \""), "needs a string score");
assert!(
raw.contains("discussion_thread"),
"needs an unknown facet token"
);
// …and the real parser copes with all of them.
let items = parse_score_response(&raw);
let items = parse_deep_response(&raw, &sections);
assert!(!items.is_empty(), "the parser salvaged nothing");
assert!(
items.iter().all(|i| (0.0..=10.0).contains(&i.score)),
"out-of-range scores must be clamped: {:?}",
items.iter().map(|i| i.score).collect::<Vec<_>>()
items
.iter()
.all(|i| (0.0..=10.0).contains(&i.quality) && (0.0..=10.0).contains(&i.fit)),
"out-of-range scores must be clamped"
);
assert!(items.iter().all(|i| i.id > 0), "id-less items are skipped");
let messy = items
.iter()
.find(|i| i.id == 202)
.expect("string-valued item");
assert!(
messy.facets.format.is_none(),
"unknown facet tokens become None"
);
assert_eq!(messy.facets.depth.as_deref(), Some("standard"));
let off_palette = items.iter().find(|i| i.id == 204).expect("clamped item");
assert!(
off_palette.category.is_none(),
"invented sections become None"
);
// A response that is not JSON at all degrades to "no scores", never a panic.
assert!(parse_score_response("I'm sorry, I can't do that.").is_empty());
assert!(parse_score_response("").is_empty());
// A response that is not JSON at all degrades to "no assessments", never a panic.
assert!(parse_deep_response("I'm sorry, I can't do that.", &sections).is_empty());
assert!(parse_deep_response("", &sections).is_empty());
}
/// Stage B responses must carry `{id, section, position, lead_story}` with
/// exactly one lead, and use only palette section names (§3.6).
/// Editor responses must carry `{id, section, position, lead_story}` with
/// exactly one lead, and use only palette section names (plan §13).
#[test]
fn stage_b_fixture_parses_into_a_lineup() {
fn editor_fixture_parses_into_a_lineup() {
// The palette from `CurationConfig::default()` (§3.14).
let sections = daily_epub::config::CurationConfig::default().sections;
@@ -130,35 +146,41 @@ fn stage_b_fixture_parses_into_a_lineup() {
);
}
/// Stage C's front-page response must deserialize into a 250–400 word editor's
/// note plus per-section intros (§3.6).
/// The Brief must deserialize into 120–200 words of plain prose that names at
/// least three picks by title (§14.2). Section intros are gone.
#[test]
fn stage_c_fixture_parses_into_a_front_page() {
let response: FrontPageResponse = serde_json::from_str(&fixture("deepseek_front_page.json"))
.expect("the front-page fixture must match FrontPageResponse");
fn stage_c_fixture_parses_into_the_brief() {
let response: BriefResponse = serde_json::from_str(&fixture("claude_brief.json"))
.expect("the brief fixture must match BriefResponse");
let words = response.from_the_editor.split_whitespace().count();
let words = response.brief.split_whitespace().count();
assert!(
(150..=450).contains(&words),
"From the Editor is {words} words; the prompt asks for 250-400"
(100..=220).contains(&words),
"The Brief is {words} words; the prompt asks for 120-200"
);
assert!(
response.from_the_editor.contains("\n\n"),
"the prompt asks for 2-4 blank-line separated paragraphs"
!response.brief.contains("- ") && !response.brief.contains('#'),
"no bullets or headings in the brief"
);
let titles = response.brief.matches('"').count() / 2;
assert!(
!response.from_the_editor.contains("- "),
"no bullet lists on the front page"
titles >= 3,
"the brief names at least three picks; found {titles}"
);
assert!(response.section_intros.len() >= 2);
for (section, intro) in &response.section_intros {
let words = intro.split_whitespace().count();
for banned in [
"delve",
"dive",
"explore",
"a mix of",
"something for everyone",
] {
assert!(
(10..=90).contains(&words),
"intro for {section} is {words} words; the prompt asks for 35-60"
!response.brief.to_lowercase().contains(banned),
"banned phrase {banned}"
);
}
let value: serde_json::Value = serde_json::from_str(&fixture("claude_brief.json")).unwrap();
assert!(value.get("section_intros").is_none());
}
/// The taste profile is seeded from this file; a broken export would silently
+75 -1
View File
@@ -89,6 +89,7 @@ fn standard_edition_is_a_well_formed_epub3_archive() {
"OEBPS/sec-niche-corner.xhtml",
"OEBPS/art-1002.xhtml",
"OEBPS/world.xhtml",
"OEBPS/behind.xhtml",
"OEBPS/colophon.xhtml",
] {
assert!(contains_entry(&zip, entry), "missing {entry}");
@@ -186,11 +187,77 @@ fn chapter_ids_hrefs_and_toc_levels_are_stable() {
),
("art-1002".into(), "art-1002.xhtml".into(), 2),
("world".into(), "world.xhtml".into(), 1),
("behind".into(), "behind.xhtml".into(), 1),
("colophon".into(), "colophon.xhtml".into(), 1),
]
);
}
/// "Behind the paper" sits between the World Briefing and the colophon in both
/// editions, carries the counts, the admission mix, the learned-signal state,
/// the near misses and the models, and has no links (§15.1).
#[test]
fn behind_the_paper_renders_counts_and_near_misses_in_both_editions() {
let issue = fixtures::issue();
for edition in [Edition::Standard, Edition::X4] {
let (_dir, _, zip) = build_edition_to_bytes(&issue, edition);
let behind = read_entry(&zip, "OEBPS/behind.xhtml");
assert!(behind.contains("<h1>Behind the paper</h1>"), "{behind}");
assert!(
behind.contains(
"Considered 412 articles from 1,465 feeds \u{00b7} 398 eligible \u{00b7} 398 triaged \u{00b7} 120 read closely \u{00b7} 60 shortlisted \u{00b7} 2 selected."
),
"{behind}"
);
assert!(
behind.contains(
"Admitted via: triage 60 \u{00b7} interests 20 \u{00b7} your ratings 12 \u{00b7} exploration 5 \u{00b7} blend 23."
),
"{behind}"
);
assert!(
behind.contains(
"Learned signals: 14 rated articles with embeddings (neighbour signal at 35%); feed affinity off."
),
"{behind}"
);
assert!(behind.contains("<h2>Near misses</h2>"), "{behind}");
assert!(
behind.contains(
"<li>The One That Got Away \u{2014} Example Feed \u{00b7} quality 8.0 \u{00b7} fit 6.5 \u{00b7} shortlisted, not selected</li>"
),
"{behind}"
);
assert!(
behind.contains(
"<li>Never Read Closely \u{2014} Other Feed \u{00b7} triaged, not admitted</li>"
),
"{behind}"
);
assert!(
behind.contains(
"Models: triage and assessment deepseek-v4-flash \u{00b7} editor and summaries claude-opus-5 \u{00b7} embeddings voyage-4-lite. Cost $0.81. Generation 23 min."
),
"{behind}"
);
assert!(!behind.contains("<a "), "no links in {edition:?}: {behind}");
assert!(!behind.contains("&nbsp;"));
for tag in ["html", "head", "body", "div", "p", "ul", "li", "h1", "h2"] {
// `<li` would also match `<link`; count real opening tags only.
let opens = behind.matches(&format!("<{tag}>")).count()
+ behind.matches(&format!("<{tag} ")).count();
let closes = behind.matches(&format!("</{tag}>")).count();
assert_eq!(opens, closes, "unbalanced <{tag}> in {edition:?}");
}
// Ordering: world → behind → colophon in the spine.
let opf = read_entry(&zip, "OEBPS/content.opf");
let position = |href: &str| opf.find(&format!("href=\"{href}\"")).expect(href);
assert!(position("world.xhtml") < position("behind.xhtml"));
assert!(position("behind.xhtml") < position("colophon.xhtml"));
}
}
#[test]
fn every_chapter_is_parseable_xhtml() {
let issue = fixtures::issue();
@@ -320,7 +387,14 @@ fn colophon_facts_are_x4_safe_distinct_paragraphs() {
for edition in [Edition::Standard, Edition::X4] {
let (_dir, _, zip) = build_edition_to_bytes(&issue, edition);
let colophon = read_entry(&zip, "OEBPS/colophon.xhtml");
assert_eq!(colophon.matches("<p class=\"fact-line\">").count(), 9);
// Issue, generated, three model lines, entries, candidates, articles,
// words, two per-provider cost lines, the total, generator (§15.1).
assert_eq!(colophon.matches("<p class=\"fact-line\">").count(), 13);
assert!(colophon.contains("<strong>Editor model:</strong> claude-opus-5"));
assert!(colophon.contains("<strong>Bulk model:</strong> deepseek-v4-flash"));
assert!(colophon.contains("<strong>anthropic cost:</strong> $0.0500"));
assert!(colophon.contains("<strong>deepseek cost:</strong> $0.0231"));
assert!(colophon.contains("<strong>Total token cost:</strong>"));
assert!(!colophon.contains("<dl"));
assert!(!colophon.contains("<dt"));
assert!(!colophon.contains("<dd"));