Curation v2 step 5: deep assessment, utility, diversified shortlist
assess.rs replaces score.rs (DEEP_INSTRUCTIONS, representative sample with [BEGINNING]/[MIDDLE]/[END], facets, cached deep rows with --rescore bypass), rank.rs adds the utility blend over present signals with gate ramps and the leader-clustered shortlist (cap 2 → 3 → uncapped, protected top-N, exploration reserve), editor.rs replaces select.rs with the §13 rendering and utility-ordered fallbacks. ScoredArticle is gone; Candidate is the only flow type. deep_batch_size replaces score_batch_size. Started by Codex (cut off by its usage limit mid-verification) and finished by a Claude agent from docs/plans/curation-v2-briefs/step5.md; reviewed against plan §12–§13. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
This commit is contained in:
Claude Fable 5.1
@@ -4,9 +4,10 @@ A personalized daily newspaper, delivered as an EPUB.
|
||||
|
||||
Every morning a systemd timer wakes one Rust binary. It pulls the last ~26 hours
|
||||
from a self-hosted [Miniflux](https://miniflux.app), deduplicates and extracts
|
||||
the articles, enriches them with HackerNews/Lobsters/Reddit social proof, filters
|
||||
300–500 candidates down to ~120 with cheap heuristics, asks DeepSeek to score them,
|
||||
and hands the shortlist to Claude Opus 5 — the editor — which assembles the issue
|
||||
the articles, enriches them with HackerNews/Lobsters/Reddit social proof, has
|
||||
DeepSeek triage every eligible opening and closely assess a 120-article union,
|
||||
then utility-ranks and diversity-caps a 60-item shortlist for Claude Opus 5 — the editor —
|
||||
which assembles the issue
|
||||
(no minimum size, a hard ceiling), writes a one-line *why* under every headline,
|
||||
the summaries and *The Brief*. It assembles two EPUB editions (a standard one
|
||||
and one tuned for the Xteink X4 e-ink reader), converts the X4 edition to XTC, and
|
||||
@@ -29,7 +30,8 @@ hard spend limits in the providers' dashboards as the real backstop.
|
||||
|
||||
```
|
||||
Miniflux ingest ─▶ dedupe ─▶ extraction ─▶ persist ─▶ social enrichment
|
||||
─▶ hygiene ─▶ embeddings (Voyage) + cheap signals ─▶ pre-filter ─▶ scoring (DeepSeek)
|
||||
─▶ hygiene ─▶ embeddings (Voyage) + cheap signals ─▶ triage (DeepSeek)
|
||||
─▶ union admission ─▶ deep assessment (DeepSeek) ─▶ utility + diversity
|
||||
─▶ editor (Claude) ─▶ comments ─▶ editorial (Claude)
|
||||
─▶ world briefing ─▶ EPUB (standard + X4) ─▶ XTC ─▶ publish ─▶ report
|
||||
```
|
||||
@@ -56,7 +58,7 @@ fallback (`fallbacks = "default"`) is enabled on every editor request.
|
||||
|---|---|---|
|
||||
| Rust (2024 edition toolchain) | building | `cargo build --release` |
|
||||
| **Miniflux** with an API key | the only content source | Settings → API Keys. The client is read-only and never mutates read state. |
|
||||
| **DeepSeek API key** | scoring, and the fallback for every editor call | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. |
|
||||
| **DeepSeek API key** | triage and deep assessment, and the fallback for every editor call | <https://platform.deepseek.com>. Optional: `--skip-llm` runs the whole pipeline without it. |
|
||||
| **Anthropic API key** | the editor: selection, summaries, The Brief, the weekly profile rebuild | <https://console.anthropic.com>. Optional: without it every editor call runs on DeepSeek. Set a dashboard spend limit; `anthropic.max_daily_usd` is only a runaway guard. |
|
||||
| **Voyage AI API key** | article and interest embeddings behind the learned ranking signals | <https://www.voyageai.com>. Optional: without it (or with `--skip-embeddings`) the run uses cached vectors only and the learned signals are absent, never a penalty. |
|
||||
| A 32+ byte random secret | signs the article rating links | `openssl rand -hex 32` |
|
||||
@@ -106,11 +108,13 @@ does not advance the ingest watermark. It prints the lineup and the cost report.
|
||||
`explain` answers "why was this (not) in the paper" from the `candidate_runs`
|
||||
row the run persisted for every considered article: the stage it reached and the
|
||||
reason it stopped, every raw and normalized signal with its presence and
|
||||
effective weight, the top interests, the nearest rated neighbours, any cached
|
||||
LLM assessments, and the editor's reason for a pick. `--url` canonicalizes the
|
||||
address; an article that is not in the database at all is reported as never
|
||||
ingested (a feed problem, not a ranking one). `--near-misses` lists the highest
|
||||
ranked articles that were not selected.
|
||||
effective weight, the top interests, the nearest rated neighbours, the triage
|
||||
and deep assessments (quality, fit, category, rationale, facets), utility and
|
||||
rank, the cluster it landed in and what suppressed it, and the editor's reason
|
||||
for a pick. `--url` canonicalizes the address; an article that is not in the
|
||||
database at all is reported as never ingested (a feed problem, not a ranking
|
||||
one). `--near-misses` lists the highest-utility articles that were not selected
|
||||
(by preliminary blend for articles the ranker never reached).
|
||||
|
||||
`features backfill` embeds the rated and published articles first (the learned
|
||||
set), then the standing interests, then — only with `--all` — every other
|
||||
@@ -159,9 +163,9 @@ Secrets belong in the environment file, never in the TOML.
|
||||
| `deepseek.base_url` | `https://api.deepseek.com/v1` | OpenAI-compatible endpoint. |
|
||||
| `deepseek.model` | `deepseek-v4-flash` | Verified 2026-08-15 (DeepSeek-V4-Flash-0731). |
|
||||
| `deepseek.api_key` | — | **`DAILY_EPUB_DEEPSEEK__API_KEY`**. Absent ⇒ the run curates heuristically. |
|
||||
| `deepseek.score_batch_size` | `12` | Articles per stage-A scoring request. |
|
||||
| `deepseek.deep_batch_size` | `8` | Articles per close-reading assessment request. The removed `score_batch_size` key is a startup error. |
|
||||
| `deepseek.triage_batch_size` | `25` | Articles per first-pass triage request. |
|
||||
| `deepseek.max_concurrent_requests` | `4` | Triage and stage-A batches in flight at once; the budget is checked before each is spawned. |
|
||||
| `deepseek.max_concurrent_requests` | `4` | Triage and deep-assessment batches in flight at once; the budget is checked before each is spawned. |
|
||||
| `deepseek.score_temperature` | `0.3` | Scoring temperature. |
|
||||
| `deepseek.editorial_temperature` | `0.8` | Summaries and The Brief, only when DeepSeek is the fallback editor. |
|
||||
| `deepseek.price_input_per_mtok` | `0.14` | USD per 1M cache-miss input tokens (cost guardrail arithmetic). |
|
||||
@@ -223,9 +227,11 @@ signal is absent. Ratings decay with `rating_half_life_days` (60) over
|
||||
`deep_keep` (120), `shortlist_keep` (60), `assessment_reuse_days` (3),
|
||||
`semantic_min_words` (300), `exploration_slots` (5), `[curation.ranking.quotas]`
|
||||
(`triage` 60 · `interest` 20 · `knn` 20), `[curation.ranking.weights.utility]`
|
||||
and `[curation.ranking.diversity]` (`cluster_threshold` 0.85, `per_cluster_cap`
|
||||
2, `utility_protected` 10) are validated now and drive the LLM triage, deep
|
||||
assessment and diversification stages as they land.
|
||||
(`quality` 0.40 · `fit` 0.20 · `knn` 0.15 · `interest` 0.10 · `feed` 0.05 ·
|
||||
`triage` 0.05 · `social` 0.03 · `heuristic` 0.02, over the signals present for
|
||||
each article of the deep set) and `[curation.ranking.diversity]`
|
||||
(`cluster_threshold` 0.85, `per_cluster_cap` 2, `utility_protected` 10) drive
|
||||
the LLM triage, deep assessment, utility ranking and diversification stages.
|
||||
`[curation.ranking.weights.preliminary]` (`interest` 0.35 · `knn` 0.25 ·
|
||||
`heuristic` 0.20 · `feed` 0.10 · `social` 0.10) blends the cheap signals; weights
|
||||
are renormalized over the signals present for each article, so they need not sum
|
||||
@@ -499,9 +505,9 @@ and database rows, with no network access anywhere.
|
||||
server. The stages themselves:
|
||||
|
||||
```text
|
||||
miniflux.rs ingest curate/ scoring and selection
|
||||
dedupe.rs clustering prefilter, llm, triage, admit, score, select,
|
||||
editorial, embedding, signals, telemetry
|
||||
miniflux.rs ingest curate/ triage, assessment and selection
|
||||
dedupe.rs clustering prefilter, llm, triage, admit, assess, rank,
|
||||
editor, editorial, embedding, signals, telemetry
|
||||
extract.rs body text profile/ the reader's taste profile
|
||||
images/ article images comments.rs discussion chapters
|
||||
normalize usable <img> world.rs the world briefing
|
||||
@@ -582,5 +588,9 @@ From spec §7, plus what implementation turned up:
|
||||
the union of triage, interest, neighbour, exploration, blend and auto-include
|
||||
retrievers. `explain` shows the assessment and `admitted_by`. Learned signals
|
||||
stay absent until their gates open (8 and 15 ratings respectively).
|
||||
- **Deep assessment and diversity are live.** DeepSeek reads a representative
|
||||
beginning/middle/end sample, separates editorial quality from reader fit, and
|
||||
records descriptive facets. Utility is normalized over the deep set; embedding
|
||||
leader clusters cap near-duplicates before the 60-item editor shortlist.
|
||||
- **One reader, one issue per day.** There is no multi-user support and no
|
||||
weekly/retrospective edition (spec §6).
|
||||
|
||||
+2
-2
@@ -36,9 +36,9 @@ page_limit = 250
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
model = "deepseek-v4-flash" # DeepSeek-V4-Flash-0731 (confirmed 2026-08-15)
|
||||
# api_key via DAILY_EPUB_DEEPSEEK__API_KEY env
|
||||
score_batch_size = 12
|
||||
deep_batch_size = 8 # articles per close-reading assessment request
|
||||
triage_batch_size = 25 # articles per first-pass triage request
|
||||
max_concurrent_requests = 4 # triage and stage-A batches in flight at once
|
||||
max_concurrent_requests = 4 # triage and deep-assessment batches in flight
|
||||
score_temperature = 0.3
|
||||
editorial_temperature = 0.8 # used only when DeepSeek is the fallback editor
|
||||
# USD per 1M tokens, used for the cost guardrail.
|
||||
|
||||
+33
-6
@@ -137,8 +137,8 @@ pub struct DeepseekConfig {
|
||||
pub model: String,
|
||||
/// Supply via `DAILY_EPUB_DEEPSEEK__API_KEY`.
|
||||
pub api_key: Option<String>,
|
||||
/// Articles per stage-A scoring request (§3.6).
|
||||
pub score_batch_size: usize,
|
||||
/// Articles per deep-assessment request (§12.1).
|
||||
pub deep_batch_size: usize,
|
||||
/// Articles per first-pass triage request (§10).
|
||||
pub triage_batch_size: usize,
|
||||
pub max_concurrent_requests: usize,
|
||||
@@ -158,7 +158,7 @@ impl Default for DeepseekConfig {
|
||||
base_url: "https://api.deepseek.com/v1".into(),
|
||||
model: "deepseek-v4-flash".into(),
|
||||
api_key: None,
|
||||
score_batch_size: 12,
|
||||
deep_batch_size: 8,
|
||||
triage_batch_size: 25,
|
||||
max_concurrent_requests: 4,
|
||||
score_temperature: 0.3,
|
||||
@@ -602,6 +602,12 @@ impl Config {
|
||||
/// Load config for the CLI: explicit `--config` path, else `./config.toml`
|
||||
/// when it exists, then `DAILY_EPUB_*` env overrides (§3.14).
|
||||
pub fn load(explicit: Option<&Path>) -> Result<Self, ConfigError> {
|
||||
if std::env::var_os("DAILY_EPUB_DEEPSEEK__SCORE_BATCH_SIZE").is_some() {
|
||||
return Err(ConfigError::Invalid(
|
||||
"DAILY_EPUB_DEEPSEEK__SCORE_BATCH_SIZE was removed; use DAILY_EPUB_DEEPSEEK__DEEP_BATCH_SIZE"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
let (path, require) = match explicit {
|
||||
Some(p) => (Some(p.to_path_buf()), true),
|
||||
None => (Some(PathBuf::from(DEFAULT_CONFIG_FILE)), false),
|
||||
@@ -621,6 +627,17 @@ impl Config {
|
||||
"prefilter_keep was removed; use curation.ranking.deep_keep".into(),
|
||||
));
|
||||
}
|
||||
if raw.lines().any(|line| {
|
||||
let line = line.trim_start();
|
||||
!line.starts_with('#')
|
||||
&& line
|
||||
.strip_prefix("score_batch_size")
|
||||
.is_some_and(|tail| tail.trim_start().starts_with('='))
|
||||
}) {
|
||||
return Err(ConfigError::Invalid(
|
||||
"deepseek.score_batch_size was removed; use deepseek.deep_batch_size".into(),
|
||||
));
|
||||
}
|
||||
}
|
||||
let mut config: Config = Self::figment(path.as_deref(), require)?.extract()?;
|
||||
// §1 tells the operator to set `DAILY_EPUB_SECRET`; §3.14 calls the key
|
||||
@@ -649,9 +666,9 @@ impl Config {
|
||||
"curation.max_article_count must be >= target_article_count".into(),
|
||||
));
|
||||
}
|
||||
if self.deepseek.score_batch_size == 0 {
|
||||
if self.deepseek.deep_batch_size == 0 {
|
||||
return Err(ConfigError::Invalid(
|
||||
"deepseek.score_batch_size must be >= 1".into(),
|
||||
"deepseek.deep_batch_size must be >= 1".into(),
|
||||
));
|
||||
}
|
||||
if self.deepseek.triage_batch_size == 0 {
|
||||
@@ -779,6 +796,7 @@ mod tests {
|
||||
assert!(c.world_briefing);
|
||||
assert_eq!(c.deepseek.model, "deepseek-v4-flash");
|
||||
assert_eq!(c.deepseek.triage_batch_size, 25);
|
||||
assert_eq!(c.deepseek.deep_batch_size, 8);
|
||||
assert_eq!(c.curation.recent_rejection_days, 7);
|
||||
assert_eq!(c.curation.recent_rejection_floor, 3.0);
|
||||
assert_eq!(c.profile_path, PathBuf::from("data/profile.md"));
|
||||
@@ -878,6 +896,15 @@ mod tests {
|
||||
assert!(message.contains("curation.ranking.deep_keep"), "{message}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn removed_score_batch_size_names_deep_batch_size() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("config.toml");
|
||||
std::fs::write(&path, "[deepseek]\nscore_batch_size = 12\n").unwrap();
|
||||
let error = Config::load(Some(&path)).expect_err("stale key must fail");
|
||||
assert!(error.to_string().contains("deep_batch_size"), "{error}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shipped_example_config_parses() {
|
||||
let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("config.example.toml");
|
||||
@@ -918,7 +945,7 @@ mod tests {
|
||||
c.anthropic.max_concurrent_requests = 0;
|
||||
assert!(c.validate().is_err());
|
||||
let mut c = Config::default();
|
||||
c.deepseek.score_batch_size = 0;
|
||||
c.deepseek.deep_batch_size = 0;
|
||||
assert!(c.validate().is_err());
|
||||
let mut c = Config::default();
|
||||
c.deepseek.triage_batch_size = 0;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+13
-6
@@ -195,22 +195,28 @@ pub fn build_brief_prompt(lineup: &Lineup, summaries: &BTreeMap<ArticleId, Strin
|
||||
for section in &lineup.section_order {
|
||||
let _ = writeln!(prompt, "\n## {section}");
|
||||
for pick in lineup.section_picks(section) {
|
||||
let score = pick
|
||||
let quality = pick
|
||||
.llm
|
||||
.as_ref()
|
||||
.map(|score| format!("{:.1}", score.score))
|
||||
.unwrap_or_else(|| "unscored".into());
|
||||
.map(|assessment| format!("{:.1}", assessment.quality))
|
||||
.unwrap_or_else(|| "unassessed".into());
|
||||
let fit = pick
|
||||
.llm
|
||||
.as_ref()
|
||||
.map(|assessment| format!("{:.1}", assessment.fit))
|
||||
.unwrap_or_else(|| "unassessed".into());
|
||||
let summary = summaries
|
||||
.get(&pick.article.id)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| excerpt_summary(pick));
|
||||
let _ = writeln!(
|
||||
prompt,
|
||||
"- {}\n feed: {}\n why: {}\n score: {}\n summary: {}",
|
||||
"- {}\n feed: {}\n why: {}\n quality: {}\n fit: {}\n summary: {}",
|
||||
pick.article.title.trim(),
|
||||
pick.article.feed_title.trim(),
|
||||
pick.why.as_deref().unwrap_or("not supplied"),
|
||||
score,
|
||||
quality,
|
||||
fit,
|
||||
summary
|
||||
);
|
||||
}
|
||||
@@ -462,7 +468,8 @@ mod tests {
|
||||
assert!(prompt.contains("- Migrating 40TB off Postgres"));
|
||||
assert!(prompt.contains("why: the Migrating 40TB off Postgres piece you'd argue with"));
|
||||
assert!(prompt.contains("summary: A migration story with numbers."));
|
||||
assert!(prompt.contains("score: unscored"));
|
||||
assert!(prompt.contains("quality: unassessed"));
|
||||
assert!(prompt.contains("fit: unassessed"));
|
||||
assert!(prompt.contains("2026-08-15"));
|
||||
assert!(
|
||||
!prompt.contains("section_intros"),
|
||||
|
||||
+39
-30
@@ -1,7 +1,7 @@
|
||||
//! Personalized curation: signals → triage → admission → assessment → editor.
|
||||
//!
|
||||
//! ```text
|
||||
//! ~400 eligible ─triage─▶ union admission (120) ─stage A─▶ editor ─▶ editorial
|
||||
//! ~400 eligible ─triage─▶ admission (120) ─assess/rank─▶ shortlist (60) ─editor─▶ editorial
|
||||
//! ```
|
||||
//!
|
||||
//! [`Curator`] is the thin orchestration layer the `generate` pipeline calls; the
|
||||
@@ -11,24 +11,25 @@
|
||||
//! client; selection and editorial on the editor with per-call bulk fallback.
|
||||
|
||||
pub mod admit;
|
||||
pub mod assess;
|
||||
pub mod editor;
|
||||
pub mod editorial;
|
||||
pub mod embedding;
|
||||
pub mod llm;
|
||||
pub mod prefilter;
|
||||
pub mod profile;
|
||||
pub mod score;
|
||||
pub mod select;
|
||||
pub mod rank;
|
||||
pub mod signals;
|
||||
pub mod telemetry;
|
||||
pub mod triage;
|
||||
|
||||
use jiff::civil::Date;
|
||||
use jiff::{Timestamp, civil::Date};
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::db::Db;
|
||||
use crate::types::{Editorial, Lineup, ScoredArticle};
|
||||
use crate::types::{Candidate, Editorial, Lineup};
|
||||
|
||||
/// Runs the three curation stages against one day's articles (§3.5, §3.6).
|
||||
/// Runs the LLM curation stages against one day's articles (§12, §13, §14).
|
||||
pub struct Curator {
|
||||
pub config: Config,
|
||||
pub db: Db,
|
||||
@@ -43,43 +44,51 @@ impl Curator {
|
||||
Self { config, db, llms }
|
||||
}
|
||||
|
||||
/// Stage A: batched LLM scoring of the surviving candidates (§3.6).
|
||||
/// Deep assessment of the admitted set on the bulk client (§12.1), reusing
|
||||
/// cached `article_assessments` rows within `assessment_reuse_days`.
|
||||
///
|
||||
/// A no-op under `--skip-llm`. Scores are persisted per `(article, date)`.
|
||||
pub async fn score(&self, candidates: &mut [ScoredArticle], _date: Date) -> anyhow::Result<()> {
|
||||
let Some(llm) = self.llms.bulk.as_ref() else {
|
||||
tracing::info!("--skip-llm: stage A scoring skipped");
|
||||
return Ok(());
|
||||
/// A no-op under `--skip-llm`: like triage, nothing is read or written and
|
||||
/// utility falls back to the present signals (§12.3, §17). When the bulk
|
||||
/// provider is down or its budget trips, cached rows are still reused and
|
||||
/// the failed batches simply stay unassessed.
|
||||
pub async fn assess(
|
||||
&self,
|
||||
candidates: &mut [Candidate],
|
||||
rescore: bool,
|
||||
profile_version: Option<i64>,
|
||||
assessed_at: Timestamp,
|
||||
) -> anyhow::Result<usize> {
|
||||
let Some(bulk) = self.llms.bulk.as_ref() else {
|
||||
tracing::info!("--skip-llm: deep assessment skipped");
|
||||
return Ok(0);
|
||||
};
|
||||
let span = tracing::info_span!("llm_score", candidates = candidates.len());
|
||||
let span = tracing::info_span!("llm_assess", candidates = candidates.len());
|
||||
let _guard = span.enter();
|
||||
|
||||
let scored = score::score_all(
|
||||
llm,
|
||||
assess::run(
|
||||
&self.db,
|
||||
Some(bulk),
|
||||
&self.config.deepseek.model,
|
||||
candidates,
|
||||
self.config.deepseek.score_batch_size,
|
||||
self.config.deepseek.deep_batch_size,
|
||||
self.config.deepseek.max_concurrent_requests,
|
||||
&self.config.curation.sections,
|
||||
self.config.curation.ranking.assessment_reuse_days,
|
||||
rescore,
|
||||
profile_version,
|
||||
assessed_at,
|
||||
self.config.deepseek.score_temperature,
|
||||
&self.config.curation.sections,
|
||||
)
|
||||
.await?;
|
||||
tracing::info!(scored, total = candidates.len(), "stage A complete");
|
||||
|
||||
Ok(())
|
||||
.await
|
||||
}
|
||||
|
||||
/// Stage B: single-call lineup selection into sections (§3.6).
|
||||
pub async fn select(
|
||||
&self,
|
||||
candidates: Vec<ScoredArticle>,
|
||||
date: Date,
|
||||
) -> anyhow::Result<Lineup> {
|
||||
/// The editor: one call that assembles the issue from the shortlist (§13).
|
||||
pub async fn select(&self, candidates: Vec<Candidate>, date: Date) -> anyhow::Result<Lineup> {
|
||||
let sections = &self.config.curation.sections;
|
||||
let soft_target = self.config.target_article_count;
|
||||
let hard_max = self.config.curation.max_article_count;
|
||||
let span = tracing::info_span!("llm_editor", candidates = candidates.len());
|
||||
let _guard = span.enter();
|
||||
match select::select(
|
||||
match editor::select(
|
||||
&self.llms,
|
||||
candidates.clone(),
|
||||
sections,
|
||||
@@ -92,7 +101,7 @@ impl Curator {
|
||||
Ok(lineup) => Ok(lineup),
|
||||
Err(error) => {
|
||||
tracing::error!(%error, "editor and bulk fallback failed; selecting heuristically");
|
||||
Ok(select::select_without_llm(
|
||||
Ok(editor::select_without_llm(
|
||||
candidates,
|
||||
sections,
|
||||
soft_target,
|
||||
|
||||
+1
-14
@@ -125,9 +125,7 @@ pub fn roundup_penalty(title: &str) -> f64 {
|
||||
#[cfg(test)]
|
||||
pub(crate) mod tests {
|
||||
use super::*;
|
||||
use crate::types::{
|
||||
ArticleId, ExtractMethod, FeedId, SocialRef, SocialSource, SourceKind, SourceRef,
|
||||
};
|
||||
use crate::types::{ArticleId, ExtractMethod, SocialRef, SocialSource, SourceKind, SourceRef};
|
||||
use jiff::Timestamp;
|
||||
|
||||
pub(crate) fn ts() -> Timestamp {
|
||||
@@ -180,17 +178,6 @@ pub(crate) mod tests {
|
||||
article
|
||||
}
|
||||
|
||||
pub(crate) fn via(mut article: Article, kind: SourceKind, feed_id: FeedId) -> Article {
|
||||
article.sources.push(SourceRef {
|
||||
entry_id: article.best_entry_id,
|
||||
feed_id,
|
||||
feed_title: format!("{kind:?} feed"),
|
||||
category: None,
|
||||
kind,
|
||||
});
|
||||
article
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn text_heuristic_has_only_text_terms() {
|
||||
let quiet = article(1, "An essay", 1200);
|
||||
|
||||
@@ -0,0 +1,586 @@
|
||||
//! Utility normalization, leader clustering and diversified shortlisting (§12.2–§12.5).
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use crate::config::{RankingConfig, UtilityWeights};
|
||||
use crate::curate::embedding::dot;
|
||||
use crate::curate::signals;
|
||||
use crate::types::{ArticleId, Candidate};
|
||||
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||
pub struct RankSummary {
|
||||
pub shortlisted: usize,
|
||||
pub clusters: usize,
|
||||
}
|
||||
|
||||
/// Re-normalize cheap signals over the deep set and calculate utility on 0–100.
|
||||
pub fn calculate_utility(candidates: &mut [Candidate], configured: &UtilityWeights) {
|
||||
let indices = (0..candidates.len()).collect::<Vec<_>>();
|
||||
calculate_utility_for(candidates, &indices, configured);
|
||||
}
|
||||
|
||||
fn calculate_utility_for(
|
||||
candidates: &mut [Candidate],
|
||||
indices: &[usize],
|
||||
configured: &UtilityWeights,
|
||||
) {
|
||||
let mut normalized = indices
|
||||
.iter()
|
||||
.map(|index| candidates[*index].signals.clone())
|
||||
.collect::<Vec<_>>();
|
||||
for signals in &mut normalized {
|
||||
signals.norm.clear();
|
||||
signals.weights.clear();
|
||||
}
|
||||
let mut signal_refs = normalized.iter_mut().collect::<Vec<_>>();
|
||||
signals::normalize(&mut signal_refs);
|
||||
for (index, signals) in indices.iter().zip(normalized) {
|
||||
candidates[*index].signals.norm = signals.norm;
|
||||
candidates[*index].signals.weights.clear();
|
||||
}
|
||||
|
||||
for index in indices {
|
||||
let candidate = &mut candidates[*index];
|
||||
if let Some(triage) = &candidate.assessment.triage {
|
||||
candidate
|
||||
.signals
|
||||
.norm
|
||||
.insert("triage".into(), (triage.interest / 10.0).clamp(0.0, 1.0));
|
||||
}
|
||||
if let Some(deep) = &candidate.assessment.deep {
|
||||
candidate
|
||||
.signals
|
||||
.norm
|
||||
.insert("quality".into(), (deep.quality / 10.0).clamp(0.0, 1.0));
|
||||
candidate
|
||||
.signals
|
||||
.norm
|
||||
.insert("fit".into(), (deep.fit / 10.0).clamp(0.0, 1.0));
|
||||
}
|
||||
let weighted = [
|
||||
("quality", configured.quality, 1.0),
|
||||
("fit", configured.fit, 1.0),
|
||||
("knn", configured.knn, candidate.signals.knn_gate),
|
||||
("interest", configured.interest, 1.0),
|
||||
("feed", configured.feed, candidate.signals.feed_gate),
|
||||
("triage", configured.triage, 1.0),
|
||||
("social", configured.social, 1.0),
|
||||
("heuristic", configured.heuristic, 1.0),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(name, weight, gate)| {
|
||||
let value = candidate.signals.norm.get(name).copied()?;
|
||||
let effective = weight * gate;
|
||||
(effective > 0.0).then_some((name, effective, value))
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let total = weighted.iter().map(|(_, weight, _)| weight).sum::<f64>();
|
||||
if total <= 0.0 {
|
||||
candidate.utility = None;
|
||||
continue;
|
||||
}
|
||||
candidate.signals.weights = weighted
|
||||
.iter()
|
||||
.map(|(name, weight, _)| ((*name).to_string(), weight / total))
|
||||
.collect();
|
||||
candidate.utility = Some(
|
||||
weighted
|
||||
.iter()
|
||||
.map(|(_, weight, value)| weight / total * value)
|
||||
.sum::<f64>()
|
||||
* 100.0,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn ranked_indices(candidates: &[Candidate], indices: &[usize]) -> Vec<usize> {
|
||||
let mut sorted = indices.to_vec();
|
||||
sorted.sort_by(|left, right| {
|
||||
candidates[*right]
|
||||
.utility
|
||||
.unwrap_or(f64::NEG_INFINITY)
|
||||
.total_cmp(&candidates[*left].utility.unwrap_or(f64::NEG_INFINITY))
|
||||
.then_with(|| {
|
||||
candidates[*left]
|
||||
.article
|
||||
.id
|
||||
.cmp(&candidates[*right].article.id)
|
||||
})
|
||||
});
|
||||
sorted
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Cluster {
|
||||
id: i64,
|
||||
leader: usize,
|
||||
}
|
||||
|
||||
/// Rank the admitted deep set and leave only the diversified shortlist at
|
||||
/// `stage = shortlisted`. All deep-set articles receive ranks and clusters.
|
||||
pub fn shortlist(
|
||||
candidates: &mut [Candidate],
|
||||
embeddings: &HashMap<ArticleId, Vec<f32>>,
|
||||
ranking: &RankingConfig,
|
||||
) -> RankSummary {
|
||||
let deep = candidates
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, candidate)| matches!(candidate.stage.as_str(), "admitted" | "assessed"))
|
||||
.map(|(index, _)| index)
|
||||
.collect::<Vec<_>>();
|
||||
calculate_utility_for(candidates, &deep, &ranking.weights.utility);
|
||||
let sorted = ranked_indices(candidates, &deep);
|
||||
for (rank, index) in sorted.iter().enumerate() {
|
||||
candidates[*index].rank_utility = Some(rank as i64 + 1);
|
||||
candidates[*index].cluster = None;
|
||||
candidates[*index].cluster_rank = None;
|
||||
}
|
||||
|
||||
let mut clusters = Vec::<Cluster>::new();
|
||||
let mut members: HashMap<i64, Vec<usize>> = HashMap::new();
|
||||
for index in &sorted {
|
||||
let assigned = embeddings
|
||||
.get(&candidates[*index].article.id)
|
||||
.and_then(|vector| {
|
||||
clusters.iter().find_map(|cluster| {
|
||||
let leader_id = candidates[cluster.leader].article.id;
|
||||
let leader = embeddings.get(&leader_id)?;
|
||||
dot(vector, leader)
|
||||
.ok()
|
||||
.filter(|cosine| *cosine >= ranking.diversity.cluster_threshold)
|
||||
.map(|_| cluster.id)
|
||||
})
|
||||
});
|
||||
let cluster_id = assigned.unwrap_or_else(|| {
|
||||
let id = clusters.len() as i64 + 1;
|
||||
clusters.push(Cluster { id, leader: *index });
|
||||
id
|
||||
});
|
||||
let cluster_members = members.entry(cluster_id).or_default();
|
||||
cluster_members.push(*index);
|
||||
candidates[*index].cluster = Some(cluster_id);
|
||||
candidates[*index].cluster_rank = Some(cluster_members.len() as i64);
|
||||
}
|
||||
|
||||
let protected = sorted
|
||||
.iter()
|
||||
.take(ranking.diversity.utility_protected)
|
||||
.copied()
|
||||
.collect::<HashSet<_>>();
|
||||
let mut admitted = HashSet::new();
|
||||
let mut admitted_per_cluster = HashMap::<i64, usize>::new();
|
||||
let admit = |index: usize, admitted: &mut HashSet<usize>, counts: &mut HashMap<i64, usize>| {
|
||||
if admitted.insert(index)
|
||||
&& let Some(cluster) = candidates[index].cluster
|
||||
{
|
||||
*counts.entry(cluster).or_default() += 1;
|
||||
}
|
||||
};
|
||||
|
||||
for index in &sorted {
|
||||
if protected.contains(index) || candidates[*index].auto_include {
|
||||
admit(*index, &mut admitted, &mut admitted_per_cluster);
|
||||
}
|
||||
}
|
||||
for index in sorted
|
||||
.iter()
|
||||
.filter(|index| candidates[**index].exploration)
|
||||
.take(3)
|
||||
{
|
||||
if admitted.len() >= ranking.shortlist_keep {
|
||||
break;
|
||||
}
|
||||
admit(*index, &mut admitted, &mut admitted_per_cluster);
|
||||
}
|
||||
|
||||
let target = ranking.shortlist_keep.max(admitted.len());
|
||||
let mut suppressed_at_base_cap = HashSet::new();
|
||||
admit_under_cap(
|
||||
candidates,
|
||||
&sorted,
|
||||
target,
|
||||
ranking.diversity.per_cluster_cap,
|
||||
&mut admitted,
|
||||
&mut admitted_per_cluster,
|
||||
Some(&mut suppressed_at_base_cap),
|
||||
);
|
||||
if admitted.len() < target {
|
||||
admit_under_cap(
|
||||
candidates,
|
||||
&sorted,
|
||||
target,
|
||||
3,
|
||||
&mut admitted,
|
||||
&mut admitted_per_cluster,
|
||||
None,
|
||||
);
|
||||
}
|
||||
if admitted.len() < target {
|
||||
for index in &sorted {
|
||||
if admitted.len() >= target {
|
||||
break;
|
||||
}
|
||||
admit(*index, &mut admitted, &mut admitted_per_cluster);
|
||||
}
|
||||
}
|
||||
|
||||
for index in deep {
|
||||
if admitted.contains(&index) {
|
||||
candidates[index].stage = "shortlisted".into();
|
||||
candidates[index].excluded_reason = None;
|
||||
} else {
|
||||
// The stage stays where the article stopped (`admitted` when the
|
||||
// deep assessment never happened, else `assessed`).
|
||||
candidates[index].excluded_reason = Some(
|
||||
if suppressed_at_base_cap.contains(&index) {
|
||||
"cluster_suppressed"
|
||||
} else {
|
||||
"shortlist_cap"
|
||||
}
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
}
|
||||
RankSummary {
|
||||
shortlisted: admitted.len(),
|
||||
clusters: clusters.len(),
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn admit_under_cap(
|
||||
candidates: &[Candidate],
|
||||
sorted: &[usize],
|
||||
target: usize,
|
||||
cap: usize,
|
||||
admitted: &mut HashSet<usize>,
|
||||
admitted_per_cluster: &mut HashMap<i64, usize>,
|
||||
mut suppressed: Option<&mut HashSet<usize>>,
|
||||
) {
|
||||
for index in sorted {
|
||||
if admitted.len() >= target || admitted.contains(index) {
|
||||
continue;
|
||||
}
|
||||
let Some(cluster) = candidates[*index].cluster else {
|
||||
continue;
|
||||
};
|
||||
if admitted_per_cluster.get(&cluster).copied().unwrap_or(0) >= cap {
|
||||
if let Some(suppressed) = suppressed.as_deref_mut() {
|
||||
suppressed.insert(*index);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
admitted.insert(*index);
|
||||
*admitted_per_cluster.entry(cluster).or_default() += 1;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::DiversityConfig;
|
||||
use crate::curate::prefilter::tests::article;
|
||||
use crate::curate::signals::Signals;
|
||||
use crate::types::{Deep, Facets};
|
||||
|
||||
fn candidate(id: i64, utility_hint: f64) -> Candidate {
|
||||
let mut candidate = Candidate::new(article(id, &format!("article {id}"), 800), false);
|
||||
candidate.stage = "assessed".into();
|
||||
candidate.signals = Signals {
|
||||
heuristic: Some(utility_hint),
|
||||
..Signals::default()
|
||||
};
|
||||
candidate.assessment.deep = Some(Deep {
|
||||
quality: utility_hint,
|
||||
fit: utility_hint,
|
||||
category: Some("Top Stories".into()),
|
||||
rationale: "specific".into(),
|
||||
paywalled_guess: false,
|
||||
facets: Facets::default(),
|
||||
model: "mock".into(),
|
||||
prompt_version: 1,
|
||||
assessed_at: "2026-09-02T00:00:00Z".parse().expect("timestamp"),
|
||||
});
|
||||
candidate
|
||||
}
|
||||
|
||||
fn vector(angle: f32) -> Vec<f32> {
|
||||
vec![angle.cos(), angle.sin()]
|
||||
}
|
||||
|
||||
fn ranking(keep: usize, protected: usize, cap: usize, threshold: f64) -> RankingConfig {
|
||||
RankingConfig {
|
||||
shortlist_keep: keep,
|
||||
diversity: DiversityConfig {
|
||||
cluster_threshold: threshold,
|
||||
per_cluster_cap: cap,
|
||||
utility_protected: protected,
|
||||
},
|
||||
..RankingConfig::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn utility_renormalizes_present_signals_and_gates_learned_ones() {
|
||||
let mut a = candidate(1, 8.0);
|
||||
let mut b = candidate(2, 4.0);
|
||||
a.signals.interest = Some(1.0);
|
||||
b.signals.interest = None;
|
||||
a.signals.knn = Some(0.9);
|
||||
a.signals.knn_gate = 0.5;
|
||||
let mut values = vec![a, b];
|
||||
calculate_utility(&mut values, &UtilityWeights::default());
|
||||
let [a, b] = values.as_slice() else {
|
||||
panic!("two values")
|
||||
};
|
||||
for candidate in [&a, &b] {
|
||||
assert!((candidate.signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
|
||||
assert!(candidate.utility.is_some());
|
||||
}
|
||||
assert!(!b.signals.weights.contains_key("interest"));
|
||||
assert!(
|
||||
(a.signals.weights["knn"] / a.signals.weights["quality"] - (0.15 * 0.5) / 0.40).abs()
|
||||
< 1e-9
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn duplicates_cluster_and_the_third_is_suppressed() {
|
||||
let ranking = ranking(3, 0, 2, 0.85);
|
||||
let mut candidates = vec![
|
||||
candidate(1, 9.0),
|
||||
candidate(2, 8.0),
|
||||
candidate(3, 7.0),
|
||||
candidate(4, 6.0),
|
||||
];
|
||||
let embeddings = HashMap::from([
|
||||
(1, vector(0.0)),
|
||||
(2, vector(0.1)),
|
||||
(3, vector(0.2)),
|
||||
(4, vector(2.0)),
|
||||
]);
|
||||
let summary = shortlist(&mut candidates, &embeddings, &ranking);
|
||||
assert_eq!(summary.shortlisted, 3);
|
||||
assert_eq!(candidates[0].cluster, candidates[1].cluster);
|
||||
assert_eq!(candidates[1].cluster, candidates[2].cluster);
|
||||
assert_eq!(
|
||||
candidates[2].excluded_reason.as_deref(),
|
||||
Some("cluster_suppressed")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn protected_items_survive_and_count_toward_the_cap() {
|
||||
let ranking = ranking(3, 2, 1, 0.85);
|
||||
let mut candidates = vec![
|
||||
candidate(1, 9.0),
|
||||
candidate(2, 8.0),
|
||||
candidate(3, 7.0),
|
||||
candidate(4, 6.0),
|
||||
];
|
||||
let embeddings = HashMap::from([
|
||||
(1, vector(0.0)),
|
||||
(2, vector(0.05)),
|
||||
(3, vector(0.1)),
|
||||
(4, vector(2.0)),
|
||||
]);
|
||||
shortlist(&mut candidates, &embeddings, &ranking);
|
||||
assert_eq!(candidates[0].stage, "shortlisted");
|
||||
assert_eq!(candidates[1].stage, "shortlisted");
|
||||
assert_ne!(candidates[2].stage, "shortlisted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bridge_case_uses_leaders_not_transitive_components() {
|
||||
let ranking = ranking(3, 0, 2, 0.80);
|
||||
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
|
||||
// A at 0°, B at 72°, C at 36°: A~C and B~C, but A not~B.
|
||||
let embeddings =
|
||||
HashMap::from([(1, vector(0.0)), (2, vector(1.2566)), (3, vector(0.6283))]);
|
||||
let summary = shortlist(&mut candidates, &embeddings, &ranking);
|
||||
assert_eq!(summary.clusters, 2);
|
||||
assert_eq!(candidates[0].cluster, candidates[2].cluster);
|
||||
assert_ne!(candidates[0].cluster, candidates[1].cluster);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_embeddings_are_singletons_and_never_suppressed() {
|
||||
let ranking = ranking(3, 0, 1, 0.85);
|
||||
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
|
||||
shortlist(&mut candidates, &HashMap::new(), &ranking);
|
||||
assert!(
|
||||
candidates
|
||||
.iter()
|
||||
.all(|candidate| candidate.stage == "shortlisted")
|
||||
);
|
||||
assert_eq!(
|
||||
candidates
|
||||
.iter()
|
||||
.filter_map(|candidate| candidate.cluster)
|
||||
.collect::<HashSet<_>>()
|
||||
.len(),
|
||||
3
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn caps_relax_to_three_then_uncapped_when_the_shortlist_is_short() {
|
||||
// Six near-duplicates, keep 5: cap 2 admits two, cap 3 admits a third,
|
||||
// and the uncapped pass fills the remaining two slots in utility order.
|
||||
let ranking = ranking(5, 0, 2, 0.85);
|
||||
let mut candidates = (1..=6)
|
||||
.map(|id| candidate(id, 10.0 - id as f64))
|
||||
.collect::<Vec<_>>();
|
||||
let embeddings = (1..=6)
|
||||
.map(|id| (id, vector(0.01 * id as f32)))
|
||||
.collect::<HashMap<_, _>>();
|
||||
let summary = shortlist(&mut candidates, &embeddings, &ranking);
|
||||
assert_eq!(summary.clusters, 1);
|
||||
assert_eq!(summary.shortlisted, 5);
|
||||
let shortlisted = candidates
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "shortlisted")
|
||||
.map(|candidate| candidate.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(shortlisted, vec![1, 2, 3, 4, 5], "filled in utility order");
|
||||
assert_eq!(candidates[5].stage, "assessed");
|
||||
assert_eq!(
|
||||
candidates[5].excluded_reason.as_deref(),
|
||||
Some("cluster_suppressed")
|
||||
);
|
||||
assert_eq!(
|
||||
candidates
|
||||
.iter()
|
||||
.map(|c| c.rank_utility)
|
||||
.collect::<Vec<_>>(),
|
||||
(1..=6).map(Some).collect::<Vec<_>>()
|
||||
);
|
||||
assert_eq!(
|
||||
candidates
|
||||
.iter()
|
||||
.map(|c| c.cluster_rank)
|
||||
.collect::<Vec<_>>(),
|
||||
(1..=6).map(Some).collect::<Vec<_>>()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shortlist_cap_is_the_reason_beyond_the_keep() {
|
||||
let ranking = ranking(2, 0, 2, 0.85);
|
||||
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 7.0)];
|
||||
shortlist(&mut candidates, &HashMap::new(), &ranking);
|
||||
assert_eq!(candidates[0].stage, "shortlisted");
|
||||
assert_eq!(candidates[1].stage, "shortlisted");
|
||||
assert_eq!(candidates[2].stage, "assessed");
|
||||
assert_eq!(
|
||||
candidates[2].excluded_reason.as_deref(),
|
||||
Some("shortlist_cap")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exploration_picks_get_up_to_three_reserved_slots() {
|
||||
// Keep 4 with the four best by utility being ordinary articles: three
|
||||
// exploration picks are still reserved seats, the fourth is not.
|
||||
let ranking = ranking(4, 0, 2, 0.85);
|
||||
let mut candidates = (1..=8)
|
||||
.map(|id| candidate(id, 10.0 - id as f64))
|
||||
.collect::<Vec<_>>();
|
||||
for candidate in candidates.iter_mut().skip(4) {
|
||||
candidate.exploration = true;
|
||||
}
|
||||
let summary = shortlist(&mut candidates, &HashMap::new(), &ranking);
|
||||
assert_eq!(summary.shortlisted, 4);
|
||||
let shortlisted = candidates
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "shortlisted")
|
||||
.map(|candidate| candidate.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(shortlisted, vec![1, 5, 6, 7]);
|
||||
assert_eq!(
|
||||
candidates[7].excluded_reason.as_deref(),
|
||||
Some("shortlist_cap")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn auto_includes_are_admitted_regardless_and_count_toward_their_cluster() {
|
||||
let ranking = ranking(2, 0, 1, 0.85);
|
||||
let mut candidates = vec![candidate(1, 9.0), candidate(2, 8.0), candidate(3, 1.0)];
|
||||
candidates[2].auto_include = true;
|
||||
let embeddings = HashMap::from([(1, vector(0.0)), (2, vector(2.0)), (3, vector(0.05))]);
|
||||
let summary = shortlist(&mut candidates, &embeddings, &ranking);
|
||||
assert_eq!(summary.shortlisted, 2);
|
||||
assert_eq!(candidates[2].stage, "shortlisted", "auto-include survives");
|
||||
// The auto-include filled its cluster's single seat, so the stronger
|
||||
// near-duplicate is suppressed and the unrelated article gets the slot.
|
||||
assert_eq!(candidates[0].cluster, candidates[2].cluster);
|
||||
assert_eq!(candidates[0].stage, "assessed");
|
||||
assert_eq!(
|
||||
candidates[0].excluded_reason.as_deref(),
|
||||
Some("cluster_suppressed")
|
||||
);
|
||||
assert_eq!(candidates[1].stage, "shortlisted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unassessed_articles_rank_on_present_signals_and_keep_their_stage() {
|
||||
// DeepSeek down: no quality/fit anywhere, utility comes from what is present.
|
||||
let ranking = ranking(2, 0, 2, 0.85);
|
||||
let mut candidates = vec![candidate(1, 3.0), candidate(2, 6.0), candidate(3, 9.0)];
|
||||
for candidate in &mut candidates {
|
||||
candidate.assessment.deep = None;
|
||||
candidate.stage = "admitted".into();
|
||||
}
|
||||
candidates[0].assessment.triage = Some(crate::types::Triage {
|
||||
interest: 9.0,
|
||||
kind: "essay".into(),
|
||||
why: "promising".into(),
|
||||
model: "mock".into(),
|
||||
prompt_version: 1,
|
||||
assessed_at: "2026-09-02T00:00:00Z".parse().expect("timestamp"),
|
||||
});
|
||||
let summary = shortlist(&mut candidates, &HashMap::new(), &ranking);
|
||||
assert_eq!(summary.shortlisted, 2);
|
||||
for candidate in &candidates {
|
||||
assert!(candidate.utility.is_some(), "scored on present signals");
|
||||
assert!(!candidate.signals.weights.contains_key("quality"));
|
||||
assert!(!candidate.signals.weights.contains_key("fit"));
|
||||
assert!((candidate.signals.weights.values().sum::<f64>() - 1.0).abs() < 1e-9);
|
||||
}
|
||||
// Triage (0.05) outweighs heuristic (0.02): the triaged article with
|
||||
// the weakest heuristic overtakes the middle one.
|
||||
assert_eq!(candidates[2].rank_utility, Some(1));
|
||||
assert_eq!(candidates[0].rank_utility, Some(2));
|
||||
assert_eq!(candidates[1].rank_utility, Some(3));
|
||||
assert_eq!(
|
||||
candidates[1].stage, "admitted",
|
||||
"never assessed, so not `assessed`"
|
||||
);
|
||||
assert_eq!(
|
||||
candidates[1].excluded_reason.as_deref(),
|
||||
Some("shortlist_cap")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn percentiles_are_taken_over_the_deep_set_only() {
|
||||
// The eligible-but-not-admitted article has the strongest heuristic;
|
||||
// it must not shift the deep set's percentiles or receive a utility.
|
||||
let ranking = ranking(10, 0, 2, 0.85);
|
||||
let mut candidates = vec![candidate(1, 5.0), candidate(2, 5.0), candidate(3, 5.0)];
|
||||
candidates[2].stage = "triaged".into();
|
||||
candidates[2].excluded_reason = Some("not_admitted".into());
|
||||
candidates[2].signals.heuristic = Some(99.0);
|
||||
candidates[1].signals.heuristic = Some(5.0);
|
||||
shortlist(&mut candidates, &HashMap::new(), &ranking);
|
||||
assert_eq!(
|
||||
candidates[0].signals.norm["heuristic"], 0.5,
|
||||
"ties share a percentile"
|
||||
);
|
||||
assert_eq!(candidates[1].signals.norm["heuristic"], 0.5);
|
||||
assert!(candidates[2].utility.is_none());
|
||||
assert!(candidates[2].rank_utility.is_none());
|
||||
}
|
||||
}
|
||||
@@ -1,584 +0,0 @@
|
||||
//! Stage A — batched LLM scoring (spec §3.6).
|
||||
//!
|
||||
//! Batches of `deepseek.score_batch_size` articles per request. Per article we
|
||||
//! send title, source feed, author, word count, social stats, sources list and a
|
||||
//! ~200-word excerpt; the model returns one JSON object per article.
|
||||
//!
|
||||
//! Parsing is deliberately forgiving: one malformed item must not cost us the
|
||||
//! other eleven, and a failed batch must not fail the run.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fmt::Write as _;
|
||||
|
||||
use futures::{StreamExt, stream};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
|
||||
use super::llm::{LlmClient, LlmError, strip_code_fence};
|
||||
use super::{prompt_text, truncate_words};
|
||||
use crate::types::{ArticleId, LlmScore, ScoredArticle, SourceKind};
|
||||
|
||||
/// Words of article text sent per candidate in stage A (§3.6).
|
||||
pub const EXCERPT_WORDS: usize = 200;
|
||||
|
||||
/// One element of the stage-A JSON response (§3.6).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct ScoreItem {
|
||||
pub id: ArticleId,
|
||||
/// 0–10.
|
||||
pub score: f64,
|
||||
pub category: String,
|
||||
/// ≤ 20 words.
|
||||
#[serde(default)]
|
||||
pub rationale: String,
|
||||
#[serde(default)]
|
||||
pub is_paywalled_guess: bool,
|
||||
}
|
||||
|
||||
impl From<ScoreItem> for LlmScore {
|
||||
fn from(i: ScoreItem) -> Self {
|
||||
LlmScore {
|
||||
score: i.score,
|
||||
category: i.category,
|
||||
rationale: i.rationale,
|
||||
is_paywalled_guess: i.is_paywalled_guess,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Envelope the model is asked to return (`{"articles": [...]}`).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct ScoreResponse {
|
||||
#[serde(default)]
|
||||
pub articles: Vec<ScoreItem>,
|
||||
}
|
||||
|
||||
/// The invariant instruction block for stage A. Everything article-specific goes
|
||||
/// in the per-batch tail so this prefix stays cacheable (§3.6).
|
||||
pub const SCORE_INSTRUCTIONS: &str = "\
|
||||
TASK: score a batch of candidate articles for today's issue of The Daily EPUB.
|
||||
|
||||
Judge each article against the reader profile in your system prompt — not against \
|
||||
a general audience, and not against what is objectively newsworthy.
|
||||
|
||||
Return one object per input article with these fields:
|
||||
\"id\" integer, copied exactly from the input
|
||||
\"score\" number 0-10, the rubric below
|
||||
\"category\" one short label from the palette below
|
||||
\"rationale\" at most 20 words, concrete, no hedging, no restating the title
|
||||
\"is_paywalled_guess\" true when the text looks truncated, teaser-like or paywalled
|
||||
|
||||
SCORING RUBRIC — calibrate hard; a normal day averages about 4, and a 9 should \
|
||||
appear a couple of times a week, not a couple of times a day:
|
||||
9-10 Exceptional. Original reporting, a deep technical dive, or an essay he \
|
||||
will still be thinking about next week. Evident effort and a real point of view.
|
||||
7-8 Strong. A well-made long-form piece squarely in his interests, or an \
|
||||
outstanding piece outside them.
|
||||
5-6 Worth a slot on a thin day. Solid, useful, a little thin or a little \
|
||||
familiar.
|
||||
3-4 Marginal. Competent news-of-the-day, short posts, incremental updates, \
|
||||
good writing about an over-covered story.
|
||||
1-2 Weak. Announcements, changelogs and release notes, link roundups, \
|
||||
listicles, rewrites of a story available at the source, thin AI-industry churn.
|
||||
0 Unusable. Press releases, sponsored content, engagement bait, spam, \
|
||||
pure crypto promotion, or an entry with no readable body.
|
||||
|
||||
CALIBRATION NOTES
|
||||
- Length alone is not quality; padding scores worse than a tight short piece. But \
|
||||
between two equally good pieces, prefer the one with more substance.
|
||||
- Social proof is evidence, not a verdict: hundreds of HN points mean a critical \
|
||||
audience read it; a quiet post from a good blog can still outrank it.
|
||||
- \"came via scour\" means the story already matched one of his standing \
|
||||
interests. \"came via hn_frontpage\" means it cleared HN's front page.
|
||||
- Boston/New England local stories and ultra-niche community news get a genuine \
|
||||
lift — this paper wants them.
|
||||
- Wire-service world/US news should score low here: the World Briefing section \
|
||||
covers that separately.
|
||||
- Excerpt-only or paywalled text is a real cost to the reader; score it lower \
|
||||
unless the piece is clearly excellent.
|
||||
|
||||
Return JSON exactly in this shape, with one entry per input article and nothing \
|
||||
else:
|
||||
{\"articles\": [{\"id\": 123, \"score\": 7.5, \"category\": \"Tech & Engineering\", \
|
||||
\"rationale\": \"first-hand account of migrating 40TB off Postgres\", \
|
||||
\"is_paywalled_guess\": false}]}";
|
||||
|
||||
/// Render the user prompt for one batch (§3.6).
|
||||
pub fn build_batch_prompt(batch: &[ScoredArticle], sections: &[String]) -> String {
|
||||
let mut prompt = String::with_capacity(4096 + batch.len() * 1500);
|
||||
prompt.push_str(SCORE_INSTRUCTIONS);
|
||||
let _ = write!(
|
||||
prompt,
|
||||
"\n\nCATEGORY PALETTE (use one of these exact strings): {}\n\nARTICLES ({} in this batch)\n",
|
||||
sections.join(" | "),
|
||||
batch.len()
|
||||
);
|
||||
for candidate in batch {
|
||||
prompt.push('\n');
|
||||
prompt.push_str(&render_candidate(candidate));
|
||||
}
|
||||
prompt
|
||||
}
|
||||
|
||||
/// One article's block in the stage-A prompt (§3.6).
|
||||
fn render_candidate(candidate: &ScoredArticle) -> String {
|
||||
let a = &candidate.article;
|
||||
let mut block = String::with_capacity(1500);
|
||||
let _ = writeln!(block, "--- id: {}", a.id);
|
||||
let _ = writeln!(block, "title: {}", a.title.trim());
|
||||
let _ = writeln!(
|
||||
block,
|
||||
"feed: {}{}",
|
||||
if a.feed_title.is_empty() {
|
||||
"unknown"
|
||||
} else {
|
||||
a.feed_title.trim()
|
||||
},
|
||||
a.category
|
||||
.as_deref()
|
||||
.filter(|c| !c.is_empty())
|
||||
.map(|c| format!(" (category: {c})"))
|
||||
.unwrap_or_default()
|
||||
);
|
||||
if let Some(author) = a.author.as_deref().filter(|s| !s.trim().is_empty()) {
|
||||
let _ = writeln!(block, "author: {}", author.trim());
|
||||
}
|
||||
let _ = writeln!(
|
||||
block,
|
||||
"length: {} words (~{} min read){}",
|
||||
a.word_count,
|
||||
a.reading_minutes(),
|
||||
if a.excerpt_only {
|
||||
" [EXCERPT ONLY — full text unavailable]"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
);
|
||||
let _ = writeln!(block, "social: {}", social_line(candidate));
|
||||
let _ = writeln!(block, "came via: {}", sources_line(candidate));
|
||||
let excerpt = truncate_words(&prompt_text(&a.content_html), EXCERPT_WORDS);
|
||||
let _ = writeln!(
|
||||
block,
|
||||
"excerpt: {}",
|
||||
if excerpt.is_empty() {
|
||||
"(no body text extracted)"
|
||||
} else {
|
||||
&excerpt
|
||||
}
|
||||
);
|
||||
block
|
||||
}
|
||||
|
||||
fn social_line(candidate: &ScoredArticle) -> String {
|
||||
if candidate.article.social.is_empty() {
|
||||
return "none found".into();
|
||||
}
|
||||
let mut parts: Vec<String> = candidate
|
||||
.article
|
||||
.social
|
||||
.iter()
|
||||
.map(|s| {
|
||||
format!(
|
||||
"{} {} points / {} comments",
|
||||
s.source.display_name(),
|
||||
s.score,
|
||||
s.num_comments
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
parts.push(format!("composite {:.2}", candidate.social_score));
|
||||
parts.join("; ")
|
||||
}
|
||||
|
||||
fn sources_line(candidate: &ScoredArticle) -> String {
|
||||
let mut kinds: Vec<&str> = candidate
|
||||
.article
|
||||
.sources
|
||||
.iter()
|
||||
.map(|s| match s.kind {
|
||||
SourceKind::Scour => "scour",
|
||||
SourceKind::HnFrontpage => "hn_frontpage",
|
||||
SourceKind::Lobsters => "lobsters",
|
||||
SourceKind::Reddit => "reddit",
|
||||
SourceKind::Feed => "feed",
|
||||
})
|
||||
.collect();
|
||||
kinds.sort_unstable();
|
||||
kinds.dedup();
|
||||
if candidate.auto_include {
|
||||
kinds.push("always-include feed (cannot be dropped)");
|
||||
}
|
||||
if kinds.is_empty() {
|
||||
"feed".into()
|
||||
} else {
|
||||
kinds.join(", ")
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Response parsing (§3.6: tolerate anything the model does to us)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Keys the model might wrap the array in, in preference order.
|
||||
const ARRAY_KEYS: &[&str] = &["articles", "scores", "results", "items", "data"];
|
||||
|
||||
/// Parse a stage-A response leniently: missing optional fields default, scores
|
||||
/// are clamped to 0–10, and malformed items are skipped with a warning (§3.6).
|
||||
pub fn parse_score_response(raw: &str) -> Vec<ScoreItem> {
|
||||
let cleaned = strip_code_fence(raw);
|
||||
let value: Value = match serde_json::from_str(cleaned) {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "stage A response was not JSON at all");
|
||||
return Vec::new();
|
||||
}
|
||||
};
|
||||
|
||||
let array = match &value {
|
||||
Value::Array(items) => Some(items),
|
||||
Value::Object(map) => ARRAY_KEYS
|
||||
.iter()
|
||||
.find_map(|k| map.get(*k).and_then(Value::as_array))
|
||||
// Some models return {"1234": {...}} or a single bare object.
|
||||
.or_else(|| map.values().find_map(Value::as_array)),
|
||||
_ => None,
|
||||
};
|
||||
let Some(array) = array else {
|
||||
tracing::warn!("stage A response contained no array of scores");
|
||||
return Vec::new();
|
||||
};
|
||||
|
||||
let mut out = Vec::with_capacity(array.len());
|
||||
let mut skipped = 0usize;
|
||||
for item in array {
|
||||
match parse_item(item) {
|
||||
Some(parsed) => out.push(parsed),
|
||||
None => {
|
||||
skipped += 1;
|
||||
tracing::warn!(item = %truncate_debug(item), "skipping malformed stage A item");
|
||||
}
|
||||
}
|
||||
}
|
||||
if skipped > 0 {
|
||||
tracing::warn!(skipped, kept = out.len(), "stage A items were dropped");
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn parse_item(item: &Value) -> Option<ScoreItem> {
|
||||
let obj = item.as_object()?;
|
||||
let id = obj.get("id").and_then(as_i64_lenient)?;
|
||||
let score = obj
|
||||
.get("score")
|
||||
.and_then(as_f64_lenient)
|
||||
.or_else(|| obj.get("rating").and_then(as_f64_lenient))?;
|
||||
Some(ScoreItem {
|
||||
id,
|
||||
score: score.clamp(0.0, 10.0),
|
||||
category: obj
|
||||
.get("category")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or_default()
|
||||
.trim()
|
||||
.to_string(),
|
||||
rationale: obj
|
||||
.get("rationale")
|
||||
.or_else(|| obj.get("reason"))
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or_default()
|
||||
.trim()
|
||||
.to_string(),
|
||||
is_paywalled_guess: obj
|
||||
.get("is_paywalled_guess")
|
||||
.or_else(|| obj.get("paywalled"))
|
||||
.and_then(as_bool_lenient)
|
||||
.unwrap_or(false),
|
||||
})
|
||||
}
|
||||
|
||||
fn as_i64_lenient(v: &Value) -> Option<i64> {
|
||||
v.as_i64()
|
||||
.or_else(|| v.as_f64().map(|f| f as i64))
|
||||
.or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
|
||||
}
|
||||
|
||||
fn as_f64_lenient(v: &Value) -> Option<f64> {
|
||||
v.as_f64()
|
||||
.or_else(|| v.as_str().and_then(|s| s.trim().parse().ok()))
|
||||
.filter(|f| f.is_finite())
|
||||
}
|
||||
|
||||
fn as_bool_lenient(v: &Value) -> Option<bool> {
|
||||
v.as_bool().or_else(|| match v.as_str()?.trim() {
|
||||
"true" | "yes" => Some(true),
|
||||
"false" | "no" => Some(false),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
fn truncate_debug(v: &Value) -> String {
|
||||
v.to_string().chars().take(160).collect()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Stage driver
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Score every candidate, filling in [`ScoredArticle::llm`] (§3.6).
|
||||
///
|
||||
/// Batches that fail are logged and left unscored rather than aborting the run.
|
||||
/// Returns how many candidates came back with a score.
|
||||
pub async fn score_all(
|
||||
llm: &LlmClient,
|
||||
candidates: &mut [ScoredArticle],
|
||||
batch_size: usize,
|
||||
max_concurrent_requests: usize,
|
||||
sections: &[String],
|
||||
temperature: f32,
|
||||
) -> Result<usize, LlmError> {
|
||||
if candidates.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let batch_size = batch_size.max(1);
|
||||
let batches = candidates.len().div_ceil(batch_size);
|
||||
let prompts = candidates
|
||||
.chunks(batch_size)
|
||||
.enumerate()
|
||||
.map(|(index, batch)| (index, batch.len(), build_batch_prompt(batch, sections)))
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let results = stream::iter(prompts)
|
||||
.map(|(index, article_count, prompt)| async move {
|
||||
if let Err(error) = llm.meter.check_budget() {
|
||||
tracing::warn!(batch = index + 1, of = batches, %error, "bulk budget tripped; skipping stage A batch");
|
||||
return (index, Vec::new());
|
||||
}
|
||||
tracing::debug!(batch = index + 1, of = batches, articles = article_count, approx_tokens = super::approx_tokens(&prompt), "stage A request");
|
||||
let items = match llm.complete(&prompt, temperature, true).await {
|
||||
Ok(raw) => parse_score_response(&raw),
|
||||
Err(error) => {
|
||||
tracing::warn!(batch = index + 1, of = batches, %error, "stage A batch failed; its articles stay unscored");
|
||||
Vec::new()
|
||||
}
|
||||
};
|
||||
(index, items)
|
||||
})
|
||||
.buffer_unordered(max_concurrent_requests.max(1))
|
||||
.collect::<Vec<_>>()
|
||||
.await;
|
||||
|
||||
let mut scores: HashMap<ArticleId, LlmScore> = HashMap::with_capacity(candidates.len());
|
||||
for (_, items) in results {
|
||||
for item in items {
|
||||
scores.insert(item.id, item.into());
|
||||
}
|
||||
}
|
||||
let mut applied = 0;
|
||||
for candidate in candidates {
|
||||
if let Some(score) = scores.remove(&candidate.article.id) {
|
||||
candidate.llm = Some(score);
|
||||
applied += 1;
|
||||
}
|
||||
}
|
||||
if !scores.is_empty() {
|
||||
tracing::warn!(unknown_ids = scores.len(), "stage A returned unknown ids");
|
||||
}
|
||||
Ok(applied)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::DeepseekConfig;
|
||||
use crate::curate::llm::{MockBackend, UsageMeter};
|
||||
use crate::curate::prefilter::tests::{article, via, with_social};
|
||||
use crate::types::TokenUsage;
|
||||
use std::sync::Arc;
|
||||
|
||||
const BATCH_FIXTURE: &str = include_str!(concat!(
|
||||
env!("CARGO_MANIFEST_DIR"),
|
||||
"/tests/fixtures/deepseek_score_batch.json"
|
||||
));
|
||||
const MESSY_FIXTURE: &str = include_str!(concat!(
|
||||
env!("CARGO_MANIFEST_DIR"),
|
||||
"/tests/fixtures/deepseek_score_batch_messy.json"
|
||||
));
|
||||
|
||||
fn sections() -> Vec<String> {
|
||||
crate::config::CurationConfig::default().sections
|
||||
}
|
||||
|
||||
fn candidate(id: i64, title: &str, words: i64) -> ScoredArticle {
|
||||
ScoredArticle {
|
||||
article: article(id, title, words),
|
||||
prefilter_score: 50.0,
|
||||
social_score: 0.0,
|
||||
llm: None,
|
||||
triage: None,
|
||||
auto_include: false,
|
||||
exploration: false,
|
||||
admitted_by: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_prompt_carries_every_documented_signal() {
|
||||
let mut c = candidate(12, "Migrating 40TB off Postgres", 3200);
|
||||
c.article = via(
|
||||
with_social(c.article, 342, 210),
|
||||
SourceKind::HnFrontpage,
|
||||
9001,
|
||||
);
|
||||
c.social_score = c.article.social_score();
|
||||
c.auto_include = true;
|
||||
let prompt = build_batch_prompt(&[c], §ions());
|
||||
|
||||
assert!(prompt.starts_with(SCORE_INSTRUCTIONS));
|
||||
assert!(prompt.contains("--- id: 12"));
|
||||
assert!(prompt.contains("title: Migrating 40TB off Postgres"));
|
||||
assert!(prompt.contains("feed: Some Blog (category: Tech)"));
|
||||
assert!(prompt.contains("author: A. Writer"));
|
||||
assert!(prompt.contains("length: 3200 words"));
|
||||
assert!(prompt.contains("HN 342 points / 210 comments"));
|
||||
assert!(prompt.contains("hn_frontpage"));
|
||||
assert!(prompt.contains("always-include feed"));
|
||||
assert!(prompt.contains("excerpt: word word"));
|
||||
assert!(prompt.contains("Tech & Engineering"));
|
||||
// The excerpt is capped.
|
||||
let excerpt_line = prompt
|
||||
.lines()
|
||||
.find(|l| l.starts_with("excerpt:"))
|
||||
.expect("excerpt line");
|
||||
assert!(excerpt_line.split_whitespace().count() <= EXCERPT_WORDS + 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_a_realistic_deepseek_batch() {
|
||||
let items = parse_score_response(BATCH_FIXTURE);
|
||||
assert_eq!(items.len(), 4);
|
||||
assert_eq!(items[0].id, 101);
|
||||
assert!((items[0].score - 8.5).abs() < 1e-9);
|
||||
assert_eq!(items[0].category, "Tech & Engineering");
|
||||
assert!(items[0].rationale.split_whitespace().count() <= 20);
|
||||
assert!(!items[0].is_paywalled_guess);
|
||||
assert!(items[3].is_paywalled_guess);
|
||||
let score: LlmScore = items[0].clone().into();
|
||||
assert_eq!(score.category, "Tech & Engineering");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parsing_survives_everything_a_model_might_do() {
|
||||
let items = parse_score_response(MESSY_FIXTURE);
|
||||
let ids: Vec<ArticleId> = items.iter().map(|i| i.id).collect();
|
||||
// 201 fine; 202 string score clamped; 203 missing rationale/category;
|
||||
// 204 out-of-range clamped; the two malformed entries are dropped.
|
||||
assert_eq!(ids, vec![201, 202, 203, 204]);
|
||||
assert!((items[1].score - 6.0).abs() < 1e-9);
|
||||
assert_eq!(items[2].rationale, "");
|
||||
assert_eq!(items[2].category, "");
|
||||
assert!(
|
||||
(items[3].score - 10.0).abs() < 1e-9,
|
||||
"clamped to the 0-10 range"
|
||||
);
|
||||
assert!(items.iter().all(|i| (0.0..=10.0).contains(&i.score)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parsing_tolerates_fences_arrays_and_junk() {
|
||||
assert_eq!(
|
||||
parse_score_response("```json\n{\"articles\":[{\"id\":1,\"score\":5}]}\n```").len(),
|
||||
1
|
||||
);
|
||||
assert_eq!(parse_score_response("[{\"id\": 2, \"score\": 3}]").len(), 1);
|
||||
assert_eq!(
|
||||
parse_score_response("{\"results\":[{\"id\":3,\"score\":\"4.5\"}]}")[0].score,
|
||||
4.5
|
||||
);
|
||||
assert!(parse_score_response("I'm sorry, I can't do that").is_empty());
|
||||
assert!(parse_score_response("{\"articles\": {}}").is_empty());
|
||||
}
|
||||
|
||||
fn client(backend: Arc<MockBackend>, limit_usd: f64) -> LlmClient {
|
||||
LlmClient::with_backend(
|
||||
"deepseek-v4-flash",
|
||||
"SYSTEM".into(),
|
||||
UsageMeter::new(&DeepseekConfig::default(), limit_usd),
|
||||
backend,
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scores_are_applied_batch_by_batch() {
|
||||
let backend = Arc::new(MockBackend::new());
|
||||
backend.push(
|
||||
r#"{"articles":[{"id":1,"score":8,"category":"Tech & Engineering","rationale":"good"},
|
||||
{"id":2,"score":2,"category":"Niche Corner","rationale":"thin"}]}"#,
|
||||
TokenUsage::default(),
|
||||
);
|
||||
backend.push(
|
||||
r#"{"articles":[{"id":3,"score":6.5,"category":"Culture & Essays","rationale":"solid"}]}"#,
|
||||
TokenUsage::default(),
|
||||
);
|
||||
let llm = client(Arc::clone(&backend), 2.0);
|
||||
|
||||
let mut candidates = vec![
|
||||
candidate(1, "One", 1000),
|
||||
candidate(2, "Two", 1000),
|
||||
candidate(3, "Three", 1000),
|
||||
];
|
||||
let scored = score_all(&llm, &mut candidates, 2, 4, §ions(), 0.3)
|
||||
.await
|
||||
.expect("scoring");
|
||||
assert_eq!(scored, 3);
|
||||
assert_eq!(backend.calls(), 2, "batched by score_batch_size");
|
||||
assert_eq!(candidates[0].llm.as_ref().map(|l| l.score), Some(8.0));
|
||||
assert_eq!(candidates[2].llm.as_ref().map(|l| l.score), Some(6.5));
|
||||
// combined_score now reflects the LLM verdict.
|
||||
assert!(candidates[0].combined_score() > candidates[1].combined_score());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_failed_batch_does_not_sink_the_run() {
|
||||
let backend = Arc::new(MockBackend::new());
|
||||
backend.push_error("500 upstream exploded");
|
||||
backend.push(
|
||||
r#"{"articles":[{"id":2,"score":7,"category":"Top Stories","rationale":"ok"}]}"#,
|
||||
TokenUsage::default(),
|
||||
);
|
||||
let llm = client(Arc::clone(&backend), 2.0);
|
||||
let mut candidates = vec![candidate(1, "One", 900), candidate(2, "Two", 900)];
|
||||
let scored = score_all(&llm, &mut candidates, 1, 4, §ions(), 0.3)
|
||||
.await
|
||||
.expect("scoring must not abort");
|
||||
assert_eq!(scored, 1);
|
||||
assert!(candidates[0].llm.is_none());
|
||||
assert!(candidates[1].llm.is_some());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scoring_stops_when_the_budget_is_gone() {
|
||||
let backend = Arc::new(MockBackend::new());
|
||||
// First batch alone blows a $0.05 ceiling ($0.14 per 1M input tokens).
|
||||
backend.push(
|
||||
r#"{"articles":[{"id":1,"score":9,"category":"Top Stories","rationale":"great"}]}"#,
|
||||
TokenUsage {
|
||||
input_tokens: 1_000_000,
|
||||
cached_tokens: 0,
|
||||
cache_write_tokens: 0,
|
||||
output_tokens: 0,
|
||||
},
|
||||
);
|
||||
backend.push(
|
||||
r#"{"articles":[{"id":2,"score":9,"category":"Top Stories","rationale":"great"}]}"#,
|
||||
TokenUsage::default(),
|
||||
);
|
||||
let llm = client(Arc::clone(&backend), 0.05);
|
||||
let mut candidates = vec![candidate(1, "One", 900), candidate(2, "Two", 900)];
|
||||
let scored = score_all(&llm, &mut candidates, 1, 4, §ions(), 0.3)
|
||||
.await
|
||||
.expect("scoring");
|
||||
assert_eq!(scored, 1, "only the first batch ran");
|
||||
assert_eq!(backend.calls(), 1);
|
||||
assert!(llm.meter.budget_exceeded());
|
||||
}
|
||||
}
|
||||
+68
-22
@@ -205,6 +205,18 @@ pub fn serialize_candidate(candidate: &Candidate) -> String {
|
||||
.insert("triage".into(), (triage.interest / 10.0).clamp(0.0, 1.0));
|
||||
value.present.insert("triage".into(), true);
|
||||
}
|
||||
if let Some(deep) = candidate.assessment.deep.as_ref() {
|
||||
value.raw.insert("quality".into(), deep.quality);
|
||||
value.raw.insert("fit".into(), deep.fit);
|
||||
value
|
||||
.norm
|
||||
.insert("quality".into(), (deep.quality / 10.0).clamp(0.0, 1.0));
|
||||
value
|
||||
.norm
|
||||
.insert("fit".into(), (deep.fit / 10.0).clamp(0.0, 1.0));
|
||||
value.present.insert("quality".into(), true);
|
||||
value.present.insert("fit".into(), true);
|
||||
}
|
||||
serde_json::to_string(&value).unwrap_or_else(|_| "{}".into())
|
||||
}
|
||||
|
||||
@@ -251,7 +263,7 @@ impl ExplainRow {
|
||||
serde_json::from_str(&self.signals_json).ok()
|
||||
}
|
||||
|
||||
/// Utility when step 5 has written it, else the preliminary blend.
|
||||
/// Utility once the deep set has been ranked, else the preliminary blend.
|
||||
pub fn score(&self) -> Option<f64> {
|
||||
self.utility
|
||||
.or_else(|| self.signals().and_then(|signals| signals.blend()))
|
||||
@@ -305,7 +317,8 @@ pub async fn explain_row(
|
||||
Ok(row.as_ref().map(ExplainRow::from_row))
|
||||
}
|
||||
|
||||
/// The top `limit` rows by utility-or-blend that were not selected (§15.2).
|
||||
/// The top `limit` rows that were not selected, by utility, falling back to
|
||||
/// the preliminary blend for rows the ranker never reached (§15.2).
|
||||
pub async fn near_misses(
|
||||
db: &Db,
|
||||
run_id: i64,
|
||||
@@ -413,12 +426,16 @@ pub async fn render_explain(db: &Db, row: &ExplainRow) -> Result<String, sqlx::E
|
||||
if !assessments.is_empty() {
|
||||
let _ = writeln!(out, "assessments:");
|
||||
for assessment in assessments {
|
||||
let stage = assessment.get::<String, _>("stage");
|
||||
let model = assessment.get::<String, _>("model");
|
||||
let score = fmt_opt(assessment.get::<Option<f64>, _>("score"));
|
||||
let rationale = assessment
|
||||
.get::<Option<String>, _>("rationale")
|
||||
.unwrap_or_default();
|
||||
if stage == "deep" {
|
||||
let _ = writeln!(
|
||||
out,
|
||||
" {} · {} · score {} · fit {} · kind {} · category {} · paywalled={} · {}",
|
||||
assessment.get::<String, _>("stage"),
|
||||
assessment.get::<String, _>("model"),
|
||||
fmt_opt(assessment.get::<Option<f64>, _>("score")),
|
||||
" deep · {model} · quality {score} · fit {} · format {} · category {} · paywalled={} · {rationale}",
|
||||
fmt_opt(assessment.get::<Option<f64>, _>("fit")),
|
||||
assessment
|
||||
.get::<Option<String>, _>("kind")
|
||||
@@ -427,10 +444,16 @@ pub async fn render_explain(db: &Db, row: &ExplainRow) -> Result<String, sqlx::E
|
||||
.get::<Option<String>, _>("category")
|
||||
.unwrap_or_else(|| "—".into()),
|
||||
assessment.get::<i64, _>("paywalled_guess") != 0,
|
||||
assessment
|
||||
.get::<Option<String>, _>("rationale")
|
||||
.unwrap_or_default(),
|
||||
);
|
||||
} else {
|
||||
let _ = writeln!(
|
||||
out,
|
||||
" triage · {model} · interest {score} · kind {} · {rationale}",
|
||||
assessment
|
||||
.get::<Option<String>, _>("kind")
|
||||
.unwrap_or_else(|| "—".into()),
|
||||
);
|
||||
}
|
||||
if let Some(facets) = assessment.get::<Option<String>, _>("facets_json") {
|
||||
let _ = writeln!(out, " facets: {facets}");
|
||||
}
|
||||
@@ -876,16 +899,21 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn near_misses_rank_by_blend_and_skip_selected_and_excluded() {
|
||||
let (_dir, db) = db_with_articles(&[1, 2, 3, 4, 5]).await;
|
||||
async fn near_misses_rank_by_utility_then_blend_and_skip_selected_and_excluded() {
|
||||
let (_dir, db) = db_with_articles(&[1, 2, 3, 4, 5, 6]).await;
|
||||
let run_id = db.start_run(date(), Timestamp::now()).await.unwrap();
|
||||
// Utility decides wherever the ranker wrote one; the preliminary blend
|
||||
// only stands in for rows the deep set never reached. Article 4's blend
|
||||
// would put it first, but its utility is the lowest; article 3 never
|
||||
// got a utility and ranks on its blend.
|
||||
let rows = [
|
||||
(1, "selected", None, 0.9),
|
||||
(2, "shortlisted", Some("not_selected"), 0.7),
|
||||
(3, "eligible", Some("not_admitted"), 0.95),
|
||||
(4, "shortlisted", Some("not_selected"), 0.1),
|
||||
(1, "selected", None, 0.9, Some(90.0)),
|
||||
(2, "shortlisted", Some("not_selected"), 0.1, Some(70.0)),
|
||||
(3, "eligible", Some("not_admitted"), 0.95, None),
|
||||
(4, "shortlisted", Some("not_selected"), 0.99, Some(10.0)),
|
||||
(6, "assessed", Some("cluster_suppressed"), 0.5, Some(40.0)),
|
||||
];
|
||||
for (id, stage, reason, norm) in rows {
|
||||
for (id, stage, reason, norm, utility) in rows {
|
||||
let json = serialize_signals(&signals(10.0, norm), false);
|
||||
write(
|
||||
&db,
|
||||
@@ -896,7 +924,7 @@ mod tests {
|
||||
excluded_reason: reason,
|
||||
admitted_by: None,
|
||||
signals_json: &json,
|
||||
utility: None,
|
||||
utility,
|
||||
rank_utility: None,
|
||||
cluster_id: None,
|
||||
cluster_rank: None,
|
||||
@@ -912,18 +940,36 @@ mod tests {
|
||||
let misses = near_misses(&db, run_id, 10).await.unwrap();
|
||||
assert_eq!(
|
||||
misses.iter().map(|row| row.article_id).collect::<Vec<_>>(),
|
||||
vec![3, 2, 4]
|
||||
vec![3, 2, 6, 4]
|
||||
);
|
||||
let text = explain_near_misses(&db, date(), None, 2).await.unwrap();
|
||||
assert!(
|
||||
text.contains("top 2 not selected, by preliminary blend"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(text.contains("top 2 not selected, by utility"), "{text}");
|
||||
assert!(
|
||||
text.contains("Article 3 · eligible, not_admitted"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("Article 2 · shortlisted, not_selected"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(!text.contains("Article 4"), "{text}");
|
||||
|
||||
// Without any utility the listing says so and orders by the blend.
|
||||
sqlx::query("UPDATE candidate_runs SET utility = NULL WHERE run_id = ?")
|
||||
.bind(run_id)
|
||||
.execute(db.pool())
|
||||
.await
|
||||
.unwrap();
|
||||
let misses = near_misses(&db, run_id, 10).await.unwrap();
|
||||
assert_eq!(
|
||||
misses.iter().map(|row| row.article_id).collect::<Vec<_>>(),
|
||||
vec![4, 3, 6, 2]
|
||||
);
|
||||
let text = explain_near_misses(&db, date(), None, 1).await.unwrap();
|
||||
assert!(
|
||||
text.contains("top 1 not selected, by preliminary blend"),
|
||||
"{text}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -357,11 +357,14 @@ pub async fn run(
|
||||
let rows = sqlx::query(
|
||||
"SELECT article_id, stage, score, kind, rationale, assessed_at
|
||||
FROM article_assessments
|
||||
WHERE model = ? AND prompt_version = ? AND assessed_at >= ?",
|
||||
WHERE model = ? AND assessed_at >= ?
|
||||
AND ((stage = 'triage' AND prompt_version = ?)
|
||||
OR (stage = 'deep' AND prompt_version = ?))",
|
||||
)
|
||||
.bind(&llm.model)
|
||||
.bind(TRIAGE_PROMPT_VERSION)
|
||||
.bind(fmt_ts(since))
|
||||
.bind(TRIAGE_PROMPT_VERSION)
|
||||
.bind(super::assess::DEEP_PROMPT_VERSION)
|
||||
.fetch_all(db.pool())
|
||||
.await?;
|
||||
let pool_ids = pool;
|
||||
|
||||
+2
-8
@@ -343,17 +343,11 @@ fn print_report(report: &RunReport) {
|
||||
report.counts.duplicates_merged,
|
||||
report.counts.entries_dropped,
|
||||
);
|
||||
let unscored = if report.counts.llm_unscored > 0 {
|
||||
format!(" ({} unscored)", report.counts.llm_unscored)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
println!(
|
||||
"curation: {} eligible · {} embedded · {} triaged → {} admitted → {} assessed{unscored} → {} shortlisted → {} selected",
|
||||
"curation: {} considered → {} eligible → {} triaged → {} assessed → {} shortlisted → {} selected",
|
||||
report.counts.articles,
|
||||
report.counts.eligible,
|
||||
report.counts.embedded,
|
||||
report.counts.triaged,
|
||||
report.counts.admitted,
|
||||
report.counts.assessed,
|
||||
report.counts.shortlisted,
|
||||
report.counts.selected,
|
||||
|
||||
+144
-43
@@ -33,7 +33,9 @@ use jiff::{Timestamp, Zoned};
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::curate::llm::{Llms, PriceTable, UsageMeter};
|
||||
use crate::curate::{Curator, admit, editorial, embedding, profile, signals, telemetry, triage};
|
||||
use crate::curate::{
|
||||
Curator, admit, editorial, embedding, profile, rank, signals, telemetry, triage,
|
||||
};
|
||||
use crate::db::Db;
|
||||
use crate::extract::Extractor;
|
||||
use crate::miniflux::MinifluxClient;
|
||||
@@ -399,10 +401,10 @@ async fn run_stages(
|
||||
report.counts.eligible = personalized.len() as i64;
|
||||
report.timings.record("hygiene", elapsed_ms(stage));
|
||||
let embeddings = build_embedding_service(ctx, report);
|
||||
prepare_features(ctx, &mut personalized, &embeddings, report).await;
|
||||
let article_embeddings = prepare_features(ctx, &mut personalized, &embeddings, report).await;
|
||||
|
||||
// Build the provider clients before triage. A missing or failed bulk client
|
||||
// skips triage and stage A, while the editor can still run on Claude (§17).
|
||||
// skips triage and deep assessment, while the editor can still run on Claude (§17).
|
||||
let stage = Timestamp::now();
|
||||
let bulk_meter =
|
||||
UsageMeter::with_prices(PriceTable::deepseek(&config.deepseek), config.max_daily_usd);
|
||||
@@ -432,13 +434,13 @@ async fn run_stages(
|
||||
// --- Stage 7: triage (§10) ---
|
||||
let stage = Timestamp::now();
|
||||
let triage_pool = triage::apply_pool_cap(&mut personalized, config.curation.ranking.triage_max);
|
||||
if let Some(bulk) = curator.llms.bulk.as_ref() {
|
||||
let profile_version = db
|
||||
.kv_get(crate::db::KV_PROFILE_VERSION)
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
.and_then(|value| value.parse().ok());
|
||||
if let Some(bulk) = curator.llms.bulk.as_ref() {
|
||||
if let Err(error) = triage::run(
|
||||
db,
|
||||
bulk,
|
||||
@@ -500,41 +502,57 @@ async fn run_stages(
|
||||
);
|
||||
report.timings.record("admit", elapsed_ms(stage));
|
||||
|
||||
let admitted = personalized
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "admitted")
|
||||
.map(|candidate| candidate.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
let mut candidates = personalized
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "admitted")
|
||||
.cloned()
|
||||
.map(Candidate::into_legacy_scored)
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
// --- Stage 9: legacy Stage A scoring, then editor (§21 step 4) ---
|
||||
// --- Stage 9: deep assessment (§12.1) ---
|
||||
let stage = Timestamp::now();
|
||||
if bulk_available && let Err(e) = curator.score(&mut candidates, date).await {
|
||||
// A dead API or a tripped budget must not cost us the issue: selection
|
||||
// degrades to preliminary-blend order exactly as `--skip-llm` does.
|
||||
report.warn(format!("LLM scoring failed; ranking heuristically: {e:#}"));
|
||||
if let Err(error) = curator
|
||||
.assess(
|
||||
&mut personalized,
|
||||
ctx.rescore,
|
||||
profile_version,
|
||||
Timestamp::now(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
report.warn(format!(
|
||||
"deep assessment degraded; ranking continues on present signals: {error:#}"
|
||||
));
|
||||
}
|
||||
report.counts.llm_scored = candidates.iter().filter(|c| c.llm.is_some()).count() as i64;
|
||||
report.counts.llm_unscored = report.counts.candidates - report.counts.llm_scored;
|
||||
report.counts.assessed = report.counts.llm_scored;
|
||||
report.counts.shortlisted = admitted.len() as i64;
|
||||
let assessed = candidates
|
||||
report.counts.assessed = personalized
|
||||
.iter()
|
||||
.filter(|candidate| candidate.llm.is_some())
|
||||
.map(|candidate| candidate.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
set_candidate_stage(&mut personalized, &assessed, "assessed", None);
|
||||
// Every admitted article goes to the old selector, scored or not.
|
||||
set_candidate_stage(&mut personalized, &admitted, "shortlisted", None);
|
||||
.filter(|candidate| candidate.assessment.deep.is_some())
|
||||
.count() as i64;
|
||||
record_candidates(ctx, &personalized)
|
||||
.await
|
||||
.context("recording assessment telemetry")?;
|
||||
report.timings.record("assess", elapsed_ms(stage));
|
||||
|
||||
// --- Stage 10: utility and diversified shortlist (§12.2–§12.5) ---
|
||||
let stage = Timestamp::now();
|
||||
let ranked = rank::shortlist(
|
||||
&mut personalized,
|
||||
&article_embeddings,
|
||||
&config.curation.ranking,
|
||||
);
|
||||
report.counts.shortlisted = ranked.shortlisted as i64;
|
||||
report.counts.clusters = ranked.clusters as i64;
|
||||
record_candidates(ctx, &personalized)
|
||||
.await
|
||||
.context("recording ranking telemetry")?;
|
||||
report.timings.record("rank", elapsed_ms(stage));
|
||||
|
||||
let mut candidates = personalized
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "shortlisted")
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
candidates.sort_by_key(|candidate| candidate.rank_utility.unwrap_or(i64::MAX));
|
||||
let shortlisted = candidates
|
||||
.iter()
|
||||
.map(|candidate| candidate.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
// --- Stage 11: editor (§13) ---
|
||||
let stage = Timestamp::now();
|
||||
let mut lineup = curator
|
||||
.select(candidates, date)
|
||||
.await
|
||||
@@ -546,7 +564,7 @@ async fn run_stages(
|
||||
.map(|pick| pick.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
let selected_set = selected.iter().copied().collect::<HashSet<_>>();
|
||||
let not_selected = admitted
|
||||
let not_selected = shortlisted
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|id| !selected_set.contains(id))
|
||||
@@ -581,7 +599,7 @@ async fn run_stages(
|
||||
if lineup.picks.is_empty() {
|
||||
report.warn("the lineup is empty — check the lookback window and admission settings");
|
||||
}
|
||||
report.timings.record("curate", elapsed_ms(stage));
|
||||
report.timings.record("editor", elapsed_ms(stage));
|
||||
|
||||
// --- Stage 8: comment chapters for the selected articles (§3.7) ---
|
||||
let stage = Timestamp::now();
|
||||
@@ -778,7 +796,7 @@ async fn prepare_features(
|
||||
candidates: &mut [Candidate],
|
||||
service: &embedding::EmbeddingService,
|
||||
report: &mut RunReport,
|
||||
) {
|
||||
) -> HashMap<ArticleId, Vec<f32>> {
|
||||
let (config, db) = (ctx.config, ctx.db);
|
||||
let eligible = candidates
|
||||
.iter()
|
||||
@@ -863,6 +881,7 @@ async fn prepare_features(
|
||||
report.warn(format!("could not record eligible candidates: {error}"));
|
||||
}
|
||||
report.timings.record("signals", elapsed_ms(stage));
|
||||
article_embeddings
|
||||
}
|
||||
|
||||
async fn record_candidates(ctx: &StageContext<'_>, candidates: &[Candidate]) -> Result<()> {
|
||||
@@ -891,9 +910,9 @@ async fn record_candidates_with_why(
|
||||
admitted_by: admitted_by.as_deref(),
|
||||
signals_json: &json,
|
||||
utility: candidate.utility,
|
||||
rank_utility: None,
|
||||
rank_utility: candidate.rank_utility,
|
||||
cluster_id: candidate.cluster,
|
||||
cluster_rank: None,
|
||||
cluster_rank: candidate.cluster_rank,
|
||||
editor_why: editor_why.get(&candidate.article.id).copied().flatten(),
|
||||
},
|
||||
)
|
||||
@@ -1077,7 +1096,7 @@ fn log_resolved_providers(config: &Config, skip_llm: bool, skip_embeddings: bool
|
||||
/// Bump a number when the corresponding instruction block changes.
|
||||
const PROMPT_VERSIONS: &[(&str, u32)] = &[
|
||||
("triage", triage::TRIAGE_PROMPT_VERSION as u32),
|
||||
("score", 1),
|
||||
("deep", crate::curate::assess::DEEP_PROMPT_VERSION as u32),
|
||||
("editor", 2),
|
||||
("summary", 1),
|
||||
("brief", 2),
|
||||
@@ -1095,6 +1114,7 @@ fn resolved_run_config(config: &Config, soft_target: usize, hard_max: usize) ->
|
||||
serde_json::json!({
|
||||
"target_article_count": soft_target,
|
||||
"TRIAGE_PROMPT_VERSION": triage::TRIAGE_PROMPT_VERSION,
|
||||
"DEEP_PROMPT_VERSION": crate::curate::assess::DEEP_PROMPT_VERSION,
|
||||
"curation": curation,
|
||||
"editorial": config.editorial,
|
||||
"voyage": voyage,
|
||||
@@ -1196,6 +1216,10 @@ mod tests {
|
||||
value["TRIAGE_PROMPT_VERSION"],
|
||||
triage::TRIAGE_PROMPT_VERSION
|
||||
);
|
||||
assert_eq!(
|
||||
value["DEEP_PROMPT_VERSION"],
|
||||
crate::curate::assess::DEEP_PROMPT_VERSION
|
||||
);
|
||||
assert_eq!(
|
||||
value["prompt_versions"]["triage"],
|
||||
triage::TRIAGE_PROMPT_VERSION
|
||||
@@ -1246,6 +1270,7 @@ mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::curate::embedding::{EmbeddingClient, EmbeddingService, MockBackend};
|
||||
use crate::curate::llm::{LlmClient, MockBackend as ChatMockBackend};
|
||||
use crate::types::{Entry, ExtractMethod, SourceKind, SourceRef};
|
||||
use sqlx::Row as _;
|
||||
|
||||
@@ -1492,7 +1517,24 @@ mod tests {
|
||||
assert_eq!(thin, "{}");
|
||||
|
||||
// Admission replaces the old prefilter and carries retriever telemetry.
|
||||
let curator = Curator::new(h.config.clone(), h.db.clone(), Llms::default());
|
||||
// DeepSeek is "down": the bulk client exists but every call fails, so
|
||||
// the deep set is ranked on present signals and the editor falls back
|
||||
// to utility order (§17).
|
||||
let bulk_backend = Arc::new(ChatMockBackend::new());
|
||||
let bulk = LlmClient::with_backend(
|
||||
&h.config.deepseek.model,
|
||||
"SYSTEM".into(),
|
||||
UsageMeter::new(&h.config.deepseek, h.config.max_daily_usd),
|
||||
bulk_backend.clone(),
|
||||
);
|
||||
let curator = Curator::new(
|
||||
h.config.clone(),
|
||||
h.db.clone(),
|
||||
Llms {
|
||||
bulk: Some(bulk),
|
||||
editor: None,
|
||||
},
|
||||
);
|
||||
admit::admit(&mut features, run_date(), &h.config.curation.ranking);
|
||||
record_candidates(&ctx, &features).await.unwrap();
|
||||
let admitted = features
|
||||
@@ -1504,19 +1546,67 @@ mod tests {
|
||||
admitted.iter().copied().collect::<BTreeSet<_>>(),
|
||||
BTreeSet::from([a, b])
|
||||
);
|
||||
let candidates = features
|
||||
|
||||
let assessed = curator
|
||||
.assess(&mut features, false, None, now())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(assessed, 0, "every deep batch failed");
|
||||
assert_eq!(bulk_backend.calls(), 1, "one batch was attempted");
|
||||
assert!(features.iter().all(|c| c.assessment.deep.is_none()));
|
||||
let embeddings = features
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "admitted")
|
||||
.map(|candidate| (candidate.article.id, vec![1.0, 0.0, 0.0, 0.0]))
|
||||
.collect::<HashMap<_, _>>();
|
||||
let ranked = rank::shortlist(&mut features, &embeddings, &h.config.curation.ranking);
|
||||
assert_eq!(ranked.shortlisted, 2);
|
||||
assert_eq!(ranked.clusters, 1, "identical embeddings share a leader");
|
||||
record_candidates(&ctx, &features).await.unwrap();
|
||||
let rows = sqlx::query(
|
||||
"SELECT article_id, stage, utility, rank_utility, cluster_id, cluster_rank
|
||||
FROM candidate_runs WHERE run_id = ? AND stage = 'shortlisted'
|
||||
ORDER BY rank_utility",
|
||||
)
|
||||
.bind(h.run_id)
|
||||
.fetch_all(h.db.pool())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(rows.len(), 2, "both admitted articles were shortlisted");
|
||||
for (index, row) in rows.iter().enumerate() {
|
||||
let rank = index as i64 + 1;
|
||||
assert_eq!(row.get::<String, _>("stage"), "shortlisted");
|
||||
assert!(
|
||||
row.get::<Option<f64>, _>("utility").is_some(),
|
||||
"utility over present signals"
|
||||
);
|
||||
assert_eq!(row.get::<Option<i64>, _>("rank_utility"), Some(rank));
|
||||
assert_eq!(row.get::<Option<i64>, _>("cluster_id"), Some(1));
|
||||
assert_eq!(row.get::<Option<i64>, _>("cluster_rank"), Some(rank));
|
||||
}
|
||||
let best = rows[0].get::<i64, _>("article_id");
|
||||
|
||||
let mut candidates = features
|
||||
.iter()
|
||||
.filter(|candidate| candidate.stage == "shortlisted")
|
||||
.cloned()
|
||||
.map(Candidate::into_legacy_scored)
|
||||
.collect();
|
||||
.collect::<Vec<_>>();
|
||||
candidates.sort_by_key(|candidate| candidate.rank_utility.unwrap_or(i64::MAX));
|
||||
let lineup = curator.select(candidates, run_date()).await.unwrap();
|
||||
assert_eq!(
|
||||
bulk_backend.calls(),
|
||||
2,
|
||||
"the editor tried the bulk fallback"
|
||||
);
|
||||
let selected = lineup
|
||||
.picks
|
||||
.iter()
|
||||
.map(|p| p.article.id)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(selected.len(), 1);
|
||||
assert_eq!(
|
||||
selected[0], best,
|
||||
"without any LLM the lineup follows utility"
|
||||
);
|
||||
let not_selected = admitted
|
||||
.iter()
|
||||
.copied()
|
||||
@@ -1562,6 +1652,17 @@ mod tests {
|
||||
text.contains("stage: shortlisted · reason: not_selected"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("utility: ") && text.contains(" · rank 2"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(text.contains("cluster: 1 · rank 2"), "{text}");
|
||||
assert!(text.contains("quality absent"), "{text}");
|
||||
let misses = telemetry::explain_near_misses(&h.db, run_date(), Some(h.run_id), 5)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(misses.contains("not selected, by utility"), "{misses}");
|
||||
assert!(misses.contains("shortlisted, not_selected"), "{misses}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
+6
-9
@@ -75,22 +75,20 @@ pub struct StageCounts {
|
||||
pub rated_with_embeddings: i64,
|
||||
/// Articles with a reusable or newly produced triage assessment.
|
||||
pub triaged: i64,
|
||||
/// Articles admitted to legacy Stage A / the editor.
|
||||
/// Articles admitted to close reading.
|
||||
pub admitted: i64,
|
||||
/// First admitting retriever counts.
|
||||
pub admitted_by: BTreeMap<String, i64>,
|
||||
pub exploration_admitted: i64,
|
||||
pub exploration_selected: i64,
|
||||
/// Legacy Stage A assessments in step 4; deep assessments beginning step 5.
|
||||
/// Deep assessments, whether reused or newly produced.
|
||||
pub assessed: i64,
|
||||
/// Candidates shown to the editor (the admitted set in step 4).
|
||||
/// Candidates shown to the editor after diversification.
|
||||
pub shortlisted: i64,
|
||||
/// Compatibility count for the admitted deep set in step 4.
|
||||
/// Leader clusters formed over the deep set.
|
||||
pub clusters: i64,
|
||||
/// Admitted deep-set count retained for the colophon and runs table.
|
||||
pub candidates: i64,
|
||||
/// Articles scored by the LLM (§3.6 stage A).
|
||||
pub llm_scored: i64,
|
||||
/// Candidates left unscored after failures or a bulk-provider budget trip (§5).
|
||||
pub llm_unscored: i64,
|
||||
/// Articles in the final lineup (§3.6 stage B).
|
||||
pub selected: i64,
|
||||
/// Discussion chapters rendered (§3.7).
|
||||
@@ -306,7 +304,6 @@ mod tests {
|
||||
fn serializes_round_trip() {
|
||||
let mut r = RunReport::new("2026-08-15".parse().unwrap(), ts("2026-08-15T05:30:00Z"));
|
||||
r.counts.entries_fetched = 412;
|
||||
r.counts.llm_unscored = 3;
|
||||
r.per_feed_counts.insert("Hacker News".into(), 30);
|
||||
r.per_feed_counts.insert("Lobsters".into(), 12);
|
||||
r.timings.record("ingest", 1500);
|
||||
|
||||
+19
-60
@@ -249,22 +249,22 @@ pub fn composite_social_score(refs: &[SocialRef]) -> f64 {
|
||||
// Curation (§3.5, §3.6)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// DeepSeek stage-A output for one article (§3.6).
|
||||
/// DeepSeek's close read of one article (§12.1).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct LlmScore {
|
||||
/// 0–10.
|
||||
pub score: f64,
|
||||
pub category: String,
|
||||
/// ≤ 20 words.
|
||||
pub struct Deep {
|
||||
/// Editorial quality on the article's own terms, clamped to 0–10.
|
||||
pub quality: f64,
|
||||
/// Fit for this reader, clamped to 0–10.
|
||||
pub fit: f64,
|
||||
pub category: Option<String>,
|
||||
pub rationale: String,
|
||||
#[serde(default)]
|
||||
pub is_paywalled_guess: bool,
|
||||
pub paywalled_guess: bool,
|
||||
pub facets: Facets,
|
||||
pub model: String,
|
||||
pub prompt_version: i64,
|
||||
pub assessed_at: Timestamp,
|
||||
}
|
||||
|
||||
/// Deep assessment output. Step 5 replaces the legacy stage-A producer while
|
||||
/// keeping its shape compatible for this transition step.
|
||||
pub type Deep = LlmScore;
|
||||
|
||||
/// Personalized first-pass judgment cached in `article_assessments` (§10).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Triage {
|
||||
@@ -282,7 +282,6 @@ pub struct Triage {
|
||||
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Assessment {
|
||||
pub triage: Option<Triage>,
|
||||
/// Filled in by step 5.
|
||||
pub deep: Option<Deep>,
|
||||
}
|
||||
|
||||
@@ -294,10 +293,10 @@ pub struct Candidate {
|
||||
pub exploration: bool,
|
||||
pub signals: crate::curate::signals::Signals,
|
||||
pub assessment: Assessment,
|
||||
/// Filled in by step 5.
|
||||
pub utility: Option<f64>,
|
||||
/// Filled in by step 5.
|
||||
pub rank_utility: Option<i64>,
|
||||
pub cluster: Option<i64>,
|
||||
pub cluster_rank: Option<i64>,
|
||||
pub admitted_by: Vec<String>,
|
||||
pub stage: String,
|
||||
pub excluded_reason: Option<String>,
|
||||
@@ -313,55 +312,14 @@ impl Candidate {
|
||||
signals,
|
||||
assessment: Assessment::default(),
|
||||
utility: None,
|
||||
rank_utility: None,
|
||||
cluster: None,
|
||||
cluster_rank: None,
|
||||
admitted_by: Vec::new(),
|
||||
stage: "eligible".into(),
|
||||
excluded_reason: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Adapter retained until step 5 retires stage A and `ScoredArticle`.
|
||||
pub fn into_legacy_scored(self) -> ScoredArticle {
|
||||
ScoredArticle {
|
||||
prefilter_score: self.signals.preliminary.unwrap_or(0.0),
|
||||
social_score: self.signals.social.unwrap_or(0.0),
|
||||
llm: self.assessment.deep,
|
||||
triage: self.assessment.triage,
|
||||
auto_include: self.auto_include,
|
||||
exploration: self.exploration,
|
||||
admitted_by: self.admitted_by,
|
||||
article: self.article,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// An article carrying every ranking signal computed so far (§3.5, §3.6).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct ScoredArticle {
|
||||
pub article: Article,
|
||||
/// Heuristic pre-filter score, 0–100 (§3.5).
|
||||
pub prefilter_score: f64,
|
||||
/// Cached [`composite_social_score`] for the article.
|
||||
pub social_score: f64,
|
||||
/// `None` until stage A has run (or when `--skip-llm`).
|
||||
pub llm: Option<LlmScore>,
|
||||
/// Transitional metadata rendered by the editor until step 5 removes this type.
|
||||
#[serde(default)]
|
||||
pub triage: Option<Triage>,
|
||||
/// From `curation.always_include_feeds`: may be scored but never dropped (§3.5).
|
||||
pub auto_include: bool,
|
||||
#[serde(default)]
|
||||
pub exploration: bool,
|
||||
#[serde(default)]
|
||||
pub admitted_by: Vec<String>,
|
||||
}
|
||||
|
||||
impl ScoredArticle {
|
||||
/// Ranking key for stage B: LLM score weighted with social proof (§3.6).
|
||||
pub fn combined_score(&self) -> f64 {
|
||||
let llm = self.llm.as_ref().map(|l| l.score).unwrap_or(0.0);
|
||||
llm * 10.0 + self.social_score * 4.0 + self.prefilter_score * 0.1
|
||||
}
|
||||
}
|
||||
|
||||
/// One selected article with its section placement (§3.6 stage B).
|
||||
@@ -377,7 +335,7 @@ pub struct Pick {
|
||||
pub why: Option<String>,
|
||||
/// Newspaper-abstract summary from stage C; `None` until editorial runs.
|
||||
pub summary: Option<String>,
|
||||
pub llm: Option<LlmScore>,
|
||||
pub llm: Option<Deep>,
|
||||
/// Rendered comment chapter, when the article had social refs (§3.7).
|
||||
pub discussion: Option<Discussion>,
|
||||
}
|
||||
@@ -706,7 +664,8 @@ pub struct RatingEvent {
|
||||
pub event_at: Timestamp,
|
||||
}
|
||||
|
||||
/// Descriptive deep-assessment facets (§12.1), populated beginning in step 5.
|
||||
/// Descriptive deep-assessment facets (§12.1); shown to the editor, the profile
|
||||
/// rebuild and `explain`, never a numeric ranking signal.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Facets {
|
||||
pub format: Option<String>,
|
||||
|
||||
+28
-17
@@ -18,7 +18,7 @@
|
||||
//! no article in the fixtures carries an image, so the EPUB builder's image
|
||||
//! downloader has nothing to fetch.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::path::Path;
|
||||
|
||||
use jiff::Timestamp;
|
||||
@@ -26,7 +26,7 @@ use jiff::civil::Date;
|
||||
|
||||
use daily_epub::config::{Config, PublishConfig, ServerConfig, XtcConfig};
|
||||
use daily_epub::curate::llm::{LlmClient, Llms, MockBackend, UsageMeter};
|
||||
use daily_epub::curate::{Curator, admit, editorial};
|
||||
use daily_epub::curate::{Curator, admit, editorial, rank};
|
||||
use daily_epub::db::Db;
|
||||
use daily_epub::extract::Extractor;
|
||||
use daily_epub::types::{
|
||||
@@ -412,7 +412,6 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
|
||||
let candidates = personalized
|
||||
.into_iter()
|
||||
.filter(|candidate| candidate.stage == "admitted")
|
||||
.map(Candidate::into_legacy_scored)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(candidates.len(), 5, "nothing is dropped at this volume");
|
||||
// The excerpt-only story is penalized (§3.5).
|
||||
@@ -421,10 +420,10 @@ async fn skip_llm_pipeline_produces_a_published_issue() {
|
||||
.find(|c| c.article.excerpt_only)
|
||||
.expect("the allocator teaser survived");
|
||||
assert!(
|
||||
allocator.prefilter_score
|
||||
allocator.signals.heuristic.unwrap_or_default()
|
||||
< candidates
|
||||
.iter()
|
||||
.map(|candidate| candidate.prefilter_score)
|
||||
.filter_map(|candidate| candidate.signals.heuristic)
|
||||
.fold(f64::NEG_INFINITY, f64::max)
|
||||
);
|
||||
|
||||
@@ -516,13 +515,12 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
|
||||
let candidates = personalized
|
||||
.into_iter()
|
||||
.filter(|candidate| candidate.stage == "admitted")
|
||||
.map(Candidate::into_legacy_scored)
|
||||
.collect::<Vec<_>>();
|
||||
let ids: Vec<i64> = candidates.iter().map(|c| c.article.id).collect();
|
||||
assert_eq!(ids.len(), 5);
|
||||
|
||||
// --- Script DeepSeek: one stage-A batch, one stage-B call, five stage-C
|
||||
// summaries and one front page (§3.6). ---
|
||||
// --- Script DeepSeek: one deep-assessment batch, one editor call, five
|
||||
// summaries and one brief. ---
|
||||
let backend = std::sync::Arc::new(MockBackend::new());
|
||||
let usage = daily_epub::types::TokenUsage {
|
||||
input_tokens: 1000,
|
||||
@@ -535,8 +533,8 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
|
||||
.enumerate()
|
||||
.map(|(i, id)| {
|
||||
format!(
|
||||
r#"{{"id": {id}, "score": {}, "category": "Tech & Engineering",
|
||||
"rationale": "solid systems writeup", "is_paywalled_guess": false}}"#,
|
||||
r#"{{"id": {id}, "quality": {}, "fit": 7, "category": "Tech & Engineering",
|
||||
"rationale": "solid systems writeup", "paywalled_guess": false, "facets": {{"format":"analysis_essay"}}}}"#,
|
||||
9 - i
|
||||
)
|
||||
})
|
||||
@@ -590,12 +588,12 @@ async fn llm_pipeline_runs_against_a_mock_backend() {
|
||||
|
||||
let mut candidates = candidates;
|
||||
curator
|
||||
.score(&mut candidates, date())
|
||||
.assess(&mut candidates, true, None, jiff::Timestamp::now())
|
||||
.await
|
||||
.expect("stage A");
|
||||
.expect("deep assessment");
|
||||
assert!(
|
||||
candidates.iter().all(|c| c.llm.is_some()),
|
||||
"every candidate came back scored"
|
||||
candidates.iter().all(|c| c.assessment.deep.is_some()),
|
||||
"every candidate came back assessed"
|
||||
);
|
||||
|
||||
let lineup = curator.select(candidates, date()).await.expect("stage B");
|
||||
@@ -680,7 +678,6 @@ async fn failing_deepseek_still_publishes_with_heuristic_fallbacks() {
|
||||
let mut candidates = personalized
|
||||
.into_iter()
|
||||
.filter(|candidate| candidate.stage == "admitted")
|
||||
.map(Candidate::into_legacy_scored)
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let backend = std::sync::Arc::new(MockBackend::new());
|
||||
@@ -699,10 +696,24 @@ async fn failing_deepseek_still_publishes_with_heuristic_fallbacks() {
|
||||
},
|
||||
);
|
||||
curator
|
||||
.score(&mut candidates, date())
|
||||
.assess(&mut candidates, true, None, jiff::Timestamp::now())
|
||||
.await
|
||||
.expect("failed batches degrade, not abort");
|
||||
assert!(candidates.iter().all(|candidate| candidate.llm.is_none()));
|
||||
assert!(
|
||||
candidates
|
||||
.iter()
|
||||
.all(|candidate| candidate.assessment.deep.is_none())
|
||||
);
|
||||
// Utility over the present signals, clustered as singletons without
|
||||
// embeddings: every admitted article is shortlisted with a cluster id.
|
||||
let ranked = rank::shortlist(&mut candidates, &HashMap::new(), &cfg.curation.ranking);
|
||||
assert_eq!(ranked.shortlisted, candidates.len());
|
||||
assert_eq!(ranked.clusters, candidates.len());
|
||||
for candidate in &candidates {
|
||||
assert_eq!(candidate.stage, "shortlisted");
|
||||
assert!(candidate.utility.is_some() && candidate.cluster.is_some());
|
||||
assert!(candidate.rank_utility.is_some());
|
||||
}
|
||||
let mut lineup = curator
|
||||
.select(candidates, date())
|
||||
.await
|
||||
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"articles": [
|
||||
{
|
||||
"id": 101,
|
||||
"quality": 8.5,
|
||||
"fit": 7.0,
|
||||
"category": "Tech & Engineering",
|
||||
"rationale": "First-hand 40TB Postgres migration with failure timeline, numbers and a rollback plan",
|
||||
"paywalled_guess": false,
|
||||
"facets": {
|
||||
"format": "first_hand_account",
|
||||
"depth": "deep",
|
||||
"evidence": "first_hand",
|
||||
"commerciality": "none",
|
||||
"topic_group": "software_engineering",
|
||||
"technicality": "advanced",
|
||||
"locality": "not_applicable",
|
||||
"specific_topics": ["Postgres migration", "storage failover", "rollback planning"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 102,
|
||||
"quality": 3.0,
|
||||
"fit": 2.5,
|
||||
"category": "AI & Machine Learning",
|
||||
"rationale": "Model release announcement, benchmark table from the vendor, no independent evaluation",
|
||||
"paywalled_guess": false,
|
||||
"facets": {
|
||||
"format": "announcement_roundup",
|
||||
"depth": "brief",
|
||||
"evidence": "speculative",
|
||||
"commerciality": "promotional",
|
||||
"topic_group": "ai_ml",
|
||||
"technicality": "light",
|
||||
"locality": "international",
|
||||
"specific_topics": ["model launch"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 103,
|
||||
"quality": 6.5,
|
||||
"fit": 8.0,
|
||||
"category": "Boston & Local",
|
||||
"rationale": "MBTA slow-zone data pulled from the tracker and charted by line with original analysis",
|
||||
"paywalled_guess": false,
|
||||
"facets": {
|
||||
"format": "analysis_essay",
|
||||
"depth": "standard",
|
||||
"evidence": "data_or_experiment",
|
||||
"commerciality": "none",
|
||||
"topic_group": "boston_new_england",
|
||||
"technicality": "intermediate",
|
||||
"locality": "boston_new_england",
|
||||
"specific_topics": ["MBTA slow zones", "transit data"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 104,
|
||||
"quality": 5.0,
|
||||
"fit": 6.0,
|
||||
"category": "Culture & Essays",
|
||||
"rationale": "Promising essay on typesetting history; the body reads cut off after the second section",
|
||||
"paywalled_guess": true,
|
||||
"facets": {
|
||||
"format": "analysis_essay",
|
||||
"depth": "standard",
|
||||
"evidence": "synthesis",
|
||||
"commerciality": "none",
|
||||
"topic_group": "books_writing",
|
||||
"technicality": "nontechnical",
|
||||
"locality": "not_applicable",
|
||||
"specific_topics": ["typesetting", "print history"]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"articles": [
|
||||
{
|
||||
"id": 201,
|
||||
"quality": 7.0,
|
||||
"fit": 6.5,
|
||||
"category": "Science & Space",
|
||||
"rationale": "careful write-up of an amateur radio occultation measurement",
|
||||
"paywalled_guess": false,
|
||||
"facets": {
|
||||
"format": "first_hand_account",
|
||||
"depth": "standard",
|
||||
"evidence": "data_or_experiment",
|
||||
"commerciality": "none",
|
||||
"topic_group": "science_space",
|
||||
"technicality": "intermediate",
|
||||
"locality": "us",
|
||||
"specific_topics": ["radio occultation"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "202",
|
||||
"quality": "6",
|
||||
"fit": "5.5",
|
||||
"category": "Niche Corner",
|
||||
"rationale": "mailing-list argument about tape drives, oddly gripping",
|
||||
"paywalled_guess": "false",
|
||||
"facets": {
|
||||
"format": "discussion_thread",
|
||||
"depth": "standard",
|
||||
"evidence": "anecdote",
|
||||
"commerciality": "none",
|
||||
"topic_group": "retro_computing",
|
||||
"technicality": "intermediate",
|
||||
"locality": "not_applicable",
|
||||
"specific_topics": ["LTO tape", "backups", "archival", "vendors", "pricing"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 203,
|
||||
"quality": 4,
|
||||
"fit": 4
|
||||
},
|
||||
{
|
||||
"id": 204,
|
||||
"quality": 12.5,
|
||||
"fit": -1,
|
||||
"category": "Sports",
|
||||
"rationale": "model ignored the rubric ceiling and invented a section here",
|
||||
"paywalled_guess": false,
|
||||
"facets": "not an object"
|
||||
},
|
||||
{
|
||||
"id": 205,
|
||||
"quality": 8.0,
|
||||
"category": "Top Stories",
|
||||
"rationale": "fit is missing, so this item is unusable"
|
||||
},
|
||||
{
|
||||
"quality": 9.0,
|
||||
"fit": 9.0,
|
||||
"category": "Top Stories",
|
||||
"rationale": "no id at all, unusable"
|
||||
},
|
||||
"a bare string where an object belongs"
|
||||
]
|
||||
}
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"articles": [
|
||||
{
|
||||
"id": 101,
|
||||
"score": 8.5,
|
||||
"category": "Tech & Engineering",
|
||||
"rationale": "first-hand 40TB Postgres migration with numbers, failures and rollback plan",
|
||||
"is_paywalled_guess": false
|
||||
},
|
||||
{
|
||||
"id": 102,
|
||||
"score": 3.0,
|
||||
"category": "AI & Machine Learning",
|
||||
"rationale": "model release announcement, no independent evaluation",
|
||||
"is_paywalled_guess": false
|
||||
},
|
||||
{
|
||||
"id": 103,
|
||||
"score": 6.5,
|
||||
"category": "Boston & Local",
|
||||
"rationale": "MBTA slow-zone data analysis with original charts",
|
||||
"is_paywalled_guess": false
|
||||
},
|
||||
{
|
||||
"id": 104,
|
||||
"score": 5.0,
|
||||
"category": "Culture & Essays",
|
||||
"rationale": "promising essay on typesetting, body appears truncated",
|
||||
"is_paywalled_guess": true
|
||||
}
|
||||
]
|
||||
}
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"articles": [
|
||||
{
|
||||
"id": 201,
|
||||
"score": 7.0,
|
||||
"category": "Science & Space",
|
||||
"rationale": "careful write-up of an amateur radio occultation measurement",
|
||||
"is_paywalled_guess": false
|
||||
},
|
||||
{
|
||||
"id": "202",
|
||||
"score": "6",
|
||||
"category": "Niche Corner",
|
||||
"rationale": "mailing-list argument about tape drives, oddly gripping",
|
||||
"is_paywalled_guess": "false"
|
||||
},
|
||||
{
|
||||
"id": 203,
|
||||
"score": 4
|
||||
},
|
||||
{
|
||||
"id": 204,
|
||||
"score": 12.5,
|
||||
"category": "Top Stories",
|
||||
"rationale": "model ignored the rubric ceiling here",
|
||||
"is_paywalled_guess": false
|
||||
},
|
||||
{
|
||||
"score": 9.0,
|
||||
"category": "Top Stories",
|
||||
"rationale": "no id at all, unusable"
|
||||
},
|
||||
"a bare string where an object belongs"
|
||||
]
|
||||
}
|
||||
+54
-38
@@ -3,8 +3,8 @@
|
||||
//! The stage logic is unit-tested inside `src/curate/*`. What this file guards is
|
||||
//! the contract *between* the curation stages and everything around them:
|
||||
//!
|
||||
//! * the recorded DeepSeek fixtures still parse through the real
|
||||
//! `score.rs` / `select.rs` / `editorial.rs` parsers into the structures the
|
||||
//! * recorded DeepSeek fixtures still parse through the real
|
||||
//! `assess.rs` / `editor.rs` / `editorial.rs` parsers into the structures the
|
||||
//! pipeline consumes, and the lenient parsers still cope with the messy one;
|
||||
//! * the shipped Scour OPML still yields the ~220 interests the taste profile is
|
||||
//! assembled from (§3.6a);
|
||||
@@ -17,11 +17,10 @@ use std::collections::BTreeSet;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
|
||||
use daily_epub::curate::assess::parse_deep_response;
|
||||
use daily_epub::curate::editor::parse_selection_response;
|
||||
use daily_epub::curate::editorial::BriefResponse;
|
||||
use daily_epub::curate::profile;
|
||||
use daily_epub::curate::score::parse_score_response;
|
||||
use daily_epub::curate::select::parse_selection_response;
|
||||
use daily_epub::types::LlmScore;
|
||||
|
||||
fn repo(rel: &str) -> PathBuf {
|
||||
Path::new(env!("CARGO_MANIFEST_DIR")).join(rel)
|
||||
@@ -32,65 +31,82 @@ fn fixture(name: &str) -> String {
|
||||
std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("reading {}: {e}", path.display()))
|
||||
}
|
||||
|
||||
/// Stage A responses must parse into `{id, score, category, rationale,
|
||||
/// is_paywalled_guess}` per article (§3.6).
|
||||
/// Deep responses must parse into `{id, quality, fit, category, rationale,
|
||||
/// paywalled_guess, facets}` per article (§12.1).
|
||||
#[test]
|
||||
fn stage_a_fixture_parses_into_scores() {
|
||||
let items = parse_score_response(&fixture("deepseek_score_batch.json"));
|
||||
fn deep_fixture_parses_into_assessments() {
|
||||
let sections = daily_epub::config::CurationConfig::default().sections;
|
||||
let items = parse_deep_response(&fixture("deepseek_deep_batch.json"), §ions);
|
||||
assert!(items.len() >= 4, "fixture should cover a realistic batch");
|
||||
|
||||
for item in &items {
|
||||
assert!(item.id > 0, "every item carries a positive article id");
|
||||
assert!((0.0..=10.0).contains(&item.quality));
|
||||
assert!((0.0..=10.0).contains(&item.fit));
|
||||
assert!(item.category.is_some(), "every category is on the palette");
|
||||
assert!(
|
||||
(0.0..=10.0).contains(&item.score),
|
||||
"score {} out of range",
|
||||
item.score
|
||||
);
|
||||
assert!(!item.category.is_empty());
|
||||
assert!(
|
||||
item.rationale.split_whitespace().count() <= 20,
|
||||
"rationale must stay under 20 words: {:?}",
|
||||
item.rationale.split_whitespace().count() <= 25,
|
||||
"rationale must stay under 25 words: {:?}",
|
||||
item.rationale
|
||||
);
|
||||
assert!(item.facets.format.is_some() && item.facets.topic_group.is_some());
|
||||
}
|
||||
assert!(
|
||||
items.iter().any(|i| i.is_paywalled_guess),
|
||||
items.iter().any(|item| item.paywalled_guess),
|
||||
"the fixture should exercise the paywall flag"
|
||||
);
|
||||
// Quality and fit are judged separately: the fixture has an article whose
|
||||
// fit exceeds its quality and one the other way round.
|
||||
assert!(items.iter().any(|item| item.fit > item.quality));
|
||||
assert!(items.iter().any(|item| item.quality > item.fit));
|
||||
|
||||
// The batch spans the rubric rather than clustering at one score.
|
||||
let scores: Vec<f64> = items.iter().map(|i| i.score).collect();
|
||||
let spread = scores.iter().cloned().fold(f64::MIN, f64::max)
|
||||
- scores.iter().cloned().fold(f64::MAX, f64::min);
|
||||
let qualities: Vec<f64> = items.iter().map(|i| i.quality).collect();
|
||||
let spread = qualities.iter().cloned().fold(f64::MIN, f64::max)
|
||||
- qualities.iter().cloned().fold(f64::MAX, f64::min);
|
||||
assert!(spread >= 3.0, "fixture scores are too uniform to be useful");
|
||||
|
||||
// Every item converts into the shared curation type.
|
||||
let converted: Vec<LlmScore> = items.into_iter().map(LlmScore::from).collect();
|
||||
assert!(converted.iter().all(|s| (0.0..=10.0).contains(&s.score)));
|
||||
}
|
||||
|
||||
/// The messy fixture must stay messy: it is what proves the parser is lenient
|
||||
/// (string ids, string scores, out-of-range scores, junk entries).
|
||||
/// (string ids and scores, out-of-range scores, unknown facet tokens, junk).
|
||||
#[test]
|
||||
fn stage_a_messy_fixture_is_salvaged_not_rejected() {
|
||||
let raw = fixture("deepseek_score_batch_messy.json");
|
||||
// The hard cases are still present in the recording…
|
||||
fn deep_messy_fixture_is_salvaged_not_rejected() {
|
||||
let sections = daily_epub::config::CurationConfig::default().sections;
|
||||
let raw = fixture("deepseek_deep_batch_messy.json");
|
||||
assert!(raw.contains("\"id\": \""), "needs a string id");
|
||||
assert!(raw.contains("\"score\": \""), "needs a string score");
|
||||
assert!(raw.contains("\"quality\": \""), "needs a string score");
|
||||
assert!(
|
||||
raw.contains("discussion_thread"),
|
||||
"needs an unknown facet token"
|
||||
);
|
||||
|
||||
// …and the real parser copes with all of them.
|
||||
let items = parse_score_response(&raw);
|
||||
let items = parse_deep_response(&raw, §ions);
|
||||
assert!(!items.is_empty(), "the parser salvaged nothing");
|
||||
assert!(
|
||||
items.iter().all(|i| (0.0..=10.0).contains(&i.score)),
|
||||
"out-of-range scores must be clamped: {:?}",
|
||||
items.iter().map(|i| i.score).collect::<Vec<_>>()
|
||||
items
|
||||
.iter()
|
||||
.all(|i| (0.0..=10.0).contains(&i.quality) && (0.0..=10.0).contains(&i.fit)),
|
||||
"out-of-range scores must be clamped"
|
||||
);
|
||||
assert!(items.iter().all(|i| i.id > 0), "id-less items are skipped");
|
||||
let messy = items
|
||||
.iter()
|
||||
.find(|i| i.id == 202)
|
||||
.expect("string-valued item");
|
||||
assert!(
|
||||
messy.facets.format.is_none(),
|
||||
"unknown facet tokens become None"
|
||||
);
|
||||
assert_eq!(messy.facets.depth.as_deref(), Some("standard"));
|
||||
let off_palette = items.iter().find(|i| i.id == 204).expect("clamped item");
|
||||
assert!(
|
||||
off_palette.category.is_none(),
|
||||
"invented sections become None"
|
||||
);
|
||||
|
||||
// A response that is not JSON at all degrades to "no scores", never a panic.
|
||||
assert!(parse_score_response("I'm sorry, I can't do that.").is_empty());
|
||||
assert!(parse_score_response("").is_empty());
|
||||
// A response that is not JSON at all degrades to "no assessments", never a panic.
|
||||
assert!(parse_deep_response("I'm sorry, I can't do that.", §ions).is_empty());
|
||||
assert!(parse_deep_response("", §ions).is_empty());
|
||||
}
|
||||
|
||||
/// Stage B responses must carry `{id, section, position, lead_story}` with
|
||||
|
||||
Reference in New Issue
Block a user