Curation v2 step 4: triage replaces the gate

DeepSeek triage over every eligible article (plan §10) with cached
assessments in article_assessments, the union admission with quotas and
exploration slots (§11), hygiene moved to admit.rs with the churn rule
reading assessments, prefilter.rs reduced to hygiene and text heuristic,
prefilter_keep removed in favour of curation.ranking.deep_keep, the
scores table dropped (migration 0003), and --rescore on generate.

Implemented by Codex (gpt-5.4, high effort) from
docs/plans/curation-v2-briefs/step4.md; reviewed against plan §10–§11.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
This commit is contained in:
thalladaandClaude Fable 5.1 committed 2026-09-02 15:25:04 +00:00
1 parent 76527e3821
commit 10f4afd091
18 files changed
+1927 -816

No files matched your search

+8 -55
View File
@@ -1,15 +1,16 @@
//! Curation pipeline: pre-filter → LLM scoring → selection → editorial (spec §3.5, §3.6).
//! Personalized curation: signals → triage → admission → assessment → editor.
//!
//! ```text
//! ~400 articles ─prefilter─▶ ~120 candidates ─stage A─▶ scored ─stage B─▶ lineup ─stage C─▶ editorial
//! ~400 eligible ─triage─▶ union admission (120) ─stage A─▶ editor ─▶ editorial
//! ```
//!
//! [`Curator`] is the thin orchestration layer the `generate` pipeline calls; the
//! interesting logic lives in the stage modules. Every stage is safe to run with
//! no provider at all (`--skip-llm`): the prefilter order stands in for selection
//! no provider at all (`--skip-llm`): the cheap-signal blend stands in for selection
//! and feed excerpts stand in for summaries (notes §6). Scoring runs on the bulk
//! client; selection and editorial on the editor with per-call bulk fallback.
pub mod admit;
pub mod editorial;
pub mod embedding;
pub mod llm;
@@ -19,12 +20,13 @@ pub mod score;
pub mod select;
pub mod signals;
pub mod telemetry;
pub mod triage;
use jiff::civil::Date;
use crate::config::Config;
use crate::db::Db;
use crate::types::{Article, Editorial, Lineup, ScoredArticle};
use crate::types::{Editorial, Lineup, ScoredArticle};
/// Runs the three curation stages against one day's articles (§3.5, §3.6).
pub struct Curator {
@@ -34,52 +36,17 @@ pub struct Curator {
}
impl Curator {
/// An empty [`llm::Llms`] corresponds to `--skip-llm`: prefilter order is
/// An empty [`llm::Llms`] corresponds to `--skip-llm`: cheap-signal order is
/// used for selection and feed excerpts stand in for summaries (notes §6).
/// With only `bulk`, every editor call runs on DeepSeek (§4.2).
pub fn new(config: Config, db: Db, llms: llm::Llms) -> Self {
Self { config, db, llms }
}
/// Heuristic pre-filter: 300–500 articles → `prefilter_keep` (§3.5).
///
/// Also persists each candidate's `prefilter_score` for the day so that a
/// re-run of the same date is idempotent (notes §12).
pub async fn prefilter(
&self,
articles: Vec<Article>,
date: Date,
) -> anyhow::Result<Vec<ScoredArticle>> {
let span = tracing::info_span!("prefilter", articles = articles.len());
let _guard = span.enter();
let ctx = prefilter::PrefilterContext::load(&self.db, date).await?;
let candidates = prefilter::run(articles, &ctx, &self.config);
for candidate in &candidates {
if candidate.article.id == 0 {
continue; // not persisted yet (dry run over synthetic articles)
}
if let Err(e) = self
.db
.upsert_score(
candidate.article.id,
date,
Some(candidate.prefilter_score),
None,
)
.await
{
tracing::warn!(article_id = candidate.article.id, error = %e,
"could not persist the prefilter score");
}
}
Ok(candidates)
}
/// Stage A: batched LLM scoring of the surviving candidates (§3.6).
///
/// A no-op under `--skip-llm`. Scores are persisted per `(article, date)`.
pub async fn score(&self, candidates: &mut [ScoredArticle], date: Date) -> anyhow::Result<()> {
pub async fn score(&self, candidates: &mut [ScoredArticle], _date: Date) -> anyhow::Result<()> {
let Some(llm) = self.llms.bulk.as_ref() else {
tracing::info!("--skip-llm: stage A scoring skipped");
return Ok(());
@@ -98,20 +65,6 @@ impl Curator {
.await?;
tracing::info!(scored, total = candidates.len(), "stage A complete");
for candidate in candidates.iter() {
if candidate.article.id == 0 {
continue;
}
if let Some(llm_score) = candidate.llm.as_ref()
&& let Err(e) = self
.db
.upsert_score(candidate.article.id, date, None, Some(llm_score))
.await
{
tracing::warn!(article_id = candidate.article.id, error = %e,
"could not persist the llm score");
}
}
Ok(())
}