Stop routine excerpt fallbacks from degrading every run

Every issue since 2026-08-15 landed as `degraded`, because the extraction
stage warned on any fetch failure at all and a few always fail: the worst
of the 21 published days was 15% (33 of 220 articles), the median under
11%. A status every run carries says nothing, so warn only past a 30%
share -- roughly twice the worst day seen -- and log the rest. The exact
count was already in `counts.excerpt_only` either way.

`rebuild()` also stored the bumped profile version before reading the
interests OPML, so a rebuild that failed on a missing file would mark the
profile fresh for another week having never rewritten its text. Read the
prompt inputs first, ahead of both the model call and the writes, so a
failure stays due and costs nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0164rQrMUZkf7uV1VGYuCFy4
This commit is contained in:
2026-09-05 04:08:06 +00:00
co-authored by Claude Opus 5
parent 17a88043ff
commit 6c2d74c734
2 changed files with 68 additions and 10 deletions
+22 -4
View File
@@ -346,6 +346,11 @@ struct StageContext<'a> {
rescore: bool,
}
/// Share of the day's articles that must fall back to a feed excerpt before the
/// run is degraded over it. Single-digit percentages are routine; the exact count
/// is always in `counts.excerpt_only`.
const EXCERPT_FALLBACK_WARN_SHARE: f64 = 0.30;
async fn run_stages(
ctx: &StageContext<'_>,
window_start: Timestamp,
@@ -412,11 +417,24 @@ async fn run_stages(
let extract_stats = extractor.extract_all(&mut articles).await;
report.counts.extracted = (extract_stats.from_miniflux + extract_stats.from_readability) as i64;
report.counts.excerpt_only = extract_stats.excerpt_only as i64;
// A handful of fetch failures is the normal state of the open web, so only a
// day well past the usual rate is worth degrading the run over.
if extract_stats.fetch_failures > 0 {
report.warn(format!(
"{} articles fell back to a feed excerpt",
extract_stats.fetch_failures
));
let share = extract_stats.fetch_failures as f64 / articles.len().max(1) as f64;
if share >= EXCERPT_FALLBACK_WARN_SHARE {
report.warn(format!(
"{} of {} articles ({:.0}%) fell back to a feed excerpt",
extract_stats.fetch_failures,
articles.len(),
share * 100.0
));
} else {
tracing::info!(
failures = extract_stats.fetch_failures,
articles = articles.len(),
"articles fell back to a feed excerpt"
);
}
}
report.timings.record("extract", elapsed_ms(stage));