thalladaandClaude Fable 5.1
76527e3821
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1rCLQeKBgnBo3oTgHuTMe
104 lines
9.1 KiB
Markdown
104 lines
9.1 KiB
Markdown
|
||
---
|
||
|
||
# YOUR STEP: 3 — Embeddings, signals, telemetry (plan §21 step 3)
|
||
|
||
You are working in a git **worktree** on branch `curation-v2-step3`, which branches from the step 1
|
||
commit. Another agent is implementing step 2 (the Claude editor) in parallel on the main checkout;
|
||
you will not see its changes and must not need them. Read `git log -3` and the current
|
||
`src/curate/`, `src/pipeline.rs`, `src/config.rs`, `src/db.rs`, `src/report.rs`, `src/main.rs` first.
|
||
Do not run `git checkout`, `git stash`, `git worktree` or `git commit`.
|
||
|
||
Scope (plan §4.3, §7.1, §7.2, §7.4, §7.5, §9, §12.2 normalization only, §12.4 preliminary blend,
|
||
§15.2 `explain`, §15.4 log line parts, §16, §17 Voyage row, §19 `[voyage]` + `[curation.ranking]`).
|
||
The old heuristic prefilter still gates in this step; signals are computed for every article that
|
||
passes hygiene and persisted, nothing about *selection* changes yet.
|
||
|
||
1. **`src/curate/embedding.rs`** (§4.3, §7.1, §7.2): `EmbeddingBackend` trait mirroring `ChatBackend`
|
||
(with a mock for tests), `VoyageBackend` (`POST {base_url}/embeddings`, `Authorization: Bearer`,
|
||
body `{input, model, input_type: "document"|"query", truncation: true, output_dimension, output_dtype: "float"}`,
|
||
response mapped by index and length-checked, usage `total_tokens` metered), batching by `batch_size`
|
||
with `buffer_unordered(max_concurrent_requests)`, retry 429/5xx via `RetryPolicy`, f32
|
||
little-endian BLOB encode/decode with length and finiteness checks, `dot(a, b)` with a dimension
|
||
check, and cache orchestration: `article_embeddings` keyed by `(model, dimension, input_hash = sha256
|
||
of the embedded text)`, overwritten on change. Embedded text = `"Title: {title}\n\n{plain body}"`
|
||
via `curate::prompt_text`, whitespace collapsed, cut at `max_input_chars` on a char boundary —
|
||
**no feed name, author or scores**. Interests go to `interest_embeddings` with `input_type = "query"`
|
||
and the bare interest string. A failed batch leaves those articles without embeddings; never fatal.
|
||
A Voyage `UsageMeter` with `price_per_mtok = 0.02` and `max_daily_usd` (0.50) as a runaway guard.
|
||
Key only from `DAILY_EPUB_VOYAGE__API_KEY`. Raw vectors never reach logs or reports.
|
||
2. **Config** (§4.3, §19): `[voyage]` (`enabled`, `base_url = "https://api.voyageai.com/v1"`,
|
||
`model = "voyage-4-lite"`, `output_dimension = 512`, `batch_size = 32`, `max_concurrent_requests = 4`,
|
||
`max_input_chars = 60000`, `max_daily_usd = 0.50`) and the complete `[curation.ranking]` block from
|
||
§19 (`triage_max`, `deep_keep`, `shortlist_keep`, `assessment_reuse_days`, `rating_lookback_days`,
|
||
`rating_half_life_days`, `neighbour_k`, `negative_coefficient`, `knn_floor/full`, `feed_floor/full`,
|
||
`semantic_min_words`, `exploration_slots`, `embedding_retention_days`, `telemetry_retention_days`,
|
||
`[curation.ranking.quotas]`, `[curation.ranking.weights.preliminary]`, `[curation.ranking.weights.utility]`,
|
||
`[curation.ranking.diversity]`) with the plan's defaults and the validation rules in §19 (weights
|
||
non-negative; `deep_keep ≥ shortlist_keep ≥ target_article_count`; `*_full > *_floor ≥ 0`;
|
||
`0 ≤ cluster_threshold ≤ 1`; `per_cluster_cap ≥ 1`; batch sizes ≥ 1; dimension ∈ {256,512,1024,2048}).
|
||
Steps 4–5 will *use* the ranking keys; you only add and validate them. Keep `prefilter_keep` for now.
|
||
Startup logs whether Voyage is enabled. Update `config.example.toml` and the README.
|
||
3. **`src/curate/signals.rs`** (§9): a `Signals` type where every signal is `Option<f64>` (absent ≠ 0):
|
||
`interest` (§9.1 z-scored interest match, top-three interests recorded; raw top-1 cosine fallback
|
||
under 30 embedded articles, logged), `knn` (§9.2 preference state from `db::current_ratings(rating_lookback_days)`
|
||
joined to `article_embeddings`, decayed weights `value × 0.5^(age/half_life)`, top-k positives and
|
||
negatives, `knn = pos − negative_coefficient × neg`, top-three neighbours recorded, gate ramp
|
||
`clamp((n − knn_floor)/(knn_full − knn_floor), 0, 1)` multiplying the weight, absent at 0),
|
||
`feed` (§9.3 Beta-smoothed decayed rate credited to distinct direct `SourceKind::Feed` feeds, mean
|
||
over the article's rated feeds, gate `feed_floor/feed_full`), `social` (existing composite; absent
|
||
with no rows), `heuristic` (`longform_points(word_count)` − excerpt-only − roundup penalty, from
|
||
`prefilter.rs` **without** the social, Scour/HN, multi-source terms; expose those pieces as
|
||
functions). Log once per run:
|
||
`preference: N rated articles with embeddings → knn gate X; feed gate Y (n=…)`.
|
||
Also implement the **mid-rank percentile normalizer** of §12.2 and the **preliminary blend** of
|
||
§12.4 (present-and-active weights renormalized) so `signals_json.norm`/`weights` can be written now.
|
||
4. **Pipeline**: after social enrichment, for every article that passes hygiene, compute embeddings
|
||
(cached), signals, and the preliminary blend, then run the old prefilter as today. New stage
|
||
timings `embed` and `signals`. `--skip-embeddings` uses cached embeddings only (zero Voyage calls);
|
||
`--skip-llm` unchanged. `StageCounts` gains `eligible`, `embedded`, `rated_with_embeddings`, and
|
||
the report/`print_report` show them. The paper must still build when Voyage is down or unconfigured
|
||
(§17): `interest`/`knn` absent, never a penalty.
|
||
5. **`src/curate/telemetry.rs`** (§7.4, §7.5): the `candidate_runs` writer. One row per considered
|
||
article per run, upserted with `INSERT … ON CONFLICT(run_id, article_id) DO UPDATE` setting every
|
||
column. Hygiene-excluded articles get thin rows (`stage='excluded'`, `excluded_reason` ∈
|
||
`blocked | published_before | recently_rejected`, `signals_json='{}'`). In this step map the old
|
||
pipeline onto the stage vocabulary: passed hygiene → `eligible`; cut by the prefilter →
|
||
`eligible` + `excluded_reason='not_admitted'`; prefilter survivors → `admitted`; Stage-A scored →
|
||
`assessed`; sent to selection → `shortlisted`; picked → `selected` (+ `editor_why` if a `why`
|
||
exists on the pick, else NULL); unpicked shortlist → `excluded_reason='not_selected'`. `signals_json`
|
||
follows §7.5 exactly (`v: 1`, `raw`, `norm`, `present`, `weights` = effective preliminary weights,
|
||
`top_interests`, `neighbours`, `exploration: false`, `auto_include`, `notes`). `utility` and
|
||
`rank_utility` stay NULL until step 5.
|
||
6. **CLI** (§15.2, §16) in `src/main.rs`:
|
||
`explain --date YYYY-MM-DD (--article ID | --url URL) [--run-id N]` and
|
||
`explain --date YYYY-MM-DD --near-misses [N]` printing the persisted `candidate_runs` row for the
|
||
latest non-dry run of that date (or `--run-id`): stage and reason; every raw/normalized signal with
|
||
presence and effective weight; top interests with z; nearest rated neighbours; assessments from
|
||
`article_assessments` when present (empty until step 4); admitted_by; the editor's `why`. `--url`
|
||
canonicalizes (`dedupe`) and looks the article up; if absent from `articles`, print that it was never
|
||
ingested. `--near-misses` lists the top N by preliminary blend (utility from step 5 when present)
|
||
that were not selected. `features backfill [--days 30] [--rated-only] [--all] [--yes]` embeds rated
|
||
and published articles first, then interests, then other recent articles only under `--all`, prints
|
||
an estimate and asks for confirmation above 5M tokens unless `--yes`, and is idempotent (warm cache
|
||
⇒ zero calls). `features prune` removes `article_embeddings` rows for articles neither rated nor
|
||
published older than `embedding_retention_days` and `candidate_runs` rows whose run started more
|
||
than `telemetry_retention_days` ago. `features backfill` takes no lock yet (step 6 adds `lock.rs`).
|
||
7. Types (§18): `Signals`, `Signal` helpers, `RatedArticle` already exists — extend if needed. Do not
|
||
create `Candidate` yet (step 4/5).
|
||
|
||
Tests (§20 "Embeddings", "Interest z-scores", "Preference", "Normalization", plus telemetry/explain):
|
||
BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed
|
||
text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the
|
||
others; embedded text contains no feed name or author; a broad interest with uniformly high cosine does
|
||
not dominate while a specific interest with one strong match does; raw fallback under 30 articles; one
|
||
loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high
|
||
(anti-centroid); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate
|
||
0 below `knn_floor`, 1 at `knn_full`, linear between; feed credit sums to 1 across direct feeds; feed
|
||
affinity uses the mean; constant signal → 0.5 for everyone; ties get equal percentiles (400 identical
|
||
zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate
|
||
missing a signal is scored on the rest; `candidate_runs` rows written for every considered article with
|
||
the right stage/reason on a mocked run; `--skip-embeddings` makes zero Voyage calls; a Voyage failure
|
||
still publishes; `explain` renders from persisted rows and reports "never ingested"; `features prune`
|
||
respects rated/published.
|