the-daily-epub/docs/plans/2026-08-15-implementation-notes.md

Implementation notes (shared brief for all implementation agents)

Authoritative spec: docs/plans/2026-08-15-the-daily-epub.md. Read it fully before writing code. For curation (§3.5, §3.6 and §3.9 of that spec) the authority is now docs/plans/2026-09-02-personalized-curation-v2.md; see "Curation v2" below. This file records implementation-time decisions and verified external facts. Follow both.

Verified external facts (2026-08-15)

• DeepSeek model id is confirmed: deepseek-v4-flash (version DeepSeek-V4-Flash-0731). Pricing per 1M tokens: $0.0028 cache-hit input, $0.14 cache-miss input, $0.28 output. OpenAI-compatible API at https://api.deepseek.com/v1, supports response_format: {"type":"json_object"}.
• epub-to-xtc-converter (github.com/bigbag/epub-to-xtc-converter) has no global npm bin. It is invoked as: node <repo>/cli/index.js convert book.epub -o book.xtch -f xtch -c settings.json (-f xtc = 1-bit, -f xtch = 2-bit grayscale; init subcommand generates default settings). Therefore config must be fully general: xtc.command = "node", xtc.args = ["/path/to/epub-to-xtc-converter/cli/index.js", "convert"] and the code appends <input.epub> -o <output.xtch> -f <format> (plus -c <settings>). Missing/failed converter is non-fatal (log + continue).
  • Corrected 2026-08-15 (post-M8, verified by running it). -c is not optional in practice: the converter validates settings before opening the EPUB and exits 2 with Configuration errors: - Font path is required. Set font.path in your config file. There is no built-in font default, and the shipped cli/settings.json points at /usr/share/fonts/Adwaita/AdwaitaSans-Regular.ttf, which most servers do not have. So xtc.settings is effectively required whenever xtc.enabled. Deps also need npm install inside cli/ (commander, jszip, minimatch, sharp).
  • That same error also means "I could not read your config." loadSettings in cli/settings.js guards with fs.existsSync(configPath), which returns false on EACCES exactly as it does for a missing file, then silently falls back to DEFAULT_SETTINGS (font.path: null). Since /etc/daily-epub is 0750 daily-epub:daily-epub, running the converter by hand as any other account reproduces the "Font path is required" error against a valid file. Verify as sudo -u daily-epub.
  • Output size. XTCH is a pre-rendered page bitmap: 480×800 at 2bpp = ~96 KB/page. A 20-article issue rendered at font.size = 34 came to 1,088 pages ≈ 104 MB, in ~13 s. retention_days = 21 therefore implies ~2 GB in publish.xtc_dir.
• CrossPoint's OPDS browser can only acquire EPUBs. Verified against the yokki-vans/InkPointX sources (an open fork of CrossPoint/CrossInk). Two independent blockers, either one fatal for serving XTC over OPDS:
  1. lib/OpdsParser/OpdsParser.cpp sets an entry's href only for an acquisition link whose type is exactly application/epub+zip (strcmp), or for a navigation link (application/atom+xml). endElement then drops any entry with an empty href. An application/octet-stream acquisition link therefore yields an empty list and the UI reports STR_NO_ENTRIES — "No entries found".
  2. OpdsBookBrowserActivity::downloadBook builds the destination filename as sanitizeFilename(author + " - " + title) + ".epub" — hardcoded, ignoring the URL and Content-Disposition — and ReaderActivity dispatches on extension only (FsHelpers::hasXtcExtension → .xtc/.xtch, no magic-byte sniffing). So even a mistyped XTC link downloads into a file the device will not open. The three OPDS failure strings are distinct and worth reading precisely: STR_FETCH_FEED_FAILED ("Failed to fetch feed"), STR_PARSE_FEED_FAILED ("Failed to parse feed") and STR_NO_ENTRIES ("No entries found") — only the last is reachable after a successful fetch and parse. Consequence: the built-in feed serves EPUBs (both editions), XTC is published but not advertised, and publish.xtc_dir is swept by count. See the amendment in spec §3.11.
• X4 firmware rendering limits (from the epub-to-xtc-converter optimizer's header, which cites papyrix-reader): 464×788 usable viewport, max image decode 2048×3072, baseline JPEG only, no GIF/SVG/WebP, max 1500 CSS rules and simple selectors only (tag, .class, tag.class — no descendant combinators), max word length 200 chars, images under 20 px treated as decorative. These bind the (X4).epub read natively off BookOrbit; they do not bind the .xtch, which CREngine pre-renders to page bitmaps (convert never calls optimizeEpub — the two subcommands are independent). epub/x4.rs and style-x4.css satisfy all of them; x4::simplify_xhtml soft-hyphenates past MAX_WORD_CHARS and the_x4_stylesheet_uses_no_descendant_selectors guards the selector rule.
• Wikipedia Current Events portal pages are created empty a day ahead. Portal:Current_events/2026_August_15 was created 2026-08-14T03:30Z as a 192-byte stub and did not get its first news item until 2026-08-15T13:28Z. The 05:30 America/New_York timer fires at ~09:30Z, so the issue day's own page is always an unpopulated stub — its only <li> elements are the current-events-navbar edit/history/watch links, which the extractor drops, so extract_events correctly returns None. world::fetch_with_fallback therefore walks back up to MAX_LOOKBACK_DAYS and the section is datelined with the day it actually covers, not the masthead date.

Verified external facts (2026-09-02, curation v2)

• Anthropic Messages API (verified 2026-09-02 against the bundled Claude API reference): POST https://api.anthropic.com/v1/messages with headers x-api-key, anthropic-version: 2023-06-01, content-type: application/json and anthropic-beta: server-side-fallback-2026-07-01. Model id claude-opus-5; pricing $5.00 / M input, $25.00 / M output, cache reads 0.1× input ($0.50/M), cache writes 1.25× ($6.25/M); the minimum cacheable prefix is 512 tokens. No sampling parameters (temperature, top_p, top_k are a 400) and no thinking block — adaptive thinking is on by default and depth is set with output_config: {"effort": "high"} (low | medium | high | xhigh | max). The system prompt goes in system: [{type: "text", text, cache_control: {type: "ephemeral"}}]; no assistant prefill, JSON is asked for in the prompt and parsed tolerantly. "fallbacks": "default" (with the beta header) routes a request the safety classifiers would refuse to a fallback model server-side; a response can still end with stop_reason: "refusal" on HTTP 200, which the code treats as an error that degrades the call to DeepSeek. Usage fields: input_tokens (uncached remainder), cache_creation_input_tokens, cache_read_input_tokens, output_tokens. Timeout 300 s; retry 429/5xx/network, never 400. Key only from DAILY_EPUB_PROVIDERS__ANTHROPIC__API_KEY (the provider registry below; the pre-registry DAILY_EPUB_ANTHROPIC__API_KEY is a startup error).
• Gemini 3.8 Flash over the OpenAI-compatible endpoint (beta, verified 2026-09-02): POST https://generativelanguage.googleapis.com/v1beta/openai/chat/completions with Authorization: Bearer <key>, the standard messages / temperature / response_format: {"type": "json_object"} body. Model id gemini-3.8-flash. Reasoning depth is the OpenAI reasoning_effort field, which Google maps onto Gemini 3.x's thinking_level (minimal | low | medium | high; none is not accepted by 3.x models). Usage: implicit cache hits are reported in prompt_tokens_details.cached_tokens (the same field DeepSeek now fills), and completion_tokens already includes the thinking tokens that completion_tokens_details.reasoning_tokens breaks out — so output is priced from completion_tokens alone, never the sum. Prices per 1M tokens (promotional through 2026-12-31): $0.75 input, $0.075 cache read, $3.75 output (thinking included); from 2027-01-01 $1.50 / $0.15 / $7.50. No cache-write charge. Key only from DAILY_EPUB_PROVIDERS__GEMINI__API_KEY. Shipped as [providers.gemini], unreferenced until a role names it.
• Voyage AI embeddings (verified 2026-09-02): POST https://api.voyageai.com/v1/embeddings with Authorization: Bearer <key>; body {input: [...], model: "voyage-4-lite", input_type: "document" | "query", truncation: true, output_dimension: 512, output_dtype: "float"}. Up to 1,000 inputs and 1M tokens per request, 32k tokens per input. Vectors are unit-normalized, so dot product = cosine. $0.02 / M tokens after a 200M-token free allocation. Key only from DAILY_EPUB_VOYAGE__API_KEY.

Cross-cutting implementation decisions

1. sqlx usage: use runtime queries (sqlx::query(...).bind(...)) and manual row mapping (or sqlx::FromRow derive with query_as). Do not use the compile-time checked query!/query_as! macros (they require DATABASE_URL/offline data at build time). Migrations via sqlx::migrate!("./migrations") embedded at compile time.
2. Time: jiff everywhere; day boundaries and --date interpretation in the configured timezone (America/New_York default). Store timestamps in SQLite as RFC3339 UTC strings.
3. Errors: modules return thiserror error types or anyhow::Result; main.rs uses anyhow. Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
4. HTTP: one shared reqwest::Client (rustls, gzip, no cookies, 10s timeouts, UA the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)), passed by clone.
5. LLM: a hand-rolled reqwest client, not async-openai (the published crate exposes neither Client nor CreateChatCompletionRequest at the pinned version). Every LLM call goes through curate/llm.rs: LlmClient { provider, system_prompt, model, effort, max_concurrent_requests, meter, backend, retry } over the ChatBackend trait, with two wire protocols — OpenAiCompatibleBackend ({base_url}/chat/completions, bearer key, response_format: json_object, reasoning_effort when the provider has an effort) and AnthropicBackend (Messages API, facts above). Providers are config, not code: the [providers.<name>] registry (kind = openai | anthropic, base_url, model, effort, max_daily_usd, max_concurrent_requests, price_*) is a BTreeMap<String, ProviderConfig>, and [llm] bulk = "<name>" / editor = "<name>" assign the two roles by name (editor = "" means everything runs on bulk; both roles on one provider share one client and one ceiling). LlmClient::for_provider(name, &cfg, ..) dispatches on kind; Llms::from_config(&config, prompt, &meters) builds the roles; editor_or_bulk() degrades to bulk when the editor client is missing or its meter is tripped. One UsageMeter per referenced provider (llm::provider_meters), keyed by provider name — the same key used for runs.provider_costs_json, the UTC-day spend preload and the log lines — plus Voyage's own. LlmClient.provider is the config name, never the kind. The system prompt is sent first and byte-identical within a run so every provider's prefix cache hits. Keys come only from DAILY_EPUB_PROVIDERS__<NAME>__API_KEY (figment lower-cases the path, so provider names are [a-z0-9_]+); daily-epub config check prints the resolved roles without opening the database.
6. Testing: unit tests inline per module; integration tests in tests/ over fixture JSON in tests/fixtures/. Never hit the network in tests: MockBackend (ChatBackend) and the embedding mock (EmbeddingBackend) stand in for all three providers. --skip-llm makes zero LLM calls (admission by cheap signals, select_without_llm by utility, excerpt summaries); --skip-embeddings makes zero Voyage calls.
7. Style: rustfmt defaults, cargo clippy clean-ish, no unwrap() outside tests, tracing spans per pipeline stage.
8. File ownership: waves of agents work in parallel on disjoint files. Do not edit files outside your assigned set (module wiring in main.rs/mod.rs is done by the scaffold and the integration wave). If you need a helper from another module that doesn't exist yet, add a // TODO(integration): ... note and code against the stub signature.
9. Dedupe module: normalize/dedupe (§3.2) lives in src/dedupe.rs (canonical URL fn + clustering), called from the generate pipeline between ingest and extraction.
10. World briefing (§3.8) lives in src/world.rs.
11. Askama templates in src/epub/templates/ (*.xhtml askama templates + style.css, style-x4.css). Askama 0.12+ configured via askama.toml if needed.
12. Determinism: chapter ids art-{entry_id}, stable filenames, issue regeneration for the same date replaces prior rows/files (idempotent upsert everywhere).

Curation v2 (2026-09-02)

The personalized ranker is specified in docs/plans/2026-09-02-personalized-curation-v2.md (§0 settled decisions, §3 target pipeline, §19 configuration, §21 the seven landed steps); docs/plans/2026-09-02-curation-v2-progress.md records per-step deviations. Facts an implementer needs that are easy to get wrong:

• Tables (migrations/0002_curation_v2.sql, 0003_drop_scores.sql; never edit 0001_init.sql): rating_events (append-only; the current verdict is the latest explicit event), article_embeddings and interest_embeddings (f32 little-endian BLOBs, input_hash = sha256 of the embedded text), article_assessments (stage IN ('triage', 'deep'), reused while model and prompt_version match and assessed_at is within assessment_reuse_days; --rescore ignores the cache), candidate_runs (one row per considered article per run, upserted with every column set on each stage transition), runs.config_json / runs.provider_costs_json, issue_articles.why. ratings, feed_priors and scores are dropped; kv keeps ingest_watermark, taste_profile, taste_profile_learned, profile_version.
• Feedback: Vote is loved | good | down (NotForMe); the HMAC message stays {issue_date}/{article_id}/{vote}. Vote::parse("up") → Loved and auth::verify_token still accepts tokens signed over the literal up segment because published issues carry those links. Keep both.
• Budget day: each provider's UsageMeter is preloaded with the spend of earlier runs on the UTC date of the run's started_at, summed from runs.provider_costs_json (db::spend_for_date by nominal issue date is gone). A tripped meter skips that provider's remaining calls; the paper always publishes.
• Lock: src/lock.rs takes libc::flock(LOCK_EX | LOCK_NB) on <database_path>.lock for generate, profile rebuild, features backfill and backfill-social; a second writer exits with " is already running". serve, explain, stats, ratings, features prune and db migrate never take it.
• Retention: telemetry::prune removes article_embeddings of unrated, unpublished articles older than embedding_retention_days (120) and candidate_runs rows plus article_assessments older than telemetry_retention_days (180). features prune runs it on demand; generate runs it once after publishing, best effort.
• Keys: DAILY_EPUB_PROVIDERS__<NAME>__API_KEY and DAILY_EPUB_VOYAGE__API_KEY map onto ProviderConfig.api_key / VoyageConfig.api_key through figment; the fields exist only for that mapping and are never documented in TOML, logged, or stored (Config::providers_redacted() is what reaches runs.config_json).
• Provider registry (2026-09-02, after step 7): the [deepseek] and [anthropic] tables and the top-level max_daily_usd are gone. [llm] holds the role names and the role-level knobs (triage_batch_size, deep_batch_size, score_temperature, editorial_temperature); [providers.deepseek], [providers.anthropic] and [providers.gemini] ship in config.example.toml and are Config::default() key for key. Stale shapes fail at load, naming the new key: a [deepseek]/[anthropic] header, a top-level max_daily_usd, any of the four role keys outside [llm], or a DAILY_EPUB_DEEPSEEK__* / DAILY_EPUB_ANTHROPIC__* environment variable. Batching concurrency (triage, assess) is the bulk provider's max_concurrent_requests; the summaries fan out at the summary provider's. Models { bulk, editor, summaries } in the colophon and Behind the paper stay model ids taken from the built clients.
• Provider rejections (2026-09-02): curate::batch bisects a triage or deep batch that fails with a non-transient error (Api, Refusal, EmptyResponse, or a response that parses to zero items) down to single articles; a rejected single is retried once on the editor client when it is another provider. An article both refuse gets an article_assessments row with kind = 'provider_rejected', score/fit NULL, rationale = "<provider>: <message ≤ 200 chars>", the bulk model and the stage's prompt_version; the cache loaders skip it (leaving the assessment absent) while it is within assessment_reuse_days, --rescore ignores it, and admit::hygiene never treats it as a low score. Because a recovered article's row carries the editor's model, reuse accepts rows whose model is either configured model (triage::reusable_models).