the-daily-epub/docs/plans/2026-08-15-implementation-notes.md

Implementation notes (shared brief for all implementation agents)

Authoritative spec: docs/plans/2026-08-15-the-daily-epub.md. Read it fully before writing code. This file records implementation-time decisions and verified external facts. Follow both.

Verified external facts (2026-08-15)

• DeepSeek model id is confirmed: deepseek-v4-flash (version DeepSeek-V4-Flash-0731). Pricing per 1M tokens: $0.0028 cache-hit input, $0.14 cache-miss input, $0.28 output. OpenAI-compatible API at https://api.deepseek.com/v1, supports response_format: {"type":"json_object"}.
• epub-to-xtc-converter (github.com/bigbag/epub-to-xtc-converter) has no global npm bin. It is invoked as: node <repo>/cli/index.js convert book.epub -o book.xtch -f xtch -c settings.json (-f xtc = 1-bit, -f xtch = 2-bit grayscale; init subcommand generates default settings). Therefore config must be fully general: xtc.command = "node", xtc.args = ["/path/to/epub-to-xtc-converter/cli/index.js", "convert"] and the code appends <input.epub> -o <output.xtch> -f <format> (plus -c <settings>). Missing/failed converter is non-fatal (log + continue).
  • Corrected 2026-08-15 (post-M8, verified by running it). -c is not optional in practice: the converter validates settings before opening the EPUB and exits 2 with Configuration errors: - Font path is required. Set font.path in your config file. There is no built-in font default, and the shipped cli/settings.json points at /usr/share/fonts/Adwaita/AdwaitaSans-Regular.ttf, which most servers do not have. So xtc.settings is effectively required whenever xtc.enabled. Deps also need npm install inside cli/ (commander, jszip, minimatch, sharp).
  • That same error also means "I could not read your config." loadSettings in cli/settings.js guards with fs.existsSync(configPath), which returns false on EACCES exactly as it does for a missing file, then silently falls back to DEFAULT_SETTINGS (font.path: null). Since /etc/daily-epub is 0750 daily-epub:daily-epub, running the converter by hand as any other account reproduces the "Font path is required" error against a valid file. Verify as sudo -u daily-epub.
  • Output size. XTCH is a pre-rendered page bitmap: 480×800 at 2bpp = ~96 KB/page. A 20-article issue rendered at font.size = 34 came to 1,088 pages ≈ 104 MB, in ~13 s. retention_days = 21 therefore implies ~2 GB in publish.xtc_dir.
• X4 firmware rendering limits (from the epub-to-xtc-converter optimizer's header, which cites papyrix-reader): 464×788 usable viewport, max image decode 2048×3072, baseline JPEG only, no GIF/SVG/WebP, max 1500 CSS rules and simple selectors only (tag, .class, tag.class — no descendant combinators), max word length 200 chars, images under 20 px treated as decorative. These bind the (X4).epub read natively off BookOrbit; they do not bind the .xtch, which CREngine pre-renders to page bitmaps (convert never calls optimizeEpub — the two subcommands are independent). epub/x4.rs and style-x4.css satisfy all of them; x4::simplify_xhtml soft-hyphenates past MAX_WORD_CHARS and the_x4_stylesheet_uses_no_descendant_selectors guards the selector rule.
• Wikipedia Current Events portal pages are created empty a day ahead. Portal:Current_events/2026_August_15 was created 2026-08-14T03:30Z as a 192-byte stub and did not get its first news item until 2026-08-15T13:28Z. The 05:30 America/New_York timer fires at ~09:30Z, so the issue day's own page is always an unpopulated stub — its only <li> elements are the current-events-navbar edit/history/watch links, which the extractor drops, so extract_events correctly returns None. world::fetch_with_fallback therefore walks back up to MAX_LOOKBACK_DAYS and the section is datelined with the day it actually covers, not the masthead date.

Cross-cutting implementation decisions

1. sqlx usage: use runtime queries (sqlx::query(...).bind(...)) and manual row mapping (or sqlx::FromRow derive with query_as). Do not use the compile-time checked query!/query_as! macros (they require DATABASE_URL/offline data at build time). Migrations via sqlx::migrate!("./migrations") embedded at compile time.
2. Time: jiff everywhere; day boundaries and --date interpretation in the configured timezone (America/New_York default). Store timestamps in SQLite as RFC3339 UTC strings.
3. Errors: modules return thiserror error types or anyhow::Result; main.rs uses anyhow. Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
4. HTTP: one shared reqwest::Client (rustls, gzip, no cookies, 10s timeouts, UA the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)), passed by clone.
5. LLM: async-openai with custom base URL. All LLM calls go through curate/llm.rs LlmClient which tracks token usage into a shared UsageMeter (input/cached/output tokens, cost usd) and enforces max_daily_usd.
6. Testing: unit tests inline per module; integration tests in tests/ over fixture JSON in tests/fixtures/. Never hit the network in tests. LLM stage mockable via --skip-llm (prefilter order used for selection, feed excerpts as summaries).
7. Style: rustfmt defaults, cargo clippy clean-ish, no unwrap() outside tests, tracing spans per pipeline stage.
8. File ownership: waves of agents work in parallel on disjoint files. Do not edit files outside your assigned set (module wiring in main.rs/mod.rs is done by the scaffold and the integration wave). If you need a helper from another module that doesn't exist yet, add a // TODO(integration): ... note and code against the stub signature.
9. Dedupe module: normalize/dedupe (§3.2) lives in src/dedupe.rs (canonical URL fn + clustering), called from the generate pipeline between ingest and extraction.
10. World briefing (§3.8) lives in src/world.rs.
11. Askama templates in src/epub/templates/ (*.xhtml askama templates + style.css, style-x4.css). Askama 0.12+ configured via askama.toml if needed.
12. Determinism: chapter ids art-{entry_id}, stable filenames, issue regeneration for the same date replaces prior rows/files (idempotent upsert everywhere).