thallada
9e30c1dcdf
Full implementation of a personalized daily newspaper delivered as an EPUB. Articles are pulled from a local self-hosted Miniflux instance, enriched with comments, summarized and filtered by DeepSeek AI, and then assembled into two EPUB editions: standard and optimized for the Xteink X4 e-ink reader. Both are served by the local self-hosted BookOrbit OPDS server in a separate library. Then the X4 edition is futher converted to XTC format and served over a separate OPDS server hosted by the Rust binary. Runs are tracked in a local SQLite database so runs are idempotent per date. Full documentation of the plan is in docs/plans and setup and install instructions are in the README.md file.
83 lines
6.4 KiB
Markdown
83 lines
6.4 KiB
Markdown
# Implementation notes (shared brief for all implementation agents)
|
||
|
||
Authoritative spec: `docs/plans/2026-08-15-the-daily-epub.md`. Read it fully before writing code.
|
||
This file records implementation-time decisions and verified external facts. Follow both.
|
||
|
||
## Verified external facts (2026-08-15)
|
||
|
||
- **DeepSeek model id is confirmed**: `deepseek-v4-flash` (version DeepSeek-V4-Flash-0731).
|
||
Pricing per 1M tokens: $0.0028 cache-hit input, $0.14 cache-miss input, $0.28 output.
|
||
OpenAI-compatible API at `https://api.deepseek.com/v1`, supports `response_format: {"type":"json_object"}`.
|
||
- **epub-to-xtc-converter** (github.com/bigbag/epub-to-xtc-converter) has **no global npm bin**.
|
||
It is invoked as: `node <repo>/cli/index.js convert book.epub -o book.xtch -f xtch -c settings.json`
|
||
(`-f xtc` = 1-bit, `-f xtch` = 2-bit grayscale; `init` subcommand generates default settings).
|
||
Therefore config must be fully general: `xtc.command = "node"`,
|
||
`xtc.args = ["/path/to/epub-to-xtc-converter/cli/index.js", "convert"]` and the code appends
|
||
`<input.epub> -o <output.xtch> -f <format>` (plus `-c <settings>`). Missing/failed converter is
|
||
non-fatal (log + continue).
|
||
- **Corrected 2026-08-15 (post-M8, verified by running it).** `-c` is *not* optional in
|
||
practice: the converter validates settings before opening the EPUB and exits **2** with
|
||
`Configuration errors: - Font path is required. Set font.path in your config file.` There is
|
||
no built-in font default, and the shipped `cli/settings.json` points at
|
||
`/usr/share/fonts/Adwaita/AdwaitaSans-Regular.ttf`, which most servers do not have. So
|
||
`xtc.settings` is effectively required whenever `xtc.enabled`. Deps also need
|
||
`npm install` **inside `cli/`** (commander, jszip, minimatch, sharp).
|
||
- **That same error also means "I could not read your config."** `loadSettings` in
|
||
`cli/settings.js` guards with `fs.existsSync(configPath)`, which returns `false` on
|
||
`EACCES` exactly as it does for a missing file, then silently falls back to
|
||
`DEFAULT_SETTINGS` (`font.path: null`). Since `/etc/daily-epub` is `0750
|
||
daily-epub:daily-epub`, running the converter by hand as any other account reproduces
|
||
the "Font path is required" error against a valid file. Verify as `sudo -u daily-epub`.
|
||
- **Output size.** XTCH is a pre-rendered page bitmap: 480×800 at 2bpp = ~96 KB/page. A
|
||
20-article issue rendered at `font.size = 34` came to 1,088 pages ≈ **104 MB**, in ~13 s.
|
||
`retention_days = 21` therefore implies ~2 GB in `publish.xtc_dir`.
|
||
- **X4 firmware rendering limits** (from the `epub-to-xtc-converter` optimizer's header, which
|
||
cites papyrix-reader): 464×788 usable viewport, max image decode 2048×3072, **baseline JPEG
|
||
only**, no GIF/SVG/WebP, **max 1500 CSS rules and simple selectors only** (`tag`, `.class`,
|
||
`tag.class` — no descendant combinators), **max word length 200 chars**, images under 20 px
|
||
treated as decorative. These bind the `(X4).epub` read natively off BookOrbit; they do *not*
|
||
bind the `.xtch`, which CREngine pre-renders to page bitmaps (`convert` never calls
|
||
`optimizeEpub` — the two subcommands are independent). `epub/x4.rs` and `style-x4.css` satisfy
|
||
all of them; `x4::simplify_xhtml` soft-hyphenates past `MAX_WORD_CHARS` and
|
||
`the_x4_stylesheet_uses_no_descendant_selectors` guards the selector rule.
|
||
- **Wikipedia Current Events portal pages are created empty a day ahead.**
|
||
`Portal:Current_events/2026_August_15` was created 2026-08-14T03:30Z as a 192-byte stub and
|
||
did not get its first news item until 2026-08-15T13:28Z. The 05:30 America/New_York timer
|
||
fires at ~09:30Z, so the issue day's own page is **always** an unpopulated stub — its only
|
||
`<li>` elements are the `current-events-navbar` edit/history/watch links, which the extractor
|
||
drops, so `extract_events` correctly returns `None`. `world::fetch_with_fallback` therefore
|
||
walks back up to `MAX_LOOKBACK_DAYS` and the section is datelined with the day it actually
|
||
covers, not the masthead date.
|
||
|
||
## Cross-cutting implementation decisions
|
||
|
||
1. **sqlx usage**: use *runtime* queries (`sqlx::query(...).bind(...)`) and manual row mapping
|
||
(or `sqlx::FromRow` derive with `query_as`). Do **not** use the compile-time checked
|
||
`query!`/`query_as!` macros (they require DATABASE_URL/offline data at build time).
|
||
Migrations via `sqlx::migrate!("./migrations")` embedded at compile time.
|
||
2. **Time**: `jiff` everywhere; day boundaries and `--date` interpretation in the configured
|
||
timezone (`America/New_York` default). Store timestamps in SQLite as RFC3339 UTC strings.
|
||
3. **Errors**: modules return `thiserror` error types or `anyhow::Result`; `main.rs` uses `anyhow`.
|
||
Pipeline stages are best-effort where the spec says so (social, XTC, world briefing, images).
|
||
4. **HTTP**: one shared `reqwest::Client` (rustls, gzip, no cookies, 10s timeouts, UA
|
||
`the-daily-epub/1.0 (personal rss digest; contact tyler@hallada.net)`), passed by clone.
|
||
5. **LLM**: `async-openai` with custom base URL. All LLM calls go through `curate/llm.rs`
|
||
`LlmClient` which tracks token usage into a shared `UsageMeter` (input/cached/output tokens,
|
||
cost usd) and enforces `max_daily_usd`.
|
||
6. **Testing**: unit tests inline per module; integration tests in `tests/` over fixture JSON in
|
||
`tests/fixtures/`. Never hit the network in tests. LLM stage mockable via `--skip-llm`
|
||
(prefilter order used for selection, feed excerpts as summaries).
|
||
7. **Style**: rustfmt defaults, `cargo clippy` clean-ish, no `unwrap()` outside tests, tracing
|
||
spans per pipeline stage.
|
||
8. **File ownership**: waves of agents work in parallel on disjoint files. Do not edit files
|
||
outside your assigned set (module wiring in `main.rs`/`mod.rs` is done by the scaffold and
|
||
the integration wave). If you need a helper from another module that doesn't exist yet, add
|
||
a `// TODO(integration): ...` note and code against the stub signature.
|
||
9. **Dedupe module**: normalize/dedupe (§3.2) lives in `src/dedupe.rs` (canonical URL fn +
|
||
clustering), called from the generate pipeline between ingest and extraction.
|
||
10. **World briefing** (§3.8) lives in `src/world.rs`.
|
||
11. **Askama templates** in `src/epub/templates/` (`*.xhtml` askama templates + `style.css`,
|
||
`style-x4.css`). Askama 0.12+ configured via `askama.toml` if needed.
|
||
12. **Determinism**: chapter ids `art-{entry_id}`, stable filenames, issue regeneration for the
|
||
same date replaces prior rows/files (idempotent upsert everywhere).
|