perf(build): upgrade rust-i18n v3 -> v4 to fix codegen blowup

Release builds consumed 5+ GB and never completed on a 4-core / 7.6 GB VM, thrashing the machine badly enough to lose SSH access. Root cause was a codegen bug in rust-i18n v3, not machine size. Its macro emitted one HashMap::from([...]) and one add_translations() call per translation string, all in a single function body -- 8,652 of each for our 21 locales / ~9,000 keys. LLVM scales superlinearly on function size, so optimising that one initialiser took ~4.5 GB. Verified via cargo expand (lib target): v3.1.5 v4.2.1 HashMap::from 8,652 0 add_translations 8,652 21 (one per locale) expanded lines 93,026 73,444 Fixed upstream in v4.0.0. Clean release builds, same machine: v3, opt-level=3, thin LTO 5.2 GB stalled >18m, never finished v3, opt-level=1, no LTO 1.8 GB 15m30s (previous workaround) v4, opt-level=3, thin LTO 2.08 GB 3m25s v4, stock cargo defaults 1.92 GB 2m17s Also: - Remove the opt-level=1 / lto=false workaround; cargo's release defaults are correct now. No shipped code is de-optimised. - Keep only genuine small-machine hygiene in .cargo/config.toml: jobs=3 (keeps the box interactive), lld linker, and trimmed debug info on dev/test profiles. - Document the diagnosis, the cargo expand / llvm-lines workflow that found it, and the systemd-based safety nets in docs/BUILDING.md. - Note in main.rs why i18n! must appear in both crate roots (t! expands to crate::_rust_i18n_t, and main.rs re-declares the module tree rather than depending on the lib target). Test suite passes; release binary verified working.
2026-08-07 00:17:21 +00:00
parent d539c8abe7
commit a728c758a7
5 changed files with 312 additions and 15 deletions
@@ -0,0 +1,55 @@
+# Build configuration for keydr.
+#
+# HISTORY: this file once contained aggressive workarounds (LTO disabled,
+# opt-level dropped to 1) because release builds consumed 5+ GB and never
+# finished on a 4-core / 7.6 GB VM. That was NOT a hardware limitation —
+# it was a codegen bug in rust-i18n v3, fixed by upgrading to v4.
+#
+# With rust-i18n v4 the same machine does a full opt-level=3 + thin-LTO
+# release build with a 2.08 GB peak in 3m25s. See docs/BUILDING.md.
+#
+# What remains here is ordinary small-machine hygiene, not a handicap:
+# nothing below reduces the optimisation level of shipped code.
+
+[build]
+# Cap parallelism at 3 of 4 cores. This is about keeping the machine
+# interactive (SSH stays responsive during a build), not about memory.
+# Remove it if you don't care about using the box while it compiles.
+jobs = 3
+
+rustflags = [
+    # Use the lld that ships with the Rust toolchain. GNU ld is
+    # single-threaded and holds the whole link graph in memory; lld is
+    # faster and leaner. The -B flag is the stable-Rust way to select it
+    # (`-Clink-self-contained=+linker` requires nightly).
+    #
+    # If you move this repo to another machine, update this path or
+    # install system lld and use plain `-fuse-ld=lld`.
+    "-Clink-arg=-fuse-ld=lld",
+    "-Clink-arg=-B/home/thallada/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld",
+]
+
+[profile.dev]
+# Default is debug = 2 (full debug info), the dominant contributor to both
+# rustc peak memory and target/ size. Level 1 keeps line numbers, so
+# backtraces and panics still point at real source lines; you only lose
+# full variable inspection in a debugger.
+debug = 1
+split-debuginfo = "unpacked"
+
+[profile.dev.package."*"]
+# Dependencies are compiled once and rarely stepped through in a debugger.
+# Dropping their debug info shrinks target/ and speeds up linking.
+debug = 0
+
+[profile.test]
+debug = 1
+split-debuginfo = "unpacked"
+
+[profile.test.package."*"]
+debug = 0
+
+# NOTE: [profile.release] is deliberately absent — cargo's defaults
+# (opt-level = 3, codegen-units = 16, no LTO) are correct for this project
+# now. Add `lto = "thin"` if you want it; measured at 2.08 GB peak / 3m25s
+# on this box, which is comfortable.
@@ -2,6 +2,19 @@
 # It is not intended for manual editing.
 version = 4

+[[package]]
+name = "ahash"
+version = "0.8.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
+dependencies = [
+ "cfg-if",
+ "getrandom 0.3.4",
+ "once_cell",
+ "version_check",
+ "zerocopy",
+]
+
 [[package]]
 name = "aho-corasick"
 version = "1.1.4"
@@ -32,6 +45,17 @@ version = "0.1.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299"

+[[package]]
+name = "annotate-snippets"
+version = "0.12.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f211a51805bc641f3ad5b7664c77d2547af685cc33b4cd8d31964027a46f13f1"
+dependencies = [
+ "anstyle",
+ "memchr",
+ "unicode-width",
+]
+
 [[package]]
 name = "anstream"
 version = "0.6.21"
@@ -97,6 +121,12 @@ dependencies = [
 "rustversion",
 ]

+[[package]]
+name = "arraydeque"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7d902e3d592a523def97af8f317b08ce16b7ab854c1985a0c671e6f15cebc236"
+
 [[package]]
 name = "atomic"
 version = "0.6.1"
@@ -633,6 +663,15 @@ dependencies = [
 "cfg-if",
 ]

+[[package]]
+name = "encoding_rs_io"
+version = "0.1.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fba3fe847045ecff794b9c138293a80db914678c453ad63fbf0c6a9eb6e00b22"
+dependencies = [
+ "encoding_rs",
+]
+
 [[package]]
 name = "equivalent"
 version = "1.0.2"
@@ -817,9 +856,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
 dependencies = [
 "cfg-if",
+ "js-sys",
 "libc",
 "r-efi",
 "wasip2",
+ "wasm-bindgen",
 ]

 [[package]]
@@ -852,6 +893,16 @@ dependencies = [
 "walkdir",
 ]

+[[package]]
+name = "granit-parser"
+version = "0.0.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d03f81ad4732830d85cfd417a9f62cde6dadda4354d37d078a6084a19560aa2d"
+dependencies = [
+ "arraydeque",
+ "smallvec",
+]
+
 [[package]]
 name = "h2"
 version = "0.4.13"
@@ -1491,6 +1542,12 @@ dependencies = [
 "memoffset",
 ]

+[[package]]
+name = "nohash-hasher"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2bf50223579dc7cdcfb3bfcacf7069ff68243f8c363f62ffa99cf000a6b9c451"
+
 [[package]]
 name = "nom"
 version = "7.1.3"
@@ -2123,12 +2180,11 @@ dependencies = [

 [[package]]
 name = "rust-i18n"
-version = "3.1.5"
+version = "4.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fda2551fdfaf6cc5ee283adc15e157047b92ae6535cf80f6d4962d05717dc332"
+checksum = "f10cee36dd3b1f7929ea12b759de9eea9eff83bfccbc71f387ef4d41a57c64a4"
 dependencies = [
 "globwalk",
- "once_cell",
 "regex",
 "rust-i18n-macro",
 "rust-i18n-support",
@@ -2137,39 +2193,33 @@ dependencies = [

 [[package]]
 name = "rust-i18n-macro"
-version = "3.1.5"
+version = "4.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "22baf7d7f56656d23ebe24f6bb57a5d40d2bce2a5f1c503e692b5b2fa450f965"
+checksum = "f0bb1ed4e04fe26c2a2652cad1c6595efaf7196f4445c0d6e13c67154347fb7e"
 dependencies = [
 "glob",
- "once_cell",
 "proc-macro2",
 "quote",
 "rust-i18n-support",
 "serde",
 "serde_json",
- "serde_yaml",
 "syn 2.0.114",
 ]

 [[package]]
 name = "rust-i18n-support"
-version = "3.1.5"
+version = "4.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "940ed4f52bba4c0152056d771e563b7133ad9607d4384af016a134b58d758f19"
+checksum = "ba1c083408a2733180ae0acf2897612f8ceec7b0c0dcd065a0a87103bfb3b1d9"
 dependencies = [
 "arc-swap",
 "base62",
 "globwalk",
 "itertools 0.11.0",
- "lazy_static",
 "normpath",
- "once_cell",
- "proc-macro2",
- "regex",
 "serde",
+ "serde-saphyr",
 "serde_json",
- "serde_yaml",
 "siphasher",
 "toml",
 "triomphe",
@@ -2318,6 +2368,25 @@ dependencies = [
 "serde_derive",
 ]

+[[package]]
+name = "serde-saphyr"
+version = "0.0.29"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7bd22781911de0ca6debda95f073c8f18bec65d1a94f1fa9573f3102e514cea4"
+dependencies = [
+ "ahash",
+ "annotate-snippets",
+ "base64",
+ "encoding_rs_io",
+ "getrandom 0.3.4",
+ "granit-parser",
+ "nohash-hasher",
+ "num-traits",
+ "serde_core",
+ "smallvec",
+ "zmij",
+]
+
 [[package]]
 name = "serde_core"
 version = "1.0.228"
@@ -20,7 +20,7 @@ anyhow = "1.0"
 thiserror = "2.0"
 reqwest = { version = "0.12", features = ["blocking"], optional = true }
 icu_normalizer = { version = "2.1", default-features = false, features = ["compiled_data"] }
-rust-i18n = "3"
+rust-i18n = "4"

 [dev-dependencies]
 tempfile = "3"
@@ -0,0 +1,163 @@
+# Building keydr
+
+## TL;DR
+
+```bash
+cargo build
+cargo test
+cargo build --release
+```
+
+Plain cargo. No wrapper, no memory ceiling, no reduced optimisation.
+A clean release build peaks at **1.92 GB** and takes **2m17s** on a
+4-core / 7.6 GB VM.
+
+If that's not what you're seeing, read on.
+
+---
+
+## The build used to be unusable — here's what was actually wrong
+
+Release builds consumed 5+ GB and never finished; the VM thrashed so
+badly that SSH stopped responding and needed a console reboot.
+
+The obvious explanations were all wrong:
+
+- **Not too many parallel jobs.** The peak came from a *single* rustc
+  process. A single process's memory is unaffected by `--jobs`.
+- **Not insufficient RAM.** 8 GB is fine for a project this size.
+- **Not LTO** (though LTO made it worse).
+
+**The cause was a codegen bug in `rust-i18n` v3.** Its macro emitted one
+`HashMap::from([...])` construction *and* one `add_translations()` call
+**per translation string** — all inside a single function body. With 21
+locales and ~9,000 keys that's 8,652 separate HashMap constructions in
+one function. LLVM's optimiser scales superlinearly on function size, so
+it consumed ~4.5 GB trying to optimise that one initialiser.
+
+Verified by expanding the macro:
+
+| | rust-i18n v3.1.5 | rust-i18n v4.2.1 |
+|---|---|---|
+| `HashMap::from` constructions | 8,652 | **0** |
+| `add_translations` calls | 8,652 | **21** (one per locale) |
+| Expanded lines (lib target) | 93,026 | 73,444 |
+
+Upstream fixed this in **v4.0.0**. Reproduce the check yourself:
+
+```bash
+cargo install cargo-expand
+cargo expand --lib | grep -c 'HashMap::from'
+```
+
+### The fix
+
+```bash
+cargo add rust-i18n@4
+```
+
+That's it. Measured on this box, same machine, clean builds:
+
+| Configuration | Peak | Time |
+|---|---|---|
+| v3, `opt-level=3`, thin LTO | 5.2 GB | stalled >18 min, never finished |
+| v3, `opt-level=3`, no LTO | 4.5 GB | stalled >12 min, never finished |
+| v3, `opt-level=1`, no LTO (workaround) | 1.8 GB | 15m30s |
+| **v4, `opt-level=3`, thin LTO** | **2.08 GB** | **3m25s** |
+| **v4, stock cargo defaults** | **1.92 GB** | **2m17s** |
+
+The earlier workaround — dropping `opt-level` to 1 and disabling LTO —
+has been **removed**. It was treating a symptom.
+
+---
+
+## What's still in `.cargo/config.toml`, and why
+
+Nothing there reduces the optimisation of shipped code.
+
+- **`jobs = 3`** — uses 3 of 4 cores so the box stays interactive while
+  compiling. Purely about responsiveness; delete it if you don't care.
+- **`lld` linker** — ships with the Rust toolchain, no install needed.
+  GNU ld is single-threaded and holds the whole link graph in memory.
+- **`debug = 1` for this crate, `debug = 0` for dependencies** (dev/test
+  profiles only). Debug info is the largest contributor to `target/` size
+  and link time. Level 1 keeps line numbers, so backtraces still work.
+
+`[profile.release]` is deliberately absent — cargo's defaults are right.
+
+---
+
+## Known remaining inefficiency
+
+`src/main.rs` re-declares the whole module tree (`mod app; mod config;
+...`) instead of depending on the `keydr` lib target, so **every module
+compiles twice** — once for the lib, once for the bin. The `i18n!` macro
+must therefore be invoked in both crate roots (`t!()` expands to
+`crate::_rust_i18n_t`, so it must exist in each crate).
+
+With v4 this duplication is cheap (21 calls, not 8,652), so it's no
+longer worth fixing for build performance alone. If you ever restructure
+`main.rs` to `use keydr::...`, build time would roughly halve again.
+
+---
+
+## If a build ever does go wild again
+
+The machine-level protections below need no per-project changes.
+
+### Keep SSH alive (recommended, needs root once)
+
+```bash
+# Reserve memory for the SSH daemon so it can't be swapped out entirely.
+sudo systemctl edit ssh        # [Service] / MemoryMin=128M
+
+# Kill on memory *pressure* rather than waiting for true OOM. This reacts
+# during thrashing — exactly the window where the box currently becomes
+# unreachable. Ubuntu ships it but leaves it inactive.
+sudo systemctl enable --now systemd-oomd
+```
+
+`earlyoom` is the lighter-weight alternative (`sudo apt install earlyoom`);
+it kills only the single highest-scoring process, which during a build is
+`rustc` itself.
+
+### Cap one build ad-hoc (no root)
+
+`memory` is delegated to the user slice on this box, so you can confine a
+single command without any wrapper script:
+
+```bash
+systemd-run --user --scope -p MemoryMax=4G -p MemorySwapMax=0 \
+    cargo build --release
+```
+
+`MemorySwapMax=0` is the important half. The lockup was never really an
+OOM — it was *thrashing*. With swap available the kernel pages `sshd` out
+to feed the build and the box goes catatonic while `oom_kill` stays at 0.
+Denying the build swap turns a slow total failure into a fast contained one.
+
+**Do not add `MemoryHigh`.** It sounds safer but traps the process in
+continuous reclaim so it grinds forever instead of dying. Measured against
+an identical 256 MB ceiling: `MemoryMax` alone → clean kill in seconds;
+`MemoryMax` + `MemoryHigh` → still spinning after 45 seconds.
+
+### Diagnosing which crate is expensive
+
+```bash
+cargo build --release --timings      # HTML report in target/cargo-timings/
+cargo install cargo-llvm-lines
+cargo llvm-lines --release | head -30 # which functions generate the most IR
+cargo expand --lib | wc -l            # how much code a macro really emits
+```
+
+`cargo llvm-lines` and `cargo expand` are what actually found this bug.
+Reach for them before touching `opt-level`.
+
+## Upstream context
+
+Cargo has no memory-aware job scheduling; it schedules on core count only.
+That's a known gap ([rust-lang/cargo#12912](https://github.com/rust-lang/cargo/issues/12912)),
+and maintainers have said they'd rather delegate resource limits to the OS
+(cgroups) than build it into cargo. So the systemd approach above isn't a
+hack — it's the sanctioned answer. But in this case none of it was needed:
+the real fix was a dependency upgrade.
@@ -1,3 +1,13 @@
+// The `i18n!` macro must be invoked in each crate root that uses `t!()`,
+// because `t!` expands to `crate::_rust_i18n_t`. main.rs re-declares the
+// module tree rather than depending on the lib target, so it is a separate
+// crate and needs its own invocation.
+//
+// This means the translation table is generated twice (once here, once in
+// lib.rs). With rust-i18n v4 that costs ~21 add_translations calls per
+// invocation instead of v3's ~8,650, so the duplication is now cheap.
+// Eliminating it entirely would require main.rs to `use keydr::...` instead
+// of re-declaring `mod app; mod config; ...` — a larger refactor.
 rust_i18n::i18n!("locales", fallback = "en");

 mod app;