perf(build): upgrade rust-i18n v3 -> v4 to fix codegen blowup

Release builds consumed 5+ GB and never completed on a 4-core / 7.6 GB VM, thrashing the machine badly enough to lose SSH access. Root cause was a codegen bug in rust-i18n v3, not machine size. Its macro emitted one HashMap::from([...]) and one add_translations() call per translation string, all in a single function body -- 8,652 of each for our 21 locales / ~9,000 keys. LLVM scales superlinearly on function size, so optimising that one initialiser took ~4.5 GB. Verified via cargo expand (lib target): v3.1.5 v4.2.1 HashMap::from 8,652 0 add_translations 8,652 21 (one per locale) expanded lines 93,026 73,444 Fixed upstream in v4.0.0. Clean release builds, same machine: v3, opt-level=3, thin LTO 5.2 GB stalled >18m, never finished v3, opt-level=1, no LTO 1.8 GB 15m30s (previous workaround) v4, opt-level=3, thin LTO 2.08 GB 3m25s v4, stock cargo defaults 1.92 GB 2m17s Also: - Remove the opt-level=1 / lto=false workaround; cargo's release defaults are correct now. No shipped code is de-optimised. - Keep only genuine small-machine hygiene in .cargo/config.toml: jobs=3 (keeps the box interactive), lld linker, and trimmed debug info on dev/test profiles. - Document the diagnosis, the cargo expand / llvm-lines workflow that found it, and the systemd-based safety nets in docs/BUILDING.md. - Note in main.rs why i18n! must appear in both crate roots (t! expands to crate::_rust_i18n_t, and main.rs re-declares the module tree rather than depending on the lib target). Test suite passes; release binary verified working.
2026-08-07 00:17:21 +00:00
parent d539c8abe7
commit a728c758a7
5 changed files with 312 additions and 15 deletions
@@ -0,0 +1,55 @@
 # Build configuration for keydr.
 #
 # HISTORY: this file once contained aggressive workarounds (LTO disabled,
 # opt-level dropped to 1) because release builds consumed 5+ GB and never
 # finished on a 4-core / 7.6 GB VM. That was NOT a hardware limitation —
 # it was a codegen bug in rust-i18n v3, fixed by upgrading to v4.
 #
 # With rust-i18n v4 the same machine does a full opt-level=3 + thin-LTO
 # release build with a 2.08 GB peak in 3m25s. See docs/BUILDING.md.
 #
 # What remains here is ordinary small-machine hygiene, not a handicap:
 # nothing below reduces the optimisation level of shipped code.
 [build]
 # Cap parallelism at 3 of 4 cores. This is about keeping the machine
 # interactive (SSH stays responsive during a build), not about memory.
 # Remove it if you don't care about using the box while it compiles.
 jobs = 3
 rustflags = [
    # Use the lld that ships with the Rust toolchain. GNU ld is
    # single-threaded and holds the whole link graph in memory; lld is
    # faster and leaner. The -B flag is the stable-Rust way to select it
    # (`-Clink-self-contained=+linker` requires nightly).
    #
    # If you move this repo to another machine, update this path or
    # install system lld and use plain `-fuse-ld=lld`.
    "-Clink-arg=-fuse-ld=lld",
    "-Clink-arg=-B/home/thallada/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld",
 ]
 [profile.dev]
 # Default is debug = 2 (full debug info), the dominant contributor to both
 # rustc peak memory and target/ size. Level 1 keeps line numbers, so
 # backtraces and panics still point at real source lines; you only lose
 # full variable inspection in a debugger.
 debug = 1
 split-debuginfo = "unpacked"
 [profile.dev.package."*"]
 # Dependencies are compiled once and rarely stepped through in a debugger.
 # Dropping their debug info shrinks target/ and speeds up linking.
 debug = 0
 [profile.test]
 debug = 1
 split-debuginfo = "unpacked"
 [profile.test.package."*"]
 debug = 0
 # NOTE: [profile.release] is deliberately absent — cargo's defaults
 # (opt-level = 3, codegen-units = 16, no LTO) are correct for this project
 # now. Add `lto = "thin"` if you want it; measured at 2.08 GB peak / 3m25s
 # on this box, which is comfortable.
@@ -2,6 +2,19 @@
 # It is not intended for manual editing.
 version = 4
 [[package]]
 name = "ahash"
 version = "0.8.12"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
 dependencies = [
 "cfg-if",
 "getrandom 0.3.4",
 "once_cell",
 "version_check",
 "zerocopy",
 ]
 [[package]]
 name = "aho-corasick"
 version = "1.1.4"
@@ -32,6 +45,17 @@ version = "0.1.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299"
 [[package]]
 name = "annotate-snippets"
 version = "0.12.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f211a51805bc641f3ad5b7664c77d2547af685cc33b4cd8d31964027a46f13f1"
 dependencies = [
 "anstyle",
 "memchr",
 "unicode-width",
 ]
 [[package]]
 name = "anstream"
 version = "0.6.21"
@@ -97,6 +121,12 @@ dependencies = [
 "rustversion",
 ]
 [[package]]
 name = "arraydeque"
 version = "0.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7d902e3d592a523def97af8f317b08ce16b7ab854c1985a0c671e6f15cebc236"
 [[package]]
 name = "atomic"
 version = "0.6.1"
@@ -633,6 +663,15 @@ dependencies = [
 "cfg-if",
 ]
 [[package]]
 name = "encoding_rs_io"
 version = "0.1.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fba3fe847045ecff794b9c138293a80db914678c453ad63fbf0c6a9eb6e00b22"
 dependencies = [
 "encoding_rs",
 ]
 [[package]]
 name = "equivalent"
 version = "1.0.2"
@@ -817,9 +856,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
 dependencies = [
 "cfg-if",
 "js-sys",
 "libc",
 "r-efi",
 "wasip2",
 "wasm-bindgen",
 ]
 [[package]]
@@ -852,6 +893,16 @@ dependencies = [
 "walkdir",
 ]
 [[package]]
 name = "granit-parser"
 version = "0.0.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d03f81ad4732830d85cfd417a9f62cde6dadda4354d37d078a6084a19560aa2d"
 dependencies = [
 "arraydeque",
 "smallvec",
 ]
 [[package]]
 name = "h2"
 version = "0.4.13"
@@ -1491,6 +1542,12 @@ dependencies = [
 "memoffset",
 ]
 [[package]]
 name = "nohash-hasher"
 version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2bf50223579dc7cdcfb3bfcacf7069ff68243f8c363f62ffa99cf000a6b9c451"
 [[package]]
 name = "nom"
 version = "7.1.3"
@@ -2123,12 +2180,11 @@ dependencies = [
 [[package]]
 name = "rust-i18n"
-version = "3.1.5"
+version = "4.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fda2551fdfaf6cc5ee283adc15e157047b92ae6535cf80f6d4962d05717dc332"
+checksum = "f10cee36dd3b1f7929ea12b759de9eea9eff83bfccbc71f387ef4d41a57c64a4"
 dependencies = [
 "globwalk",
 "once_cell",
 "regex",
 "rust-i18n-macro",
 "rust-i18n-support",
@@ -2137,39 +2193,33 @@ dependencies = [
 [[package]]
 name = "rust-i18n-macro"
-version = "3.1.5"
+version = "4.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "22baf7d7f56656d23ebe24f6bb57a5d40d2bce2a5f1c503e692b5b2fa450f965"
+checksum = "f0bb1ed4e04fe26c2a2652cad1c6595efaf7196f4445c0d6e13c67154347fb7e"
 dependencies = [
 "glob",
 "once_cell",
 "proc-macro2",
 "quote",
 "rust-i18n-support",
 "serde",
 "serde_json",
 "serde_yaml",
 "syn 2.0.114",
 ]
 [[package]]
 name = "rust-i18n-support"
-version = "3.1.5"
+version = "4.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "940ed4f52bba4c0152056d771e563b7133ad9607d4384af016a134b58d758f19"
+checksum = "ba1c083408a2733180ae0acf2897612f8ceec7b0c0dcd065a0a87103bfb3b1d9"
 dependencies = [
 "arc-swap",
 "base62",
 "globwalk",
 "itertools 0.11.0",
 "lazy_static",
 "normpath",
 "once_cell",
 "proc-macro2",
 "regex",
 "serde",
 "serde-saphyr",
 "serde_json",
 "serde_yaml",
 "siphasher",
 "toml",
 "triomphe",
@@ -2318,6 +2368,25 @@ dependencies = [
 "serde_derive",
 ]
 [[package]]
 name = "serde-saphyr"
 version = "0.0.29"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7bd22781911de0ca6debda95f073c8f18bec65d1a94f1fa9573f3102e514cea4"
 dependencies = [
 "ahash",
 "annotate-snippets",
 "base64",
 "encoding_rs_io",
 "getrandom 0.3.4",
 "granit-parser",
 "nohash-hasher",
 "num-traits",
 "serde_core",
 "smallvec",
 "zmij",
 ]
 [[package]]
 name = "serde_core"
 version = "1.0.228"
@@ -20,7 +20,7 @@ anyhow = "1.0"
 thiserror = "2.0"
 reqwest = { version = "0.12", features = ["blocking"], optional = true }
 icu_normalizer = { version = "2.1", default-features = false, features = ["compiled_data"] }
-rust-i18n = "3"
+rust-i18n = "4"
 [dev-dependencies]
 tempfile = "3"
@@ -0,0 +1,163 @@
 # Building keydr
 ## TL;DR
 ```bash
 cargo build
 cargo test
 cargo build --release
 ```
 Plain cargo. No wrapper, no memory ceiling, no reduced optimisation.
 A clean release build peaks at **1.92 GB** and takes **2m17s** on a
 4-core / 7.6 GB VM.
 If that's not what you're seeing, read on.
 ---
 ## The build used to be unusable — here's what was actually wrong
 Release builds consumed 5+ GB and never finished; the VM thrashed so
 badly that SSH stopped responding and needed a console reboot.
 The obvious explanations were all wrong:
 - **Not too many parallel jobs.** The peak came from a *single* rustc
  process. A single process's memory is unaffected by `--jobs`.
 - **Not insufficient RAM.** 8 GB is fine for a project this size.
 - **Not LTO** (though LTO made it worse).
 **The cause was a codegen bug in `rust-i18n` v3.** Its macro emitted one
 `HashMap::from([...])` construction *and* one `add_translations()` call
 **per translation string** — all inside a single function body. With 21
 locales and ~9,000 keys that's 8,652 separate HashMap constructions in
 one function. LLVM's optimiser scales superlinearly on function size, so
 it consumed ~4.5 GB trying to optimise that one initialiser.
 Verified by expanding the macro:
 | | rust-i18n v3.1.5 | rust-i18n v4.2.1 |
 |---|---|---|
 | `HashMap::from` constructions | 8,652 | **0** |
 | `add_translations` calls | 8,652 | **21** (one per locale) |
 | Expanded lines (lib target) | 93,026 | 73,444 |
 Upstream fixed this in **v4.0.0**. Reproduce the check yourself:
 ```bash
 cargo install cargo-expand
 cargo expand --lib | grep -c 'HashMap::from'
 ```
 ### The fix
 ```bash
 cargo add rust-i18n@4
 ```
 That's it. Measured on this box, same machine, clean builds:
 | Configuration | Peak | Time |
 |---|---|---|
 | v3, `opt-level=3`, thin LTO | 5.2 GB | stalled >18 min, never finished |
 | v3, `opt-level=3`, no LTO | 4.5 GB | stalled >12 min, never finished |
 | v3, `opt-level=1`, no LTO (workaround) | 1.8 GB | 15m30s |
 | **v4, `opt-level=3`, thin LTO** | **2.08 GB** | **3m25s** |
 | **v4, stock cargo defaults** | **1.92 GB** | **2m17s** |
 The earlier workaround — dropping `opt-level` to 1 and disabling LTO —
 has been **removed**. It was treating a symptom.
 ---
 ## What's still in `.cargo/config.toml`, and why
 Nothing there reduces the optimisation of shipped code.
 - **`jobs = 3`** — uses 3 of 4 cores so the box stays interactive while
  compiling. Purely about responsiveness; delete it if you don't care.
 - **`lld` linker** — ships with the Rust toolchain, no install needed.
  GNU ld is single-threaded and holds the whole link graph in memory.
 - **`debug = 1` for this crate, `debug = 0` for dependencies** (dev/test
  profiles only). Debug info is the largest contributor to `target/` size
  and link time. Level 1 keeps line numbers, so backtraces still work.
 `[profile.release]` is deliberately absent — cargo's defaults are right.
 ---
 ## Known remaining inefficiency
 `src/main.rs` re-declares the whole module tree (`mod app; mod config;
 ...`) instead of depending on the `keydr` lib target, so **every module
 compiles twice** — once for the lib, once for the bin. The `i18n!` macro
 must therefore be invoked in both crate roots (`t!()` expands to
 `crate::_rust_i18n_t`, so it must exist in each crate).
 With v4 this duplication is cheap (21 calls, not 8,652), so it's no
 longer worth fixing for build performance alone. If you ever restructure
 `main.rs` to `use keydr::...`, build time would roughly halve again.
 ---
 ## If a build ever does go wild again
 The machine-level protections below need no per-project changes.
 ### Keep SSH alive (recommended, needs root once)
 ```bash
 # Reserve memory for the SSH daemon so it can't be swapped out entirely.
 sudo systemctl edit ssh        # [Service] / MemoryMin=128M
 # Kill on memory *pressure* rather than waiting for true OOM. This reacts
 # during thrashing — exactly the window where the box currently becomes
 # unreachable. Ubuntu ships it but leaves it inactive.
 sudo systemctl enable --now systemd-oomd
 ```
 `earlyoom` is the lighter-weight alternative (`sudo apt install earlyoom`);
 it kills only the single highest-scoring process, which during a build is
 `rustc` itself.
 ### Cap one build ad-hoc (no root)
 `memory` is delegated to the user slice on this box, so you can confine a
 single command without any wrapper script:
 ```bash
 systemd-run --user --scope -p MemoryMax=4G -p MemorySwapMax=0 \
    cargo build --release
 ```
 `MemorySwapMax=0` is the important half. The lockup was never really an
 OOM — it was *thrashing*. With swap available the kernel pages `sshd` out
 to feed the build and the box goes catatonic while `oom_kill` stays at 0.
 Denying the build swap turns a slow total failure into a fast contained one.
 **Do not add `MemoryHigh`.** It sounds safer but traps the process in
 continuous reclaim so it grinds forever instead of dying. Measured against
 an identical 256 MB ceiling: `MemoryMax` alone → clean kill in seconds;
 `MemoryMax` + `MemoryHigh` → still spinning after 45 seconds.
 ### Diagnosing which crate is expensive
 ```bash
 cargo build --release --timings      # HTML report in target/cargo-timings/
 cargo install cargo-llvm-lines
 cargo llvm-lines --release | head -30 # which functions generate the most IR
 cargo expand --lib | wc -l            # how much code a macro really emits
 ```
 `cargo llvm-lines` and `cargo expand` are what actually found this bug.
 Reach for them before touching `opt-level`.
 ## Upstream context
 Cargo has no memory-aware job scheduling; it schedules on core count only.
 That's a known gap ([rust-lang/cargo#12912](https://github.com/rust-lang/cargo/issues/12912)),
 and maintainers have said they'd rather delegate resource limits to the OS
 (cgroups) than build it into cargo. So the systemd approach above isn't a
 hack — it's the sanctioned answer. But in this case none of it was needed:
 the real fix was a dependency upgrade.
@@ -1,3 +1,13 @@
 // The `i18n!` macro must be invoked in each crate root that uses `t!()`,
 // because `t!` expands to `crate::_rust_i18n_t`. main.rs re-declares the
 // module tree rather than depending on the lib target, so it is a separate
 // crate and needs its own invocation.
 //
 // This means the translation table is generated twice (once here, once in
 // lib.rs). With rust-i18n v4 that costs ~21 add_translations calls per
 // invocation instead of v3's ~8,650, so the duplication is now cheap.
 // Eliminating it entirely would require main.rs to `use keydr::...` instead
 // of re-declaring `mod app; mod config; ...` — a larger refactor.
 rust_i18n::i18n!("locales", fallback = "en");
 mod app;