thallada
a728c758a7
Release builds consumed 5+ GB and never completed on a 4-core / 7.6 GB
VM, thrashing the machine badly enough to lose SSH access.
Root cause was a codegen bug in rust-i18n v3, not machine size. Its
macro emitted one HashMap::from([...]) and one add_translations() call
per translation string, all in a single function body -- 8,652 of each
for our 21 locales / ~9,000 keys. LLVM scales superlinearly on function
size, so optimising that one initialiser took ~4.5 GB.
Verified via cargo expand (lib target):
v3.1.5 v4.2.1
HashMap::from 8,652 0
add_translations 8,652 21 (one per locale)
expanded lines 93,026 73,444
Fixed upstream in v4.0.0.
Clean release builds, same machine:
v3, opt-level=3, thin LTO 5.2 GB stalled >18m, never finished
v3, opt-level=1, no LTO 1.8 GB 15m30s (previous workaround)
v4, opt-level=3, thin LTO 2.08 GB 3m25s
v4, stock cargo defaults 1.92 GB 2m17s
Also:
- Remove the opt-level=1 / lto=false workaround; cargo's release
defaults are correct now. No shipped code is de-optimised.
- Keep only genuine small-machine hygiene in .cargo/config.toml:
jobs=3 (keeps the box interactive), lld linker, and trimmed debug
info on dev/test profiles.
- Document the diagnosis, the cargo expand / llvm-lines workflow that
found it, and the systemd-based safety nets in docs/BUILDING.md.
- Note in main.rs why i18n! must appear in both crate roots (t! expands
to crate::_rust_i18n_t, and main.rs re-declares the module tree
rather than depending on the lib target).
Test suite passes; release binary verified working.
164 lines
5.8 KiB
Markdown
164 lines
5.8 KiB
Markdown
# Building keydr
|
|
|
|
## TL;DR
|
|
|
|
```bash
|
|
cargo build
|
|
cargo test
|
|
cargo build --release
|
|
```
|
|
|
|
Plain cargo. No wrapper, no memory ceiling, no reduced optimisation.
|
|
A clean release build peaks at **1.92 GB** and takes **2m17s** on a
|
|
4-core / 7.6 GB VM.
|
|
|
|
If that's not what you're seeing, read on.
|
|
|
|
---
|
|
|
|
## The build used to be unusable — here's what was actually wrong
|
|
|
|
Release builds consumed 5+ GB and never finished; the VM thrashed so
|
|
badly that SSH stopped responding and needed a console reboot.
|
|
|
|
The obvious explanations were all wrong:
|
|
|
|
- **Not too many parallel jobs.** The peak came from a *single* rustc
|
|
process. A single process's memory is unaffected by `--jobs`.
|
|
- **Not insufficient RAM.** 8 GB is fine for a project this size.
|
|
- **Not LTO** (though LTO made it worse).
|
|
|
|
**The cause was a codegen bug in `rust-i18n` v3.** Its macro emitted one
|
|
`HashMap::from([...])` construction *and* one `add_translations()` call
|
|
**per translation string** — all inside a single function body. With 21
|
|
locales and ~9,000 keys that's 8,652 separate HashMap constructions in
|
|
one function. LLVM's optimiser scales superlinearly on function size, so
|
|
it consumed ~4.5 GB trying to optimise that one initialiser.
|
|
|
|
Verified by expanding the macro:
|
|
|
|
| | rust-i18n v3.1.5 | rust-i18n v4.2.1 |
|
|
|---|---|---|
|
|
| `HashMap::from` constructions | 8,652 | **0** |
|
|
| `add_translations` calls | 8,652 | **21** (one per locale) |
|
|
| Expanded lines (lib target) | 93,026 | 73,444 |
|
|
|
|
Upstream fixed this in **v4.0.0**. Reproduce the check yourself:
|
|
|
|
```bash
|
|
cargo install cargo-expand
|
|
cargo expand --lib | grep -c 'HashMap::from'
|
|
```
|
|
|
|
### The fix
|
|
|
|
```bash
|
|
cargo add rust-i18n@4
|
|
```
|
|
|
|
That's it. Measured on this box, same machine, clean builds:
|
|
|
|
| Configuration | Peak | Time |
|
|
|---|---|---|
|
|
| v3, `opt-level=3`, thin LTO | 5.2 GB | stalled >18 min, never finished |
|
|
| v3, `opt-level=3`, no LTO | 4.5 GB | stalled >12 min, never finished |
|
|
| v3, `opt-level=1`, no LTO (workaround) | 1.8 GB | 15m30s |
|
|
| **v4, `opt-level=3`, thin LTO** | **2.08 GB** | **3m25s** |
|
|
| **v4, stock cargo defaults** | **1.92 GB** | **2m17s** |
|
|
|
|
The earlier workaround — dropping `opt-level` to 1 and disabling LTO —
|
|
has been **removed**. It was treating a symptom.
|
|
|
|
---
|
|
|
|
## What's still in `.cargo/config.toml`, and why
|
|
|
|
Nothing there reduces the optimisation of shipped code.
|
|
|
|
- **`jobs = 3`** — uses 3 of 4 cores so the box stays interactive while
|
|
compiling. Purely about responsiveness; delete it if you don't care.
|
|
- **`lld` linker** — ships with the Rust toolchain, no install needed.
|
|
GNU ld is single-threaded and holds the whole link graph in memory.
|
|
- **`debug = 1` for this crate, `debug = 0` for dependencies** (dev/test
|
|
profiles only). Debug info is the largest contributor to `target/` size
|
|
and link time. Level 1 keeps line numbers, so backtraces still work.
|
|
|
|
`[profile.release]` is deliberately absent — cargo's defaults are right.
|
|
|
|
---
|
|
|
|
## Known remaining inefficiency
|
|
|
|
`src/main.rs` re-declares the whole module tree (`mod app; mod config;
|
|
...`) instead of depending on the `keydr` lib target, so **every module
|
|
compiles twice** — once for the lib, once for the bin. The `i18n!` macro
|
|
must therefore be invoked in both crate roots (`t!()` expands to
|
|
`crate::_rust_i18n_t`, so it must exist in each crate).
|
|
|
|
With v4 this duplication is cheap (21 calls, not 8,652), so it's no
|
|
longer worth fixing for build performance alone. If you ever restructure
|
|
`main.rs` to `use keydr::...`, build time would roughly halve again.
|
|
|
|
---
|
|
|
|
## If a build ever does go wild again
|
|
|
|
The machine-level protections below need no per-project changes.
|
|
|
|
### Keep SSH alive (recommended, needs root once)
|
|
|
|
```bash
|
|
# Reserve memory for the SSH daemon so it can't be swapped out entirely.
|
|
sudo systemctl edit ssh # [Service] / MemoryMin=128M
|
|
|
|
# Kill on memory *pressure* rather than waiting for true OOM. This reacts
|
|
# during thrashing — exactly the window where the box currently becomes
|
|
# unreachable. Ubuntu ships it but leaves it inactive.
|
|
sudo systemctl enable --now systemd-oomd
|
|
```
|
|
|
|
`earlyoom` is the lighter-weight alternative (`sudo apt install earlyoom`);
|
|
it kills only the single highest-scoring process, which during a build is
|
|
`rustc` itself.
|
|
|
|
### Cap one build ad-hoc (no root)
|
|
|
|
`memory` is delegated to the user slice on this box, so you can confine a
|
|
single command without any wrapper script:
|
|
|
|
```bash
|
|
systemd-run --user --scope -p MemoryMax=4G -p MemorySwapMax=0 \
|
|
cargo build --release
|
|
```
|
|
|
|
`MemorySwapMax=0` is the important half. The lockup was never really an
|
|
OOM — it was *thrashing*. With swap available the kernel pages `sshd` out
|
|
to feed the build and the box goes catatonic while `oom_kill` stays at 0.
|
|
Denying the build swap turns a slow total failure into a fast contained one.
|
|
|
|
**Do not add `MemoryHigh`.** It sounds safer but traps the process in
|
|
continuous reclaim so it grinds forever instead of dying. Measured against
|
|
an identical 256 MB ceiling: `MemoryMax` alone → clean kill in seconds;
|
|
`MemoryMax` + `MemoryHigh` → still spinning after 45 seconds.
|
|
|
|
### Diagnosing which crate is expensive
|
|
|
|
```bash
|
|
cargo build --release --timings # HTML report in target/cargo-timings/
|
|
cargo install cargo-llvm-lines
|
|
cargo llvm-lines --release | head -30 # which functions generate the most IR
|
|
cargo expand --lib | wc -l # how much code a macro really emits
|
|
```
|
|
|
|
`cargo llvm-lines` and `cargo expand` are what actually found this bug.
|
|
Reach for them before touching `opt-level`.
|
|
|
|
## Upstream context
|
|
|
|
Cargo has no memory-aware job scheduling; it schedules on core count only.
|
|
That's a known gap ([rust-lang/cargo#12912](https://github.com/rust-lang/cargo/issues/12912)),
|
|
and maintainers have said they'd rather delegate resource limits to the OS
|
|
(cgroups) than build it into cargo. So the systemd approach above isn't a
|
|
hack — it's the sanctioned answer. But in this case none of it was needed:
|
|
the real fix was a dependency upgrade.
|