keydr/docs/plans/2026-04-18-remove-drill-history-cap.md

Remove Drill History Cap

Context

The app currently truncates persisted drill_history to the most recent 500 drills in both:

• finish_drill()
• finish_partial_drill()

This is implemented in src/app.rs by removing the oldest history entry once drill_history.len() > 500.

The original motivation for revisiting this came from the sticky-mastery work, where replay correctness becomes more important. But this history-cap issue is broader than mastery and should be handled as a separate task.

Current Findings

1. The cap affects persisted data, not just UI

lesson_history.json only retains the most recent 500 entries because the in-memory history is pruned before save.

2. Full-history replay already exists in some paths

The app already walks all retained drill_history in several places:

• startup: App::new() calls rebuild_ngram_stats()
• import: imported drill_history is followed by rebuild_ngram_stats()
• delete-history-entry: rebuild_from_history() replays remaining history, then calls rebuild_ngram_stats()

So removing the cap is not just a data-retention change. It also changes the amount of work done by:

• startup
• import
• delete-history rebuilds

3. Saves rewrite the full history file

After each completed or partial drill, save_data() writes the entire drill_history back to lesson_history.json.

That means uncapping history will likely increase:

• per-drill save latency
• JSON serialization cost
• disk usage

Questions To Resolve Later

1. Should history become fully uncapped immediately, or should there be a configurable retention policy?
2. Is JSON blob storage still acceptable for very large histories, or should history move to a more append-friendly format?
3. Should startup continue rebuilding n-gram state from full history on every launch, or should the app persist derived caches?
4. Is delete-history-entry expected to remain a full rebuild operation, or should that workflow be redesigned for large histories?
5. Do we want the exported data format to always include full history, even if local persistence later changes format?

Likely Scope

At minimum, this future plan will need to cover:

Data retention behavior

• remove the 500-entry truncation logic in src/app.rs
• verify save_drill_history() persists full history
• update stale comments/docs that still describe history as capped

Performance analysis

• startup replay cost from rebuild_ngram_stats()
• import cost
• delete-history rebuild cost
• save latency from rewriting the full file every drill

Possible optimization directions

• keep full history but persist derived n-gram caches
• keep full history but move storage away from one monolithic JSON blob
• keep full history but make rebuilds incremental where possible

Non-Goals For This Stub

• choosing the final storage redesign now
• changing history persistence in the mastery plan
• making performance guarantees before measuring realistic history sizes

Files Likely Involved

• src/app.rs
• src/store/json_store.rs
• src/store/schema.rs
• docs/plans/2026-02-09-initial-plan.md
• docs/plans/2026-02-22-n-gram-error-tracking-adaptive-drill-selection.md

Acceptance Criteria For The Future Task

To be defined in the full plan after deeper investigation, but likely to include:

1. users retain full drill history
2. startup performance remains acceptable
3. per-drill save latency remains acceptable
4. replay-derived features remain correct with large histories