# Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating

arXiv: https://arxiv.org/abs/2609.03900 · HTML: https://mlsanity.com/paper/2609.03900/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [MEMORYLLM: Towards Self-Updatable Large Language Models](https://mlsanity.com/paper/2402.04624/) (2024)
- [WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models](https://mlsanity.com/paper/2405.14768/) (2024)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [Merge before Forget: A Single LoRA Continual Learning via Continual Merging](https://mlsanity.com/paper/2512.23017/) (2025)
- [MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs](https://mlsanity.com/paper/2506.07899/) (2025)
- [Reasons and Solutions for the Decline in Model Performance after Editing](https://mlsanity.com/paper/2410.23843/) (2024)
- [EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing](https://mlsanity.com/paper/2510.13851/) (2025)
- [Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs](https://mlsanity.com/paper/2510.22139/) (2025)
- [When Does Continual Learning Require Learning](https://mlsanity.com/paper/2607.07847/) (2026)
- [Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams](https://mlsanity.com/paper/2603.07392/) (2026)
- [RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World](https://mlsanity.com/paper/2604.05096/) (2026)
- [Energy-Structured Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2605.27482/) (2026)
- [Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning](https://mlsanity.com/paper/2412.01004/) (2024)
- [Towards a Principled Evaluation of Knowledge Editors](https://mlsanity.com/paper/2507.05937/) (2025)
- [Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learning](https://mlsanity.com/paper/2604.21927/) (2026)
- [Continual Knowledge Updating in LLM Systems: Learning Through Multi-Timescale Memory Dynamics](https://mlsanity.com/paper/2605.05097/) (2026)
- [Continual Fine-Tuning of Large Language Models via Program Memory](https://mlsanity.com/paper/2605.13162/) (2026)
- [ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models](https://mlsanity.com/paper/2607.26455/) (2026)

## Abstract (arXiv)

Continual knowledge-updating methods are often declared superior from one final checkpoint and one conventional adapter rank. We show that this can be insufficient to identify the better operating point. Holding a periodic hierarchy fixed, we compare it with cumulative replay over a 24-month Wikidata stream while varying evaluation month, replay LoRA rank, and query formulation. The apparent winner changes across this region: on Qwen2.5-1.5B, the hierarchy's 5.0-point advantage over rank-8 replay becomes an 11.6-point deficit against rank-72 replay, and at high ranks a consolidation-aligned endpoint can suggest a tie while time-averaged replay leads by 9-13 points. The same rank-conditioned reversal appears on Llama-3.2-1B and held-out paraphrases. These results show that method ranking in continual updating can depend jointly on when performance is measured and how much replay-side adaptation capacity the baseline receives. We therefore propose reporting trajectories and capacity sweeps, and declaring a robust winner only when the ordering is stable across the evaluation region; otherwise, comparisons should report winner regions and retention-stability-cost frontiers. Under this protocol, the periodic hierarchy is a lower-update-cost operating point, not a quality winner.