mlsanityData to 14 Oct 2026

Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating

arXiv 2609.03900 ↗First posted 3 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Continual knowledge-updating methods are often declared superior from one final checkpoint and one conventional adapter rank. We show that this can be insufficient to identify the better operating point. Holding a periodic hierarchy fixed, we compare it with cumulative replay over a 24-month Wikidata stream while varying evaluation month, replay LoRA rank, and query formulation. The apparent winner changes across this region: on Qwen2.5-1.5B, the hierarchy's 5.0-point advantage over rank-8 replay becomes an 11.6-point deficit against rank-72 replay, and at high ranks a consolidation-aligned endpoint can suggest a tie while time-averaged replay leads by 9-13 points. The same rank-conditioned reversal appears on Llama-3.2-1B and held-out paraphrases. These results show that method ranking in continual updating can depend jointly on when performance is measured and how much replay-side adaptation capacity the baseline receives. We therefore propose reporting trajectories and capacity sweeps, and declaring a robust winner only when the ordering is stable across the evaluation region; otherwise, comparisons should report winner regions and retention-stability-cost frontiers. Under this protocol, the periodic hierarchy is a lower-update-cost operating point, not a quality winner.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

23 of 25 references.

References · 23

  1. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  2. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  3. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  4. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  5. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 93
  6. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 92
  7. 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 56
  8. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 12
  9. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 11
  10. 2025MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMscited by 10
  11. 2024Reasons and Solutions for the Decline in Model Performance after Editingcited by 7
  12. 2025EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editingcited by 7
11 more
  1. 2025Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMscited by 7
  2. 2026When Does Continual Learning Require Learningcited by 7
  3. 2026Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streamscited by 5
  4. 2026RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real Worldcited by 4
  5. 2026Energy-Structured Low-Rank Adaptation for Continual Learningcited by 3
  6. 2024Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learningcited by 2
  7. 2025Towards a Principled Evaluation of Knowledge Editorscited by 1
  8. 2026Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learningcited by 1
  9. 2026Continual Knowledge Updating in LLM Systems: Learning Through Multi-Timescale Memory Dynamicscited by 1
  10. 2026Continual Fine-Tuning of Large Language Models via Program Memorycited by 1
  11. 2026ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Modelscited by 1

Built on by

Not cited yet.