# Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

arXiv: https://arxiv.org/abs/2603.07392 · HTML: https://mlsanity.com/paper/2603.07392/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 8 Mar 2026.

## Builds on (most cited first, 30 of 36)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [From RAG to Memory: Non-Parametric Continual Learning for Large Language Models](https://mlsanity.com/paper/2502.14802/) (2025)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization](https://mlsanity.com/paper/2310.10134/) (2023)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners](https://mlsanity.com/paper/2505.11942/) (2025)
- [ReasonIR: Training Retrievers for Reasoning Tasks](https://mlsanity.com/paper/2504.20595/) (2025)
- [SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking](https://mlsanity.com/paper/1907.07421/) (2019)
- [Meta-Learning Online Adaptation of Language Models](https://mlsanity.com/paper/2305.15076/) (2023)
- [REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation](https://mlsanity.com/paper/2502.13270/) (2025)
- [Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation](https://mlsanity.com/paper/2405.13037/) (2024)
- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns](https://mlsanity.com/paper/2506.13356/) (2025)
- [Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models](https://mlsanity.com/paper/2311.08106/) (2023)

## Built on by (most cited first, 5 of 5)

- [EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding](https://mlsanity.com/paper/2605.09874/) (2026)
- [Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding](https://mlsanity.com/paper/2608.08512/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating](https://mlsanity.com/paper/2609.03900/) (2026)
- [Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models](https://mlsanity.com/paper/2609.27510/) (2026)

## Abstract (arXiv)

LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.