mlsanityData to 14 Oct 2026

Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

arXiv 2609.00103 ↗First posted 31 Aug 2026cs.LG cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accuracy alone misses properties that matter for real long-horizon tasks. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system's capacity--its raw accuracy at a specific budget--via three axes we call ECC: efficiency--the computation, in FLOPs, needed to answer from memory; compression--whether compressible inputs are remembered more accurately or efficiently; and calibration--whether the system abstains in response to its own uncertainty and the cost of an error. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both. Among a broader set of memory backbones, several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 67 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

50 of 67 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  11. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  12. 2018Datasheets for Datasetscited by 1,386
38 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  3. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  5. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  8. 2014Neural Turing Machinescited by 644
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  10. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  13. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  14. 2023Cognitive Architectures for Language Agentscited by 429
  15. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  16. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  17. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  18. 2014Memory Networkscited by 363
  19. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  20. 2024Titans: Learning to Memorize at Test Timecited by 328
  21. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  22. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  23. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  24. 2023Language Modeling Is Compressioncited by 225
  25. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  26. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  27. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  28. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  29. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
  30. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 84
  31. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  32. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  33. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  34. 2024Just read twice: closing the recall gap for recurrent language modelscited by 60
  35. 2024Needle In A Multimodal Haystackcited by 58
  36. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 47
  37. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 37
  38. 2024Memory Mosaicscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models
  2. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  3. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  4. 2025LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
  5. 2026NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
  6. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  7. 2026Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference
  8. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models