mlsanityData to 14 Oct 2026

ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

arXiv 2606.28719 ↗First posted 27 Jun 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Test-time adaptation (TTA) of vision-language models (VLMs) is essential for their robust deployment in dynamic, real-world environments. However, existing TTA methods often adapt locally without accumulating knowledge over time, or operating within a single modality without exploiting VLMs' inherently multi-modal nature. Inspired by the Complementary Memory systems of the biological brain, we propose ComMem, an innovative approach that mimics the distinct but cooperative roles of the hippocampus and neocortex to enable effective TTA for VLMs. ComMem consists of two key components: a fast-adapting detailed memory, akin to the hippocampus, that forms a dynamic visual cache from high-confidence test samples; and a slow-integrating abstract memory, akin to the neocortex, that continually refines global textual prototypes. For each test instance, ComMem jointly optimizes both memory systems to ensure cross-modal consistency. Extensive experiments on 15 benchmark datasets show that ComMem significantly outperforms state-of-the-art methods under both natural distribution shifts and cross-dataset generalization, offering a promising direction for enhancing VLMs' practical adaptability.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 52 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

31 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey7 Aug
  2. 2025BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysis11 Jul
  3. 2025Bayesian Test-Time Adaptation for Vision-Language Models12 Mar
  4. 2025DynaPrompt: Dynamic Test-Time Prompt Tuning27 Jan
  5. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models16 Oct
  6. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  7. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  8. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models19 Mar
  9. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  10. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  11. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  12. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
1 more
  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2019Natural Adversarial Examplescited by 1,408
6 more
  1. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  2. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  3. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  4. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 189
  5. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  6. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Sampling headroom is not selection gain: a compute-value audit of test-time scaling for video world modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
  2. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
  3. 2025Adaptive Cache Enhancement for Test-Time Adaptation of Vision-Language Models
  4. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Models
  5. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models
  6. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  7. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  8. 2026UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models