mlsanityData to 7 Oct 2026

VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models

arXiv 2511.11007 ↗First posted 14 Nov 2025cs.CV cs.AI cs.LGcited by 31

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.0% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 86 references that are in the map. First posted on arXiv 14 Nov 2025.

Builds on

76 of 86 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoning8 Aug
  2. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens20 Jun
  3. 2025Kimi-VL Technical Report10 Apr
  4. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts28 Feb
  5. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought13 Jan
  6. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Models4 Dec
  7. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  9. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  10. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  11. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions20 Feb
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 64

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  11. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  12. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
38 more
  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  3. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  5. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  6. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  7. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  9. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  10. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  11. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  12. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  13. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  14. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  15. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  16. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  17. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  18. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  19. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  20. 2025GRIT: Teaching MLLMs to Think with Imagescited by 121
  21. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 114
  22. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  23. 2025SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMscited by 110
  24. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 99
  25. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  26. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  27. 2025Visual Planning: Let's Think Only with Imagescited by 85
  28. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  29. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 72
  30. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  31. 2025VGR: Visual Grounded Reasoningcited by 70
  32. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  33. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 55
  34. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuningcited by 51
  35. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  36. 2024Interleaved-Modal Chain-of-Thoughtcited by 47
  37. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  38. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Modelscited by 41

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 27.

31 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 15
  3. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  4. 2025TeleMem: Building Long-Term and Multimodal Memory for Agentic AIcited by 14
  5. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  6. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  7. 2026Residual Context Diffusion Language Modelscited by 9
  8. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  9. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  10. 2026DOPD: Dual On-policy Distillationcited by 9
  11. 2026SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topologycited by 8
  12. 2026Latent On-Policy Self-Distillationcited by 6
19 more
  1. 2026Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulationcited by 4
  2. 2026PATRA: Pattern-Aware Alignment and Balanced Reasoning for Time Series Question Answeringcited by 3
  3. 2026Visual Enhanced Depth Scaling for Multimodal Latent Reasoningcited by 3
  4. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  5. 2026Mem-W: Latent Memory-Native GUI Agentscited by 3
  6. 2026Multimodal Classification via Total Correlation Maximizationcited by 2
  7. 2026FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memorycited by 2
  8. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 1
  9. 2026DeepLatent: Think with Images via Parallel Latent Visual Reasoningcited by 1
  10. 2026IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrievalcited by 1
  11. 2026SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMscited by 1
  12. 2026PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Vision-Language Modelscited by 0
  13. 2026Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptationcited by 0
  14. 2026Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Modelscited by 0
  15. 2026Seek to Segment: Active Perception for Panoramic Referring Segmentationcited by 0
  16. 2026Quo Vadis, World Modeling?cited by 0
  17. 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understandingcited by 0
  18. 2026Dual-Latent Memory Routing for Vision-Language Reasoningcited by 0
  19. 2026VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory
  2. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
  3. 2026Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning
  4. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  5. 2026ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models
  6. 2026FlashBack: Knowing When to Remember in Streaming Vision-Language Models
  7. 2025Vision Language Models: A Survey of 26K Papers
  8. 2025Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models