# Dual Latent Memory for Visual Multi-agent System

arXiv: https://arxiv.org/abs/2602.00471 · HTML: https://mlsanity.com/paper/2602.00471/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 78 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 30 of 65)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training](https://mlsanity.com/paper/2509.23661/) (2025)
- [Scaling Large Language Model-based Multi-Agent Collaboration](https://mlsanity.com/paper/2406.07155/) (2024)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2406.01900/) (2024)
- [Towards a Science of Scaling Agent Systems](https://mlsanity.com/paper/2512.08296/) (2025)
- [Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens](https://mlsanity.com/paper/2506.17218/) (2025)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [Latent Visual Reasoning](https://mlsanity.com/paper/2509.24251/) (2025)
- [Latent Collaboration in Multi-Agent Systems](https://mlsanity.com/paper/2511.20639/) (2025)
- [Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens](https://mlsanity.com/paper/2511.19418/) (2025)
- [MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding](https://mlsanity.com/paper/2503.13964/) (2025)
- [SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models](https://mlsanity.com/paper/2502.13059/) (2025)
- [Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views](https://mlsanity.com/paper/2510.18632/) (2025)
- [Cache-to-Cache: Direct Semantic Communication Between Large Language Models](https://mlsanity.com/paper/2510.03215/) (2025)
- [OneThinker: All-in-one Reasoning Model for Image and Video](https://mlsanity.com/paper/2512.03043/) (2025)
- [MuMA-ToM: Multi-modal Multi-Agent Theory of Mind](https://mlsanity.com/paper/2408.12574/) (2024)
- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)

## Built on by (most cited first, 10 of 10)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [SEAL: Synergistic Co-Evolution of Agents and Learning Environments](https://mlsanity.com/paper/2605.24426/) (2026)
- [The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems](https://mlsanity.com/paper/2602.15382/) (2026)
- [SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topology](https://mlsanity.com/paper/2604.17503/) (2026)
- [Mem-W: Latent Memory-Native GUI Agents](https://mlsanity.com/paper/2605.09317/) (2026)
- [SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs](https://mlsanity.com/paper/2606.20244/) (2026)
- [A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications](https://mlsanity.com/paper/2608.20379/) (2026)
- [Latent Action Reparameterization for Efficient Agent Inference](https://mlsanity.com/paper/2605.18597/) (2026)
- [Collaborative Memory for Multi-Agent VLM Systems](https://mlsanity.com/paper/2609.17921/) (2026)
- [FrameMorrow: Future-guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation](https://mlsanity.com/paper/2609.38839/) (2026)

## Abstract (arXiv)

While Visual Multi-Agent Systems (VMAS) promise to enhance comprehensive abilities through inter-agent collaboration, empirical evidence reveals a counter-intuitive "scaling wall": increasing agent turns often degrades performance while exponentially inflating token costs. We attribute this failure to the information bottleneck inherent in text-centric communication, where converting perceptual and thinking trajectories into discrete natural language inevitably induces semantic loss. To this end, we propose Lmathbf2-VMAS, a novel model-agnostic framework that enables inter-agent collaboration with dual latent memories. Furthermore, we decouple the perception and thinking while dynamically synthesizing dual latent memories. Additionally, we introduce an entropy-driven proactive triggering that replaces passive information transmission with efficient, on-demand memory access. Extensive experiments among backbones, sizes, and multi-agent structures demonstrate that our method effectively breaks the "scaling wall" with superb scalability, improving average accuracy by 2.7-5.4% while reducing token usage by 21.3-44.8%.