DiffuseAgent-MI: Distributionally-Grounded,Tool-Integrated Self-Evolving Agents for Faithful Visual Reasoning
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Tool-integrated vision-language agents have made remarkable progress on compositional and multi-step visual reasoning. Yet their outputs frequently exhibit unfaithfulness: the stated reasoning path diverges from the computation that actually produced the answer, undermining reliability in safety-critical applications. We present DiffuseAgent-MI, a self-evolving agent whose perceptual grounding is governed by a KL-minimal energy model over feature units, providing a distributional view of visual mechanistic interpretability. The agent learns an energy landscape that softly constrains generated samples to lie near the native prior conditioned on the chosen interpretable unit, closing the gap between the explanation and the internal representation. A verifier then supplies trajectory-level faithfulness rewards, and a repair branch re-conditions the energy when the verifier flags an unfaithful step. On GeoQA, SciVis, VQA-v2 and an in-house multimodal reasoning set, DiffuseAgent-MI improves accuracy by up to 5.1 points over prior self-evolving agents while more than doubling mutual-information faithfulness and human-interpretability agreement. Our analysis shows the energy term and the verifier are complementary: the former guarantees distributional faithfulness, the latter trajectory-level faithfulness, and only their combination closes both gaps.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 25 references that are in the map. First posted on arXiv 1 Aug 2026.
Builds on
22 of 25 references.
Closest ancestors · 3
Same direction, newest first
- 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning25 Nov
- 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
- 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
Other references · 19
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
7 more
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
- 2025UniHOI: Unified Human-Object Interaction Understanding via Unified Token Spacecited by 10
- 2026MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonalitycited by 3
- 2026A Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principlecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning
- 2026Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents
- 2026Evaluating Self-Correcting Vision Agents Through Quantitative and Qualitative Metrics
- 2026Visual Para-Thinker++: A Single-Policy Multi-Agent Framework for Visual Reasoning
- 2026VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
- 2025Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
- 2026DeepLatent: Think with Images via Parallel Latent Visual Reasoning
- 2025Orion: A Unified Visual Agent for Multimodal Perception, Advanced Visual Reasoning and Execution