mlsanityData to 14 Oct 2026

PRISM: Perception Reasoning Interleaved for Sequential Decision Making

arXiv 2605.05407 ↗First posted 6 May 2026cs.AIcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception-reasoning-decision gap in standalone Vision-Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question-answer (DQA) pipeline. Instead of passively accepting the VLM's description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 38 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

24 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making19 Mar
  2. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  9. 2023Mistral 7Bcited by 3,164
  10. 2025Qwen3-VL Technical Reportcited by 2,793
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
10 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  3. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  4. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  5. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  6. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  7. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 110
  8. 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Modelscited by 12
  9. 2025EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLMcited by 5
  10. 2023ELBA: Learning by Asking for Embodied Visual Navigation and Task Completioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
  2. 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
  3. 2026Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models
  4. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  5. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
  6. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  7. 2025See, Think, Learn: A Self-Taught Multimodal Reasoner
  8. 2025Unbiased Visual Reasoning with Controlled Visual Inputs