mlsanityData to 14 Oct 2026

GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning

arXiv 2604.17241 ↗First posted 19 Apr 2026cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Implicit spatial relations and deep semantic structures encoded in object attributes are crucial for procedural planning in embodied AI systems. However, existing approaches often over rely on the reasoning capabilities of vision language models (VLMs) themselves, while overlooking the rich structured semantic information that can be mined from multimodal inputs. As a result, models struggle to effectively understand functional spatial relationships in complex scenes. To fully exploit implicit spatial relations and deep semantic structures in multimodal data, we propose GaLa, a vision language framework for multimodal procedural planning. GaLa introduces a hypergraph-based representation, where object instances in the image are modeled as nodes, and region-level hyperedges are constructed by aggregating objects according to their attributes and functional semantics. This design explicitly captures implicit semantic relations among objects as well as the hierarchical organization of functional regions. Furthermore, we design a TriView HyperGraph Encoder that enforces semantic consistency across the node view, area view, and node area association view via contrastive learning, enabling hypergraph semantics to be more effectively injected into downstream VLM reasoning. Extensive experiments on the ActPlan1K and ALFRED benchmarks demonstrate that GaLa significantly outperforms existing methods in terms of execution success rate, LCS, and planning correctness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 51 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

30 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning11 Jul
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
  3. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities4 Oct
  4. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
14 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  4. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  6. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  7. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  8. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 71
  9. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  10. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  11. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 41
  12. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  13. 2026Chain of World: World Model Thinking in Latent Motioncited by 11
  14. 2025Why Multi-Interest Fairness Matters: Hypergraph Contrastive Multi-Interest Learning for Fair Conversational Recommender Systemcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models
  2. 2025MMPlanner: Zero-Shot Multimodal Procedural Planning with Chain-of-Thought Object State Reasoning
  3. 2025Vision Language Models Cannot Plan, but Can They Formalize?
  4. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation
  5. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  6. 2025Visual Planning: Let's Think Only with Images
  7. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning
  8. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks