mlsanityData to 14 Oct 2026

A Modular Vision-Language-Action Robotics Framework for Indoor Environments

arXiv 2606.31144 ↗First posted 30 Jun 2026cs.RO cs.AIcited by 0

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions. Our framework employs a modular architecture that orchestrates environment mapping, question processing, and navigation. The system operates in two parallel streams: a perception pipeline that constructs a semantic voxel map from real-time camera feeds using OwlViT embeddings, and a language pipeline that classifies user commands with a Vision-Language Model. The mapping is time-constrained; the system proceeds with a partial map if a 500-second exploration limit is reached. The classified query is then grounded in the geometric and semantic context of the map to generate a detailed prompt for the VLM. This yields an actionable output, demonstrating a capable solution for bridging the gap between human language and robotic action.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 13 of its 22 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

13 of 22 references.

References · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  6. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  7. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  8. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  9. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  10. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  11. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigationcited by 21
  12. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 11
1 more
  1. 2020Voxel Map for Visual SLAMcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
  3. 2026Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline
  4. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challenges
  5. 2025A Navigation Framework Utilizing Vision-Language Models
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
  7. 2026CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts
  8. 2025Scalable, Training-Free Visual Language Robotics: A Modular Multi-Model Framework for Consumer-Grade GPUs