# A Modular Vision-Language-Action Robotics Framework for Indoor Environments

arXiv: https://arxiv.org/abs/2606.31144 · HTML: https://mlsanity.com/paper/2606.31144/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding](https://mlsanity.com/paper/2105.09996/) (2021)
- [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023)
- [VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation](https://mlsanity.com/paper/2411.03540/) (2024)
- [SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models](https://mlsanity.com/paper/2504.18684/) (2025)
- [Voxel Map for Visual SLAM](https://mlsanity.com/paper/2003.02247/) (2020)

## Abstract (arXiv)

This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions. Our framework employs a modular architecture that orchestrates environment mapping, question processing, and navigation. The system operates in two parallel streams: a perception pipeline that constructs a semantic voxel map from real-time camera feeds using OwlViT embeddings, and a language pipeline that classifies user commands with a Vision-Language Model. The mapping is time-constrained; the system proceeds with a partial map if a 500-second exploration limit is reached. The classified query is then grounded in the geometric and semantic context of the map to generate a detailed prompt for the VLM. This yields an actionable output, demonstrating a capable solution for bridging the gap between human language and robotic action.