mlsanityData to 14 Oct 2026

Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

arXiv 2505.14366 ↗First posted 20 May 2025cs.AI cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We present a conceptual framework for training Vision-Language Models (VLMs) to perform Visual Perspective Taking (VPT), a core capability for embodied cognition essential for Human-Robot Interaction (HRI). As a first step toward this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse, that enables supervised learning for spatial reasoning tasks. Each instance includes an RGB image, a natural language description, and a ground-truth 4X4 transformation matrix representing object pose. We focus on inferring Z-axis distance as a foundational skill, with future extensions targeting full 6 Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to support further research. This work serves as a foundational step toward embodied AI systems capable of spatial understanding in interactive human-robot scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 4 of its 21 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

4 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
  2. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models2 Sep

Other references · 2

  1. 2024Vision Language Models See What You Want but not What You Seecited by 10
  2. 2025The Philosophical Foundations of Growing AI Like A Childcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning
  2. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
  3. 2026ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models
  4. 2025Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models
  5. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks
  6. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  7. 2026Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models
  8. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representation