mlsanityData to 7 Oct 2026

PRISM: A Multi-View Multi-Capability Retail Video Dataset for Embodied Vision-Language Models

arXiv 2603.29281 ↗First posted 31 Mar 2026cs.CV cs.AI cs.ROcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360deg viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans approximately 11.8M video frames and approximately 730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at https://dreamvu.ai/prism

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 32 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

22 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 20

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  9. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
8 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  4. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  5. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 202
  6. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  7. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  8. 2025Sari Sandbox: A Virtual Retail Store Environment for Embodied AI Agentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces
  2. 2026Thinker: A vision-language foundation model for embodied intelligence
  3. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
  4. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models
  5. 2026EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
  6. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
  7. 2025Igniting VLMs toward the Embodied Space
  8. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning