mlsanityData to 7 Oct 2026

PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models

arXiv 2609.14973 ↗First posted 14 Sep 2026cs.CV cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We present PhysBrain 1.5, a unified model for understanding physical environments, generating actions, and predicting future states. Motivated by the physical loop of observation, interaction, and environmental change, we bring these capabilities into a common learning framework. Starting from a general vision--language model, we encode language responses, end-effector motion, and dense visual targets as discrete sequences and jointly optimize them with autoregressive next-token prediction. Pre-training draws its embodied supervision entirely from human interaction videos, using task-centered episodes to pair semantic and spatial context with recovered motion and subsequent observations. We then adapt the model through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Across 28 embodied understanding benchmarks, our 8B model achieves an average score of 72.5, setting a new open-source state of the art and performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on 14 benchmarks while retaining general multimodal capabilities. Beyond these understanding evaluations, qualitative examples show the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 128 of its 141 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

128 of 141 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding6 Nov
  2. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models9 Oct
  3. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding30 Sep
  4. 2025MindCube: Spatial Mental Modeling from Limited Views26 Jun
  5. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
  6. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence29 May
  7. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models27 May
  8. 2025Can Multimodal Large Language Models Understand Spatial Relations?25 May
  9. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation13 May
  10. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete28 Feb
  11. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  12. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models10 Dec
6 more
  1. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios5 Dec
  2. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
  3. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  4. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  5. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
  6. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning11 Dec

Other references · 110

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  11. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  13. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  18. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  19. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  20. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  21. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  22. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  23. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  24. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  25. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  26. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  27. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  28. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  29. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  30. 2026World Action Models are Zero-shot Policiescited by 411
  31. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  32. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  33. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  34. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  35. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  36. 2022Visual Spatial Reasoningcited by 288
  37. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  38. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 250

50 most cited of 110 All 110 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
  2. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
  3. 2026PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
  4. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
  5. 2026GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture
  6. 2026GEM: Generative Supervision Helps Embodied Intelligence
  7. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
  8. 2025EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence