PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
We present PhysBrain 1.5, a unified model for understanding physical environments, generating actions, and predicting future states. Motivated by the physical loop of observation, interaction, and environmental change, we bring these capabilities into a common learning framework. Starting from a general vision--language model, we encode language responses, end-effector motion, and dense visual targets as discrete sequences and jointly optimize them with autoregressive next-token prediction. Pre-training draws its embodied supervision entirely from human interaction videos, using task-centered episodes to pair semantic and spatial context with recovered motion and subsequent observations. We then adapt the model through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Across 28 embodied understanding benchmarks, our 8B model achieves an average score of 72.5, setting a new open-source state of the art and performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on 14 benchmarks while retaining general multimodal capabilities. Beyond these understanding evaluations, qualitative examples show the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 128 of its 141 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
128 of 141 references.
Closest ancestors · 18
Same direction, newest first
- 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding6 Nov
- 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models9 Oct
- 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding30 Sep
- 2025MindCube: Spatial Mental Modeling from Limited Views26 Jun
- 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
- 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence29 May
- 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models27 May
- 2025Can Multimodal Large Language Models Understand Spatial Relations?25 May
- 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation13 May
- 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete28 Feb
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
- 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models10 Dec
6 more
- 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios5 Dec
- 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
- 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
- 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
- 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
- 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning11 Dec
Other references · 110
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2019Towards VQA Models That Can Readcited by 1,696
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
- 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
- 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
- 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
- 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
- 2026World Action Models are Zero-shot Policiescited by 411
- 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
- 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
- 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
- 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
- 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
- 2022Visual Spatial Reasoningcited by 288
- 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
- 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 250
50 most cited of 110 All 110 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
- 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
- 2026PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
- 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
- 2026GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture
- 2026GEM: Generative Supervision Helps Embodied Intelligence
- 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
- 2025EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence