mlsanityData to 7 Oct 2026

DeepInsight II: One Trace from Benchmark to Robot

arXiv 2608.16556 ↗First posted 17 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment actually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions---task, resource, and result---but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight II keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured repairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 48 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

41 of 48 references.

References · 41

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  9. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  10. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  12. 2018On Evaluation of Embodied Navigation Agentscited by 526
29 more
  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  2. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  5. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  6. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  7. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  8. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  9. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  10. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  11. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  12. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  13. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  14. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 83
  15. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  16. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 65
  17. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  18. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  19. 2026RynnBrain: Open Embodied Foundation Modelscited by 33
  20. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  21. 2025Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twincited by 11
  22. 2026HoloMotion-1 Technical Reportcited by 8
  23. 2025Towards bridging the gap: Systematic sim-to-real transfer for diverse legged robotscited by 7
  24. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  25. 2026DeepInsight: A Unified Evaluation Infrastructure Across the Physical AI Stackcited by 3
  26. 2025Post-Convergence Sim-to-Real Policy Transfer: A Principled Alternative to Cherry-Pickingcited by 1
  27. 2026Athena-WBC: Capability-Aligned Policy Experts for Long-Tail Humanoid Whole-Body Controlcited by 1
  28. 2026Athena-Brain Technical Report: An Efficient Robot Brain for General Intelligence and Embodied Interactioncited by 1
  29. 2026R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Simcited by 1

Built on by

Not cited yet.