mlsanityData to 7 Oct 2026

ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality

arXiv 2608.00775 ↗First posted 1 Aug 2026cs.RO cs.CV cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

ORCESTRA is a mixed-reality system for programming robot digital twins through no-code waypoint teaching and language-guided control. In a passthrough mixed-reality workspace, users place robot twins on real surfaces, teach trajectories, save robot-relative episodes, or issue spoken/typed commands that a vision-language model converts into structured digital-twin plans. Both interaction modes share a backend for metric grounding, embodiment-aware validation, preview, confirmation, and digital-twin execution. The system supports heterogeneous robot embodiments, including fixed-base manipulators, a mobile base, and a humanoid robot, demonstrating MR validation as a safety layer for language-guided robot programming before physical deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 13 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

7 of 13 references.

References · 7

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  4. 2022Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfacescited by 19
  5. 2021WareVR: Virtual Reality Interface for Supervision of Autonomous Robotic System Aimed at Warehouse Stocktakingcited by 5
  6. 2024Enabling Waypoint Generation for Collaborative Robots using LLMs and Mixed Realitycited by 3
  7. 2026HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
  2. 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation
  3. 2026TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation
  4. 2025AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation
  5. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  6. 2026Decoupling Vision, Language, and Action for Efficient Multi-Task Robot Policies
  7. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
  8. 2026D-VLC: Decentralized Vision-Language Collaboration for Heterogeneous Embodied Multi-Robot Systems in Unknown Environments