mlsanityData to 14 Oct 2026

SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World

arXiv 2312.02976 ↗First posted 5 Dec 2023cs.RO cs.AI cs.CVcited by 48

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents. RL requires extensive reward shaping and auxiliary losses and is often too slow and ineffective for long-horizon tasks. While IL with human supervision is effective, collecting human trajectories at scale is extremely expensive. In this work, we show that imitating shortest-path planners in simulation produces agents that, given a language instruction, can proficiently navigate, explore, and manipulate objects in both simulation and in the real world using only RGB sensors (no depth map or GPS coordinates). This surprising result is enabled by our end-to-end, transformer-based, SPOC architecture, powerful visual encoders paired with extensive image augmentation, and the dramatic scale and diversity of our training data: millions of frames of shortest-path-expert trajectories collected inside approximately 200,000 procedurally generated houses containing 40,000 unique 3D assets. Our models, data, training code, and newly proposed 10-task benchmarking suite CHORES are available in https://spoc-robot.github.io.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 70 of its 92 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

70 of 92 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Bridging the Imitation Gap by Adaptive Insubordination23 Jul

Other references · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  8. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  9. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  10. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  11. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  13. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  14. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  15. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  16. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  17. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  18. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  19. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  20. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  21. 2020Learning to Explore using Active Neural SLAMcited by 316
  22. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  23. 2023Fast Segment Anythingcited by 290
  24. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  25. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  26. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  27. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  28. 2020Neural Topological SLAM for Visual Navigationcited by 195
  29. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 194
  30. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
  31. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  32. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 155
  33. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  34. 2022Navigating to Objects in the Real Worldcited by 106
  35. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  36. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  37. 2021Visual Room Rearrangementcited by 102
  38. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 28; 2026: 11.

48 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  4. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  5. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  6. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  7. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 49
  8. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  9. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  10. 2025MetaScenes: Towards Automated Replica Creation for Real-world 3D Scanscited by 31
  11. 2024TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learningcited by 22
  12. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 18
36 more
  1. 2024Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomycited by 16
  2. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  3. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understandingcited by 16
  4. 2024FlowNav: Combining Flow Matching and Depth Priors for Efficient Navigationcited by 12
  5. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  6. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 10
  7. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 8
  8. 2025cVLA: Towards Efficient Camera-Space VLAscited by 8
  9. 2025DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Roboticscited by 8
  10. 2025TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goalscited by 7
  11. 2025Constrained Decoding for Safe Robot Navigation Foundation Modelscited by 5
  12. 2025ObjectReact: Learning Object-Relative Control for Visual Navigationcited by 5
  13. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  14. 2025BiFold: Bimanual Cloth Folding with Language Guidancecited by 4
  15. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 4
  16. 2024λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Roboticscited by 3
  17. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  18. 2025MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learningcited by 3
  19. 2026Generalizable Dense Reward for Long-Horizon Robotic Taskscited by 3
  20. 2025LagMemo: Language 3D Gaussian Splatting Memory for Multi-modal Open-vocabulary Multi-goal Visual Navigationcited by 2
  21. 2025Synthetic vs. Real Training Data for Visual Navigationcited by 1
  22. 2025Shedding Light on VLN Robustness: A Black-box Framework for Indoor Lighting-based Adversarial Attackcited by 1
  23. 2026ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulationcited by 1
  24. 2026RedVLA: Physical Red Teaming for Vision-Language-Action Modelscited by 1
  25. 2026Image2Sim: Scaling Embodied Navigation via Generative Neural Simulatorcited by 1
  26. 2025Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Foldingcited by 0
  27. 2025Recursive Visual Imagination and Adaptive Linguistic Grounding for Vision Language Navigationcited by 0
  28. 2025FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Taskscited by 0
  29. 2025RANGER: A Monocular Zero-Shot Semantic Navigation Framework through Visual Contextual Adaptationcited by 0
  30. 2026Enhancing Linguistic Generalization of VLA: Fine-Tuning OpenVLA via Synthetic Instruction Augmentationcited by 0
  31. 2026Specification-Aware Distribution Shaping for Robotics Foundation Modelscited by 0
  32. 2026StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigationcited by 0
  33. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planningcited by 0
  34. 2026ShieldVLA: Feasibility-Aware Safety Alignment for Vision-Language-Action Modelscited by 0
  35. 2026Active Spatial Inspection for Effective and Efficient Embodied Explorationcited by 0
  36. 2026Interactive Navigation Beyond Reachability: Benchmark and Agentic Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Model Predictive Adversarial Imitation Learning for Planning from Observation
  2. 2025CompassNav: Steering From Path Imitation To Decision Understanding In Navigation
  3. 2026Self-Imitated Diffusion Policy for Efficient and Robust Visual Navigation
  4. 2026NavOL: Navigation Policy with Online Imitation Learning
  5. 2024Explorative Imitation Learning: A Path Signature Approach for Continuous Environments
  6. 2025Adapting a World Model for Trajectory Following in a 3D Game
  7. 2018Imitation Learning with Concurrent Actions in 3D Games
  8. 2026Less Is More: Scalable Visual Navigation from Limited Data