mlsanityData to 13 Oct 2026

VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement

arXiv 2210.05064 ↗First posted 11 Oct 2022cs.LG cs.AI cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL). VER leads to significant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments. Specifically, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art distributed SyncOnRL, with similar sample efficiency. For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup) on 8 GPUs than DD-PPO. Compared to SampleFactory (the current state-of-the-art AsyncOnRL), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efficiency. We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB). We find a surprising emergence of navigation in skills that do not ostensible require any navigation. Specifically, the Pick skill involves a robot picking an object from a table. During training the robot was always spawned close to the table and never needed to navigate. However, we find that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high out-of-distribution generalization.

Key facts. Cited by 18 papers in the map (data through 13 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

35 of 51 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2022A ConvNet for the 2020scited by 3,563
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  6. 2018Group Normalizationcited by 1,518
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,488
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,360
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,185
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,169
  11. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,129
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 890
23 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 871
  2. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 859
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
  4. 2021Going deeper with Image Transformerscited by 762
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 673
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 583
  7. 2018On Evaluation of Embodied Navigation Agentscited by 531
  8. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 503
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 379
  10. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  11. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 301
  12. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  13. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  14. 2020Rearrangement: A Challenge for Embodied AIcited by 154
  15. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  16. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 85
  17. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 64
  18. 2020Auxiliary Tasks Speed Up Learning PointGoal Navigationcited by 52
  19. 2018Neural Predictive Belief Representationscited by 39
  20. 2021Batch size-invariance for policy optimizationcited by 34
  21. 2021Megaverse: Simulating Embodied Agents at One Million Experiences per Secondcited by 19
  22. 2021Large Batch Simulation for Deep Reinforcement Learningcited by 18
  23. 2020High-Throughput Synchronous Deep RLcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 4; 2025: 4; 2026: 3.

18 papers.

Sort:

By area:

  1. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 216
  2. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  3. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 164
  4. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  5. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  6. 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 53
  7. 2023Navigating to Objects Specified by Imagescited by 50
  8. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  9. 2023Skill Transformer: A Monolithic Policy for Mobile Manipulationcited by 15
  10. 2023Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Secondcited by 5
  11. 2024Find Everything: A General Vision Language Model Approach to Multi-Object Searchcited by 4
  12. 2025History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigationcited by 4
6 more
  1. 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigationcited by 2
  2. 2025PIGEON: VLM-Driven Object Navigation via Points of Interest Selectioncited by 1
  3. 2025ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 1
  4. 2026ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Explorationcited by 1
  5. 2024Balancing Performance and Efficiency in Zero-shot Robotic Navigationcited by 0
  6. 2026FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Drivingcited by 0