mlsanityData to 7 Oct 2026

Pushing it out of the Way: Interactive Visual Navigation

arXiv 2104.14040 ↗First posted 28 Apr 2021cs.CV cs.AI cs.ROcited by 14

Not on the map. Why?

Abstract (arXiv)

We have observed significant progress in visual navigation for embodied agents. A common assumption in studying visual navigation is that the environments are static; this is a limiting assumption. Intelligent navigation may involve interacting with the environment beyond just moving forward/backward and turning left/right. Sometimes, the best way to navigate is to push something out of the way. In this paper, we study the problem of interactive navigation where agents learn to change the environment to navigate more efficiently to their goals. To this end, we introduce the Neural Interaction Engine (NIE) to explicitly predict the change in the environment caused by the agent's actions. By modeling the changes while planning, we find that agents exhibit significant improvements in their navigational capabilities. More specifically, we consider two downstream tasks in the physics-enabled, visually rich, AI2-THOR environment: (1) reaching a target while the path to the target is blocked (2) moving an object to a target location by pushing it. For both tasks, agents equipped with an NIE significantly outperform agents without the understanding of the effect of the actions indicating the benefits of our approach.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

38 of 46 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Mask R-CNNcited by 7,259
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  6. 2018On Evaluation of Embodied Navigation Agentscited by 526
  7. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  8. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  9. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  10. 2020Learning to Explore using Active Neural SLAMcited by 308
  11. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  12. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
26 more
  1. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  2. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  3. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  4. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  5. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  6. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  7. 2017Visual Interaction Networkscited by 94
  8. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  9. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  10. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89
  11. 2018Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoningcited by 79
  12. 2020Learning Object Relation Graph and Tentative Policy for Visual Navigationcited by 62
  13. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  14. 2020Visual Grounding of Learned Physical Modelscited by 54
  15. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  16. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  17. 2016SE3-Nets: Learning Rigid Body Motion using Deep Neural Networkscited by 48
  18. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  19. 2018Neural Predictive Belief Representationscited by 39
  20. 2015Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Imagescited by 26
  21. 2016"What happens if..." Learning to Predict the Effect of Forces in Imagescited by 23
  22. 2017Visual Forecasting by Imitating Dynamics in Natural Sequencescited by 14
  23. 2020Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interactioncited by 8
  24. 2020Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokobancited by 7
  25. 2019Visual Reaction: Learning to Play Catch with Your Dronecited by 6
  26. 2020Physically Embedded Planning Problems: New Challenges for Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 3; 2025: 2; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  2. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  3. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  4. 2025Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Modelscited by 24
  5. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  6. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 15
  7. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  8. 2022Cascaded Compositional Residual Learning for Complex Interactive Behaviorscited by 9
  9. 2022Object Manipulation via Visual Target Localizationcited by 6
  10. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  11. 2023Moving Forward by Moving Backward: Embedding Action Impact over Action Semanticscited by 3
  12. 2024IN-Sight: Interactive Navigation through Sightcited by 2
2 more
  1. 2022ASC me to Do Anything: Multi-task Training for Embodied AIcited by 1
  2. 2026CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigationcited by 0