mlsanityData to 7 Oct 2026

PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning

arXiv 2201.10029 ↗First posted 25 Jan 2022cs.CV cs.AIcited by 149

Not on the map. Why?

Abstract (arXiv)

State-of-the-art approaches to ObjectGoal navigation rely on reinforcement learning and typically require significant computational resources and time for learning. We propose Potential functions for ObjectGoal Navigation with Interaction-free learning (PONI), a modular approach that disentangles the skills of `where to look?' for an object and `how to navigate to (x, y)?'. Our key insight is that `where to look?' can be treated purely as a perception problem, and learned without environment interactions. To address this, we propose a network that predicts two complementary potential functions conditioned on a semantic map and uses them to decide where to look for an unseen object. We train the potential function network using supervised learning on a passive dataset of top-down semantic maps, and integrate it into a modular framework to perform ObjectGoal navigation. Experiments on Gibson and Matterport3D demonstrate that our method achieves the state-of-the-art for ObjectGoal navigation while incurring up to 1,600x less computational cost for training. Code and pre-trained models are available: https://vision.cs.utexas.edu/projects/poni/

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 25 Jan 2022.

Builds on

49 of 66 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2017Mask R-CNNcited by 7,259
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  10. 2018On Evaluation of Embodied Navigation Agentscited by 526
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
37 more
  1. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  2. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  3. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  4. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  5. 2017Embodied Question Answeringcited by 335
  6. 2020Learning to Explore using Active Neural SLAMcited by 308
  7. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  8. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  9. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  10. 2018Semi-parametric Topological Memory for Navigationcited by 203
  11. 2020Neural Topological SLAM for Visual Navigationcited by 193
  12. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  13. 2018Visual Semantic Navigation using Scene Priorscited by 127
  14. 2019Learning Exploration Policies for Navigationcited by 113
  15. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  16. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  17. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  18. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  19. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  20. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  21. 2020Semantic Audio-Visual Navigationcited by 79
  22. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
  23. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  24. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 65
  25. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  26. 2020Learning Object Relation Graph and Tentative Policy for Visual Navigationcited by 62
  27. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  28. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 57
  29. 2020An Exploration of Embodied Visual Explorationcited by 56
  30. 2019Combining Optimal Control and Learning for Visual Navigation in Novel Environmentscited by 50
  31. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 43
  32. 2021Visual Navigation with Spatial Attentioncited by 43
  33. 2021Hierarchical Object-to-Zone Graph for Object Navigationcited by 34
  34. 2020Spatial Action Maps for Mobile Manipulationcited by 31
  35. 2019Learning to Navigate Using Mid-Level Visual Priorscited by 25
  36. 2020Exploiting Scene-specific Features for Object Goal Navigationcited by 20
  37. 2021Environment Predictive Coding for Embodied Agentscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 32; 2024: 44; 2025: 28; 2026: 26.

149 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  3. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  4. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  5. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  6. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  7. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  8. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  9. 2022Navigating to Objects in the Real Worldcited by 104
  10. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  11. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  12. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
38 more
  1. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  2. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  3. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  4. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  5. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 66
  6. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  7. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  8. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 55
  9. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 53
  10. 2023Navigating to Objects Specified by Imagescited by 50
  11. 2024GaussNav: Gaussian Splatting for Visual Navigationcited by 49
  12. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  13. 2022Retrospectives on the Embodied AI Workshopcited by 46
  14. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  15. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  16. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  17. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  18. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  19. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  20. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  21. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 26
  22. 2022PEANUT: Predicting and Navigating to Unseen Targetscited by 22
  23. 2022Search for or Navigate to? Dual Adaptive Thinking for Object Navigationcited by 19
  24. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  25. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  26. 2023Object Goal Navigation with Recursive Implicit Mapscited by 16
  27. 2023End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenoncited by 16
  28. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  29. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  30. 2024Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigationcited by 15
  31. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigationcited by 15
  32. 2024Memory-based Adapters for Online 3D Scene Perceptioncited by 13
  33. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  34. 2023Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulationcited by 12
  35. 2024Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environmentscited by 11
  36. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  37. 2025From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learningcited by 11
  38. 2023AutoNeRF: Training Implicit Scene Representations with Autonomous Agentscited by 10

50 most cited of 149 All 149 (JSON)