mlsanityData to 7 Oct 2026

TANGO: Training-free Embodied AI Agents for Open-world Tasks

arXiv 2412.10402 ↗First posted 5 Dec 2024cs.AI cs.ROcited by 26

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated excellent capabilities in composing various modules together to create programs that can perform complex reasoning tasks on images. In this paper, we propose TANGO, an approach that extends the program composition via LLMs already observed for images, aiming to integrate those capabilities into embodied agents capable of observing and acting in the world. Specifically, by employing a simple PointGoal Navigation model combined with a memory-based exploration policy as a foundational primitive for guiding an agent through the world, we show how a single model can address diverse tasks without additional training. We task an LLM with composing the provided primitives to solve a specific task, using only a few in-context examples in the prompt. We evaluate our approach on three key Embodied AI tasks: Open-Set ObjectGoal Navigation, Multi-Modal Lifelong Navigation, and Open Embodied Question Answering, achieving state-of-the-art results without any specific fine-tuning in challenging zero-shot scenarios.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 57 of its 64 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

57 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation6 Mar
  2. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models8 Dec
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2023Mistral 7Bcited by 3,065
  10. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  11. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
38 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  7. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  8. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  9. 2015Neural Module Networkscited by 375
  10. 2022Visual Language Maps for Robot Navigationcited by 362
  11. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  12. 2017Embodied Question Answeringcited by 335
  13. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  14. 2020Learning to Explore using Active Neural SLAMcited by 308
  15. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  16. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  17. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  18. 2020Neural Topological SLAM for Visual Navigationcited by 193
  19. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  20. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  21. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  22. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  23. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
  24. 2022Navigating to Objects in the Real Worldcited by 104
  25. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  26. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  27. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  28. 2023GOAT: GO to Any Thingcited by 89
  29. 2023Embodied Task Planning with Large Language Modelscited by 84
  30. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
  31. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  32. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  33. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  34. 2018Explainable Neural Computation via Stack Neural Module Networkscited by 54
  35. 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 53
  36. 2020Auxiliary Tasks Speed Up Learning PointGoal Navigationcited by 52
  37. 2023Navigating to Objects Specified by Imagescited by 50
  38. 2020Exploiting Scene-specific Features for Object Goal Navigationcited by 20

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 16.

26 papers.

Sort:

By area:

  1. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  2. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  3. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  4. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  5. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  6. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  7. 2026GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoningcited by 5
  8. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  9. 2025OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesiscited by 3
  10. 2025Reflex First, Reflect Later: Latency-Aware Embodied LLM Agents for Dynamic Responsecited by 3
  11. 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentationcited by 3
  12. 2026SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoningcited by 3
14 more
  1. 2025BridgeEQA: Virtual Embodied Agents for Real Bridge Inspectionscited by 2
  2. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  3. 2026PSG-Nav: Probabilistic Scene Graph Navigation via Multiverse Decision Makingcited by 2
  4. 2026MerNav: A Highly Generalizable Memory-Execute-Review Framework for Zero-Shot Object Goal Navigationcited by 1
  5. 2026Memory Over Maps: 3D Object Localization Without Reconstructioncited by 1
  6. 2026ViewSAM: Learning View-aware Cross-modal Semantics for Weakly Supervised Cross-view Referring Multi-Object Trackingcited by 1
  7. 2026ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Explorationcited by 1
  8. 2026ProCompNav: Proactive Instance Navigation with Comparative Judgment for Ambiguous User Queriescited by 0
  9. 2026AnyGoal: Vision-Language Guided Multi-Agent Exploration for Training-Free Lifelong Navigationcited by 0
  10. 2026XiDepth: a Lightweight and Efficient Network for Self-supervised Monocular Depth Estimationcited by 0
  11. 2026Hierarchical Fast-Slow ReAct Agent for Zero-Shot Object-Goal Navigationcited by 0
  12. 2026CGFM-Nav: Cognitive Graph-Field Memory for Semantic-Guided Lifelong Multimodal Embodied Navigationcited by 0
  13. 2026Active Spatial Inspection for Effective and Efficient Embodied Explorationcited by 0
  14. 2026Beyond the Remembered World: Predictive 4D Belief for Persistent Navigation in Evolving Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness
  2. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  3. 2026Vesta: A Generalist Embodied Reasoning Model
  4. 2026Lifelong Embodied Navigation Learning
  5. 2024Explore until Confident: Efficient Exploration for Embodied Question Answering
  6. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
  7. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents
  8. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation