mlsanityData to 7 Oct 2026

Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents

arXiv 2305.02412 ↗First posted 3 May 2023cs.CL cs.AI cs.LGcited by 32

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Pre-trained large language models (LLMs) capture procedural knowledge about the world. Recent work has leveraged LLM's ability to generate abstract plans to simplify challenging control tasks, either by action scoring, or action modeling (fine-tuning). However, the transformer architecture inherits several constraints that make it difficult for the LLM to directly serve as the agent: e.g. limited input lengths, fine-tuning inefficiency, bias from pre-training, and incompatibility with non-text environments. To maintain compatibility with a low-level trainable actor, we propose to instead use the knowledge in LLMs to simplify the control problem, rather than solving it. We propose the Plan, Eliminate, and Track (PET) framework. The Plan module translates a task description into a list of high-level sub-tasks. The Eliminate module masks out irrelevant objects and receptacles from the observation for the current sub-task. Finally, the Track module determines whether the agent has accomplished each sub-task. On the AlfWorld instruction following benchmark, the PET framework leads to a significant 15% improvement over SOTA for generalization to human goal specifications.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 41 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

33 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models8 Dec
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  12. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
19 more
  1. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  2. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  3. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  4. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  5. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  6. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  7. 2021Skill Induction and Planning with Latent Languagecited by 99
  8. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  9. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  10. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  11. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  12. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  13. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  14. 2016Learning to Perform Physics Experiments via Deep Reinforcement Learningcited by 47
  15. 2021General-Purpose Question-Answering with Macawcited by 40
  16. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  17. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  18. 2021Language Models are Few-Shot Butlerscited by 27
  19. 2017What can you do with a rock? Affordance extraction via word embeddingscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 12; 2025: 5; 2026: 2.

32 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  4. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  5. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  6. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  7. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  8. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  9. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  10. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  11. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  12. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
20 more
  1. 2023SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Datacited by 24
  2. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agentcited by 23
  3. 2024MobileAgent: enhancing mobile control via human-machine interaction and SOP integrationcited by 18
  4. 2023Responsible Task Automation: Empowering Large Language Models as Responsible Task Automatorscited by 14
  5. 2023ProTIP: Progressive Tool Retrieval Improves Planningcited by 12
  6. 2024AgentKit: Structured LLM Reasoning with Dynamic Graphscited by 12
  7. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  8. 2025Retrospex: Language Agent Meets Offline Reinforcement Learning Criticcited by 12
  9. 2023SPRING: Studying the Paper and Reasoning to Play Gamescited by 11
  10. 2024Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvementscited by 11
  11. 2024Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoningcited by 9
  12. 2025Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimizationcited by 9
  13. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Modelscited by 5
  14. 2024Can Agents Spontaneously Form a Society? Introducing a Novel Architecture for Generative Multi-Agents to Elicit Social Emergencecited by 5
  15. 2024ITCMA: A Generative Agent Based on a Computational Consciousness Structurecited by 3
  16. 2024Situated Instruction Followingcited by 3
  17. 2024Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attackscited by 2
  18. 2026CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulationcited by 2
  19. 2025Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challengescited by 1
  20. 2026A Multi-Agent Framework for Zero-Dimensional Reduced-Order Model Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  2. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models
  3. 2025Think, Act, Learn: A Framework for Autonomous Robotic Agents using Closed-Loop Large Language Models
  4. 2025Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments
  5. 2023Collaborating with language models for embodied reasoning
  6. 2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents
  7. 2024Embodied CoT Distillation From LLM To Off-the-shelf Agents
  8. 2024Multi-Agent Planning Using Visual Language Models