mlsanityData to 7 Oct 2026

A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution

arXiv 2107.05612 ↗First posted 12 Jul 2021cs.RO cs.AI cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

Natural language provides an accessible and expressive interface to specify long-term tasks for robotic agents. However, non-experts are likely to specify such tasks with high-level instructions, which abstract over specific robot actions through several layers of abstraction. We propose that key to bridging this gap between language and robot actions over long execution horizons are persistent representations. We propose a persistent spatial semantic representation method, and show how it enables building an agent that performs hierarchical reasoning to effectively execute long-term tasks. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 41 of its 80 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

41 of 80 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  8. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  12. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
29 more
  1. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  2. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  3. 2016Value Iteration Networkscited by 158
  4. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  5. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  6. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  7. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  8. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  9. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  10. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  11. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  12. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  13. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  14. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  15. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  16. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  17. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
  18. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  19. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  20. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  21. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  22. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  23. 2020Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Followingcited by 18
  24. 2019Conditional Driving from Natural Language Instructionscited by 17
  25. 2018Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observationcited by 12
  26. 2019Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructionscited by 9
  27. 2015Learning Models for Following Natural Language Directions in Unknown Environmentscited by 8
  28. 2021A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environmentcited by 6
  29. 2019Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environmentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 27; 2023: 23; 2024: 24; 2025: 6; 2026: 5.

91 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  5. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  6. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  7. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  8. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  9. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  10. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  11. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  12. 2021Skill Induction and Planning with Latent Languagecited by 99
38 more
  1. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  2. 2023Embodied Task Planning with Large Language Modelscited by 84
  3. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  4. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  5. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  6. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  7. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  8. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  9. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  10. 2022Retrospectives on the Embodied AI Workshopcited by 46
  11. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  12. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  13. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  14. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  15. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  16. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  17. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  18. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  19. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  20. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  21. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  22. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  23. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 23
  24. 2022Few-shot Subgoal Planning with Language Modelscited by 22
  25. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  26. 2022Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planningcited by 19
  27. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  28. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  29. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
  30. 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Followingcited by 14
  31. 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Taskcited by 14
  32. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  33. 2021LUMINOUS: Indoor Scene Generation for Embodied AI Challengescited by 12
  34. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11
  35. 2022Voxel-informed Language Groundingcited by 11
  36. 2022Reasoning about Actions over Visual and Linguistic Modalities: A Surveycited by 10
  37. 2022Don't Copy the Teacher: Data and Model Challenges in Embodied Dialoguecited by 10
  38. 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9

50 most cited of 91 All 91 (JSON)