mlsanityData to 7 Oct 2026

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

arXiv 1809.00786 ↗First posted 4 Sep 2018cs.CLcited by 101

Not on the map. Why?

Abstract (arXiv)

We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 23; 2022: 19; 2023: 15; 2024: 24; 2025: 12; 2026: 6.

101 papers.

Sort:

By area:

  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  4. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  6. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  7. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  8. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  9. 2023AerialVLN: Vision-and-Language Navigation for UAVscited by 117
  10. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  11. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  12. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
38 more
  1. 2024Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodologycited by 92
  2. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  3. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  4. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  5. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  6. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  7. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  8. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  9. 2022Aerial Vision-and-Dialog Navigationcited by 48
  10. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  11. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  12. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  13. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  14. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  15. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  16. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  17. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  18. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  19. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  20. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  21. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  22. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  23. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  24. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  25. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  26. 2022Controllable Text Generation with Language Constraintscited by 18
  27. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  28. 2024Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigationcited by 18
  29. 2024NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigationcited by 18
  30. 2025See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigationcited by 18
  31. 2025LLM-Powered Decentralized Generative Agents with Adaptive Hierarchical Knowledge Graph for Cooperative Planningcited by 17
  32. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  33. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  34. 2021Draw Me a Flower: Processing and Grounding Abstraction in Natural Languagecited by 14
  35. 2024VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigationcited by 14
  36. 2024Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challengecited by 13
  37. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  38. 2024Language-Guided World Models: A Model-Based Approach to AI Controlcited by 12

50 most cited of 101 All 101 (JSON)