mlsanityData to 7 Oct 2026

Gated-Attention Architectures for Task-Oriented Language Grounding

arXiv 1706.07230 ↗First posted 22 Jun 2017cs.LG cs.AI cs.CLcited by 72

Not on the map. Why?

Abstract (arXiv)

To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 1; 2021: 28; 2022: 12; 2023: 15; 2024: 6; 2025: 4; 2026: 4.

72 papers.

Sort:

By area:

  1. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  2. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  3. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  4. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  5. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  6. 2022Navigating to Objects in the Real Worldcited by 104
  7. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  8. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  9. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  10. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  11. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  12. 2022Training Language Models with Language Feedbackcited by 48
38 more
  1. 2022Retrospectives on the Embodied AI Workshopcited by 46
  2. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  3. 2021Visual Navigation with Spatial Attentioncited by 43
  4. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  5. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  6. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  7. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  8. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  9. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  10. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  11. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  12. 2021LILA: Language-Informed Latent Actionscited by 27
  13. 2021Interactive Learning from Activity Descriptioncited by 26
  14. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
  15. 2024Algorithmic Capabilities of Random Transformerscited by 17
  16. 2021Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videoscited by 16
  17. 2021Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?cited by 15
  18. 2022Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Modelcited by 11
  19. 2024Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigationcited by 11
  20. 2021Agent-Centric Relation Graph for Object Visual Navigationcited by 10
  21. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigationcited by 10
  22. 2023Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learningcited by 9
  23. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  24. 2021M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformerscited by 8
  25. 2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
  26. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8
  27. 2025Attribute-Based Robotic Grasping with Data-Efficient Adaptationcited by 8
  28. 2021Attribute-Based Robotic Grasping with One-Grasp Adaptationcited by 7
  29. 2021Grounding Spatio-Temporal Language with Transformerscited by 7
  30. 2021Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scenecited by 7
  31. 2021Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learningcited by 7
  32. 2023Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translationcited by 7
  33. 2021Learning Visual-Audio Representations for Voice-Controlled Robotscited by 5
  34. 2021Visually Grounded Concept Compositioncited by 5
  35. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  36. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  37. 2020Combating False Negatives in Adversarial Imitation Learningcited by 4
  38. 2021Reinforcement Learning of Implicit and Explicit Control Flow in Instructionscited by 4

50 most cited of 72 All 72 (JSON)