Modular Framework for Visuomotor Language Grounding
Not on the map. Why?
Abstract (arXiv)
Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research. However, data collection for these tasks is expensive and end-to-end approaches suffer from data inefficiency. We propose the structuring of language, acting, and visual tasks into separate modules that can be trained independently. Using a Language, Action, and Vision (LAV) framework removes the dependence of action and vision modules on instruction following datasets, making them more efficient to train. We also present a preliminary evaluation of LAV on the ALFRED task for visual and interactive instruction following.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 10 of its 13 references that are in the map. First posted on arXiv 5 Sep 2021.
Builds on
10 of 13 references.
References · 10
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
- 2020Experience Grounds Languagecited by 241
- 2021Visual Room Rearrangementcited by 101
- 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89
- 2021A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environmentcited by 6
- 2020Modular Networks for Compositional Instruction Followingcited by 4
Built on by
Citing papers per year: 2022: 3; 2023: 5; 2024: 1.
9 papers.
- 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
- 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
- 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
- 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Followingcited by 14
- 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11
- 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9
- 2022Moment-based Adversarial Training for Embodied Language Comprehensioncited by 8
- 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
- 2023DetermiNet: A Large-Scale Diagnostic Dataset for Complex Visually-Grounded Referencing using Determinerscited by 3
- 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following5 Mar 2024
- 2023DetermiNet: A Large-Scale Diagnostic Dataset for Complex Visually-Grounded Referencing using Determiners7 Sep 2023
- 2023Multi-Level Compositional Reasoning for Interactive Instruction Following18 Aug 2023
- 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulation26 Apr 2023
- 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following7 Apr 2023
- 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling28 Jan 2023
- 2022Moment-based Adversarial Training for Embodied Language Comprehension2 Apr 2022
- 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agents9 Mar 2022
- 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones14 Feb 2022