mlsanityData to 7 Oct 2026

Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion

arXiv 2108.04927 ↗First posted 10 Aug 2021cs.CV cs.AI cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Language-guided robots performing home and office tasks must navigate in and interact with the world. Grounding language instructions against visual observations and actions to take in an environment is an open challenge. We present Embodied BERT (EmBERT), a transformer-based model which can attend to high-dimensional, multi-modal inputs across long temporal horizons for language-conditioned task completion. Additionally, we bridge the gap between successful object-centric navigation models used for non-interactive agents and the language-guided visual task completion benchmark, ALFRED, by introducing object navigation targets for EmBERT training. We achieve competitive performance on the ALFRED benchmark, and EmBERT marks the first transformer-based model to successfully handle the long-horizon, dense, multi-modal histories of ALFRED, and the first ALFRED model to utilize object-centric navigation targets.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 60 of its 94 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

60 of 94 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2017Mask R-CNNcited by 7,259
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  10. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2016Visual Dialogcited by 377
  13. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  14. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  15. 2017Embodied Question Answeringcited by 335
  16. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  17. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  18. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  19. 2019Vision-and-Dialog Navigationcited by 246
  20. 2020Experience Grounds Languagecited by 241
  21. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  22. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  23. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  24. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  25. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  26. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  27. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  28. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  29. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  30. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  31. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  32. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  33. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  34. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  35. 2020Imitating Interactive Intelligencecited by 55
  36. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  37. 2020Universal Adversarial Attacks with Natural Triggers for Text Classificationcited by 50
  38. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 18; 2023: 18; 2024: 9; 2025: 8; 2026: 1.

57 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  8. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  9. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  11. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  12. 2025Agentic Large Language Models, a surveycited by 105
38 more
  1. 2021Skill Induction and Planning with Latent Languagecited by 99
  2. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  3. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  4. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  5. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  6. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  7. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  8. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  9. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  10. 2022Few-shot Subgoal Planning with Language Modelscited by 22
  11. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  12. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  13. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  14. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  15. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  16. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
  17. 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Followingcited by 14
  18. 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
  19. 2021LUMINOUS: Indoor Scene Generation for Embodied AI Challengescited by 12
  20. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11
  21. 2024Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigationcited by 11
  22. 2023ENTL: Embodied Navigation Trajectory Learnercited by 9
  23. 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9
  24. 2024MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Makingcited by 9
  25. 2024Infer Human's Intentions Before Following Natural Language Instructionscited by 9
  26. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Taskscited by 9
  27. 2022Learning to Act with Affordance-Aware Multimodal Neural SLAMcited by 8
  28. 2023LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Followingcited by 8
  29. 2023Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertaintycited by 8
  30. 2023Egocentric Planning for Scalable Embodied Task Achievementcited by 7
  31. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  32. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  33. 2022Pretraining on Interactions for Learning Grounded Affordance Representationscited by 5
  34. 2022A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cuescited by 5
  35. 2022OpenD: A Benchmark for Language-Driven Door and Drawer Openingcited by 3
  36. 2023LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Modelscited by 3
  37. 2023DetermiNet: A Large-Scale Diagnostic Dataset for Complex Visually-Grounded Referencing using Determinerscited by 3
  38. 2023Natural Language Robot Programming: NLP integrated with autonomous robotic graspingcited by 2

50 most cited of 57 All 57 (JSON)