mlsanityData to 7 Oct 2026

Multitask Multimodal Prompted Training for Interactive Embodied Task Completion

arXiv 2311.04067 ↗First posted 7 Nov 2023cs.LG cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Interactive and embodied tasks pose at least two fundamental challenges to existing Vision & Language (VL) models, including 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation. To tackle these challenges, we propose an Embodied MultiModal Agent (EMMA): a unified encoder-decoder model that reasons over images and trajectories, and casts action prediction as multimodal text generation. By unifying all tasks as text generation, EMMA learns a language of actions which facilitates transfer across tasks. Different to previous modular approaches with independently trained components, we use a single multitask model where each task contributes to goal completion. EMMA performs on par with similar models on several VL benchmarks and sets a new state-of-the-art performance (36.81% success rate) on the Dialog-guided Task Completion (DTC), a benchmark to evaluate dialog-guided agents in the Alexa Arena

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

52 of 76 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  13. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  17. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  18. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  19. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  20. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  21. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  22. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  23. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  24. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  25. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  26. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  27. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  28. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  29. 2021Towards General Purpose Vision Systemscited by 46
  30. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  31. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  32. 2020Factorizing Perception and Policy for Interactive Instruction Followingcited by 28
  33. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  34. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  35. 2021Image Scene Graph Generation (SGG) Benchmarkcited by 15
  36. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11
  37. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  38. 2022Don't Copy the Teacher: Data and Model Challenges in Embodied Dialoguecited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 1; 2026: 1.

9 papers.

  1. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  2. 2024An Interactive Agent Foundation Modelcited by 25
  3. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understandingcited by 22
  4. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  5. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  6. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modelingcited by 6
  7. 2024LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemblecited by 3
  8. 2024Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environmentscited by 0
  9. 2026Environmental Understanding Vision-Language Model for Embodied Agentcited by 0