mlsanityData to 7 Oct 2026

Are you doing what I say? On modalities alignment in ALFRED

arXiv 2110.05665 ↗First posted 12 Oct 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

ALFRED is a recently proposed benchmark that requires a model to complete tasks in simulated house environments specified by instructions in natural language. We hypothesize that key to success is accurately aligning the text modality with visual inputs. Motivated by this, we inspect how well existing models can align these modalities using our proposed intrinsic metric, boundary adherence score (BAS). The results show the previous models are indeed failing to perform proper alignment. To address this issue, we introduce approaches aimed at improving model alignment and demonstrate how improved alignment, improves end task performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

23 of 29 references.

References · 23

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  5. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  6. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  7. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  8. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  9. 2016Learning to Navigate in Complex Environmentscited by 192
  10. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  11. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  12. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
11 more
  1. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  2. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  3. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  4. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  5. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  6. 2019Learning To Follow Directions in Street Viewcited by 29
  7. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  8. 2015Alignment-based compositional semantics for instruction followingcited by 21
  9. 2020Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Followingcited by 18
  10. 2021Are We There Yet? Learning to Localize in Embodied Instruction Followingcited by 7
  11. 2020Modular Networks for Compositional Instruction Followingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Alignment Illusion in Multimodal Large Language Models
  2. 2024How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignment
  3. 2025Aligning Multimodal LLM with Human Preference: A Survey
  4. 2026Anisotropic Modality Align
  5. 2026The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
  6. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
  7. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  8. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models