mlsanityData to 7 Oct 2026

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

arXiv 1912.01734 ↗First posted 3 Dec 2019cs.CV cs.AI cs.CLcited by 773

Not on the map. Why?

Cited by 150 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level language instructions like "Walk to the coffee maker on the right." ALFRED tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly on ALFRED, suggesting that there is significant room for developing innovative grounded visual language understanding models with this benchmark.

Key facts. Cited by 773 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 64; 2022: 106; 2023: 131; 2024: 145; 2025: 171; 2026: 156.

773 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
38 more
  1. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  3. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  6. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  9. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  13. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  14. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  15. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  16. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  17. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  18. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  19. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  20. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  21. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  22. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  23. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  24. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  25. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  26. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  27. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  28. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  29. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  30. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  31. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  32. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  33. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  34. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  35. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  36. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  37. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  38. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162

50 most cited of 773 All 773 (JSON)