mlsanityData to 7 Oct 2026

CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

arXiv 2112.03227 ↗First posted 6 Dec 2021cs.RO cs.AI cs.CLcited by 564

Not on the map. Why?

Cited by 241 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this paper, we present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments and objects. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Key facts. Cited by 564 papers in the map (data through 7 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

32 of 40 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
20 more
  1. 2019Causal Confusion in Imitation Learningcited by 286
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  4. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  5. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  6. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  7. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  8. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  9. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 59
  10. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  11. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  12. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  13. 2017Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environmentscited by 34
  14. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  15. 2020Robot Object Retrieval with Contextual Natural Language Queriescited by 26
  16. 2018Interactive Grounded Language Acquisition and Generalization in a 2D Worldcited by 21
  17. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  18. 2019Adversarial Skill Networks: Unsupervised Robot Skill Learning from Videocited by 17
  19. 2020Spatial Reasoning from Natural Language Instructions for Robot Manipulationcited by 16
  20. 2021Playful Interactions for Representation Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 33; 2024: 77; 2025: 173; 2026: 262.

564 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  3. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  4. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  5. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  6. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  7. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  8. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  9. 2022Visual Language Maps for Robot Navigationcited by 362
  10. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  11. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  12. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
38 more
  1. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  2. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  3. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  4. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  5. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  8. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  9. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  10. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  11. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  12. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  13. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  14. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  15. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  16. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  17. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  18. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  19. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  20. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  21. 2025Unified Vision-Language-Action Modelcited by 128
  22. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  23. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  24. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  25. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  26. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  27. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  28. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  29. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  30. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  31. 2026A Pragmatic VLA Foundation Modelcited by 97
  32. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  33. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  34. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  35. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  36. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  37. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  38. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 80

50 most cited of 564 All 564 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  2. 2025LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation
  3. 2023Conditionally Combining Robot Skills using Large Language Models
  4. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  5. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  6. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  7. 2024Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping
  8. 2025π0.5: a Vision-Language-Action Model with Open-World Generalization