mlsanityData to 7 Oct 2026

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

arXiv 2206.08522 ↗First posted 17 Jun 2022cs.RO cs.CL cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation. In this work, we aim to fill the blank of the last mile of embodied agents -- object manipulation by following human guidance, e.g., "move the red mug next to the box while keeping it upright." To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robotic manipulation tasks. Specifically, modular rule-based task templates are created to automatically generate robot demonstrations with language instructions, consisting of diverse object shapes and appearances, action types, and motion constraints. We also develop a keypoint-based model 6D-CLIPort to deal with multi-view observations and language input and output a sequence of 6 degrees of freedom (DoF) actions. We hope the new simulator and benchmark will facilitate future research on language-guided robotic manipulation.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

40 of 51 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
28 more
  1. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  2. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  3. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  4. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  5. 2017Embodied Question Answeringcited by 335
  6. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  7. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  8. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  9. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  10. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  11. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  12. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  13. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  14. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  15. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  16. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  17. 2017Grasp Pose Detection in Point Cloudscited by 107
  18. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  19. 2018Reconstruction Network for Video Captioningcited by 98
  20. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  21. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  22. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  23. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  24. 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
  25. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 43
  26. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  27. 2020Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rulecited by 22
  28. 2021Zero-shot Task Adaptation using Natural Languagecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 22; 2025: 20; 2026: 14.

68 papers.

Sort:

By area:

  1. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  2. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  3. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  4. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  5. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  6. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  7. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  8. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  9. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  10. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  11. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  12. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
38 more
  1. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  2. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  3. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  4. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 33
  5. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  6. 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluationcited by 31
  7. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  8. 2024Language-driven Grasp Detectioncited by 26
  9. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  10. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  11. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 20
  12. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 17
  13. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  14. 2023LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulationcited by 16
  15. 2024How and where does CLIP process negation?cited by 16
  16. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  17. 2023Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methodscited by 14
  18. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 13
  19. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12
  20. 2024NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulationcited by 10
  21. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  22. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?cited by 10
  23. 2024CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluationcited by 8
  24. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  25. 2023LEMMA: Learning Language-Conditioned Multi-Robot Manipulationcited by 7
  26. 2024One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skillcited by 7
  27. 2024Contrast Sets for Evaluating Language-Guided Robot Policiescited by 7
  28. 2024RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarioscited by 7
  29. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  30. 2024UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Modelscited by 6
  31. 2025How Do VLAs Effectively Inherit from VLMs?cited by 6
  32. 2024MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Modelscited by 5
  33. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 5
  34. 2025AVA: Attentive VLM Agent for Mastering StarCraft IIcited by 5
  35. 2024Closed Loop Interactive Embodied Reasoning for Robot Manipulationcited by 4
  36. 2022OpenD: A Benchmark for Language-Driven Door and Drawer Openingcited by 3
  37. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learningcited by 3
  38. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3

50 most cited of 68 All 68 (JSON)