mlsanityData to 7 Oct 2026

Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy

arXiv 2410.01345 ↗First posted 2 Oct 2024cs.RO cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

Generalizing language-conditioned robotic policies to new tasks remains a significant challenge, hampered by the lack of suitable simulation benchmarks. In this paper, we address this gap by introducing GemBench, a novel benchmark to assess generalization capabilities of vision-language robotic manipulation policies. GemBench incorporates seven general action primitives and four levels of generalization, spanning novel placements, rigid and articulated objects, and complex long-horizon tasks. We evaluate state-of-the-art approaches on GemBench and also introduce a new method. Our approach 3D-LOTUS leverages rich 3D information for action prediction conditioned on language. While 3D-LOTUS excels in both efficiency and performance on seen tasks, it struggles with novel tasks. To address this, we present 3D-LOTUS++, a framework that integrates 3D-LOTUS's motion planning capabilities with the task planning capabilities of LLMs and the object grounding accuracy of VLMs. 3D-LOTUS++ achieves state-of-the-art performance on novel tasks of GemBench, setting a new standard for generalization in robotic manipulation. The benchmark, codes and trained models are available at https://www.di.ens.fr/willow/research/gembench/.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

49 of 58 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Segment Anythingcited by 8,158
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
37 more
  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  5. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  13. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  14. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  15. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  16. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  17. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  18. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  19. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  20. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  21. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  22. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  23. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  24. 2023Language-Driven Representation Learning for Roboticscited by 163
  25. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  26. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  27. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  28. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  29. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  30. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  31. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  32. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  33. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  34. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  35. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  36. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  37. 2022AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 17.

37 papers.

Sort:

By area:

  1. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  4. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  5. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  6. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  7. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  8. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  9. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  10. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 17
  11. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  12. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12
25 more
  1. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  2. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Enginescited by 6
  3. 2026PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interactioncited by 6
  4. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
  5. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  6. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
  7. 2025Dejavu: Towards Experience Feedback Learning for Embodied Intelligencecited by 3
  8. 2026MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalizationcited by 3
  9. 2025Generalizable Hierarchical Skill Learning via Object-Centric Representationcited by 2
  10. 2026Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Handscited by 2
  11. 2025Gondola: Grounded Vision Language Planning for Robotic Manipulationcited by 1
  12. 2025ResponsibleRobotBench: Benchmarking Responsible Robot Manipulation using Multi-modal Large Language Modelscited by 1
  13. 2026Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Modelscited by 1
  14. 2026BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulationcited by 1
  15. 2025Mini Diffuser: Fast Multi-task Diffusion Policy Training Using Two-level Mini-batchescited by 0
  16. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  17. 2025Intent-Driven LLM Ensemble Planning for Flexible Multi-Robot Disassembly: Demonstration on EV Batteriescited by 0
  18. 2025ArtiBench and ArtiBrain: Benchmarking Generalizable Vision-Language Articulated Object Manipulationcited by 0
  19. 2026HeteroHub: An Applicable Data Management Framework for Heterogeneous Multi-Embodied Agent Systemcited by 0
  20. 2026Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulationcited by 0
  21. 2026RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?cited by 0
  22. 2026MAGMA-GEN: Validated Recovery Supervision from Ambiguous Failures via Counterfactual Re-Executioncited by 0
  23. 2026Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligencecited by 0
  24. 2026LIBERO-MAX: Do Robot Policies Adapt When the World Changes?cited by 0
  25. 2026Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models
  2. 2026GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning
  3. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks
  4. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  5. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  6. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policies
  7. 20263DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints
  8. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Model