mlsanityData to 7 Oct 2026

THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

arXiv 2402.08191 ↗First posted 13 Feb 2024cs.RO cs.AI cs.LGcited by 128

Not on the map. Why?

Cited by 45 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup. We present THE COLOSSEUM, a novel simulation benchmark, with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, and backgrounds; we also vary lighting, distractors, physical properties perturbations and camera pose. Using THE COLOSSEUM, we compare 5 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors. When multiple perturbations are applied in unison, the success rate degrades ≥75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated (barR2 = 0.614) to similar perturbations in real-world experiments. We open source code for others to use THE COLOSSEUM, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that THE COLOSSEUM will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation. See https://robot-colosseum.github.io/ for more details.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 60 of its 78 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

60 of 78 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
38 more
  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  13. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  14. 2020Reinforcement Learning with Augmented Datacited by 452
  15. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  16. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  17. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  18. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  19. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  20. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  21. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  22. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  23. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  24. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  25. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  26. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  27. 2019Active Domain Randomizationcited by 161
  28. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  29. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  30. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  31. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  32. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  33. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  34. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  35. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  36. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 79
  37. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  38. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 57; 2026: 53.

128 papers.

Sort:

By area:

  1. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  2. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  3. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  4. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  5. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  7. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  8. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  9. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  10. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  11. 2025Evaluating Gemini Robotics Policies in a Veo World Simulatorcited by 50
  12. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
38 more
  1. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  2. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  3. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  4. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  5. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  6. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 36
  7. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  8. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  9. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  10. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  11. 2025RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluationcited by 29
  12. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  13. 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27
  14. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 25
  15. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  16. 2025SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrationscited by 23
  17. 2025What Matters in Learning from Large-Scale Datasets for Robot Manipulationcited by 20
  18. 2024Generative Image as Action Modelscited by 18
  19. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 17
  20. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 17
  21. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  22. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 15
  23. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 15
  24. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 15
  25. 2025Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stoppingcited by 14
  26. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  27. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Modelscited by 14
  28. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 13
  29. 2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
  30. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
  31. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12
  32. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  33. 20243D-MVP: 3D Multiview Pretraining for Robotic Manipulationcited by 10
  34. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  35. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 10
  36. 2025Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learningcited by 9
  37. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  38. 2025CORE: Full-Path Evaluation of LLM Agents Beyond Final Statecited by 8

50 most cited of 128 All 128 (JSON)