mlsanityData to 7 Oct 2026

Contrast Sets for Evaluating Language-Guided Robot Policies

arXiv 2406.13636 ↗First posted 19 Jun 2024cs.RO cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

38 of 45 references.

References · 38

  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
26 more
  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  5. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  6. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  7. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  8. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  9. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  10. 2021Counterfactual Generative Networkscited by 111
  11. 2022Navigating to Objects in the Real Worldcited by 104
  12. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  13. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  14. 2022Diffusion Models for Counterfactual Explanationscited by 79
  15. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  16. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  17. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  18. 2023Adversarial Counterfactual Visual Explanationscited by 47
  19. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  20. 2023Deploying and Evaluating LLMs to Program Service Mobile Robotscited by 36
  21. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  22. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  23. 2022OCTET: Object-aware Counterfactual Explanationscited by 28
  24. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  25. 2024How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluationcited by 14
  26. 2022Self-Supervised Object Goal Navigation with In-Situ Finetuningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  2. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  3. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7
  4. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Modelscited by 3
  5. 2026SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibrationcited by 2
  6. 2026Active Real-World Factor-Based Evaluation for Generalist Robot Policiescited by 1
  7. 2026Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retentioncited by 0