mlsanityData to 7 Oct 2026

The MAGICAL Benchmark for Robust Imitation

arXiv 2011.00401 ↗First posted 1 Nov 2020cs.LG cs.AIcited by 30

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation Learning (IL) algorithms are typically evaluated in the same environment that was used to create demonstrations. This rewards precise reproduction of demonstrations in one particular environment, but provides little information about how robustly an algorithm can generalise the demonstrator's intent to substantially different deployment settings. This paper presents the MAGICAL benchmark suite, which permits systematic evaluation of generalisation by quantifying robustness to different kinds of distribution shift that an IL algorithm is likely to encounter in practice. Using the MAGICAL suite, we confirm that existing IL algorithms overfit significantly to the context in which demonstrations are provided. We also show that standard methods for reducing overfitting are effective at creating narrow perceptual invariances, but are not sufficient to enable transfer to contexts that require substantially different behaviour, which suggests that new approaches will be needed in order to robustly generalise demonstrator intent. Code and data for the MAGICAL suite is available at https://github.com/qxcv/magical/.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 1 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 5; 2024: 6; 2025: 2; 2026: 1.

30 papers.

Sort:

By area:

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  5. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  6. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  7. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  8. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  9. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  10. 2024Equivariant Diffusion Policycited by 61
  11. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  12. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
18 more
  1. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  2. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
  3. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  4. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  5. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  6. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  7. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  8. 2022Learn what matters: cross-domain imitation learning with task-relevant embeddingscited by 14
  9. 2021The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulationcited by 11
  10. 2021Multi-Task Learning with Sequence-Conditioned Transporter Networkscited by 8
  11. 2021Interactive Inverse Reinforcement Learning for Cooperative Gamescited by 7
  12. 2022Environment Design for Inverse Reinforcement Learningcited by 4
  13. 2024Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Reviewcited by 1
  14. 2024RILe: Reinforced Imitation Learningcited by 1
  15. 2025Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusioncited by 1
  16. 2022Entropy-driven Unsupervised Keypoint Representation Learning in Videoscited by 0
  17. 2025Moving Out: Physically-grounded Human-AI Collaborationcited by 0
  18. 2026Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Imitation Learning from Imperfect Demonstration
  2. 2025Quantifying Generalisation in Imitation Learning
  3. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  4. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
  5. 2022Robust Learning from Observation with Model Misspecification
  6. 2022Imitation Learning by Estimating Expertise of Demonstrators
  7. 2022Robust Imitation Learning from Corrupted Demonstrations
  8. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections