# Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation

arXiv: https://arxiv.org/abs/2508.06426 · HTML: https://mlsanity.com/paper/2508.06426/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models.
- Builds on 88 of its 100 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Placed in

- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)

## Builds on (most cited first, 30 of 88)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)

## Built on by (most cited first, 30 of 31)

- [Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos](https://mlsanity.com/paper/2510.21571/) (2025)
- [GigaBrain-0: A World Model-Powered Vision-Language-Action Model](https://mlsanity.com/paper/2510.19430/) (2025)
- [LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries](https://mlsanity.com/paper/2601.15197/) (2026)
- [RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation](https://mlsanity.com/paper/2510.23571/) (2025)
- [RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models](https://mlsanity.com/paper/2511.01331/) (2025)
- [PlayWorld: Learning Robot World Models from Autonomous Play](https://mlsanity.com/paper/2603.09030/) (2026)
- [Action Images: End-to-End Policy Learning via Multiview Video Generation](https://mlsanity.com/paper/2604.06168/) (2026)
- [Transport Discrepancy as a Reliability Signal for Vision-Language-Action Models](https://mlsanity.com/paper/2512.01715/) (2025)
- [Limited Linguistic Diversity in Embodied AI Datasets](https://mlsanity.com/paper/2601.03136/) (2026)
- [Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Training](https://mlsanity.com/paper/2604.23121/) (2026)
- [D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI](https://mlsanity.com/paper/2510.05684/) (2025)
- [Fidelity-Aware Data Composition for Robust Robot Generalization](https://mlsanity.com/paper/2509.24797/) (2025)
- [UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning](https://mlsanity.com/paper/2510.10642/) (2025)
- [Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds](https://mlsanity.com/paper/2603.18532/) (2026)
- [EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training](https://mlsanity.com/paper/2604.20012/) (2026)
- [DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimization](https://mlsanity.com/paper/2605.17486/) (2026)
- [Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking](https://mlsanity.com/paper/2602.24143/) (2026)
- [Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents](https://mlsanity.com/paper/2604.17019/) (2026)
- [Active Real-World Factor-Based Evaluation for Generalist Robot Policies](https://mlsanity.com/paper/2607.14439/) (2026)
- [CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models](https://mlsanity.com/paper/2605.10903/) (2026)
- [Mitigating State Aliasing in Vision-Language-Action Models via Inverse Dynamics Learning](https://mlsanity.com/paper/2605.29577/) (2026)
- [Perturbation-Based Epistemic Uncertainty for Failure Detection in Vision-Language-Action Models](https://mlsanity.com/paper/2606.20754/) (2026)
- [EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories](https://mlsanity.com/paper/2607.00020/) (2026)
- [SIEVE: Structure-Aware Data Selection for Imitation Learning with VLA Models](https://mlsanity.com/paper/2607.06442/) (2026)
- [Semantic Anchoring for Robotic Action Representations](https://mlsanity.com/paper/2607.13597/) (2026)
- [Closing the Lab-to-Store Gap: A Data-Efficient Post-Training and Experience-Driven Learning VLA Framework for Retail Humanoids](https://mlsanity.com/paper/2607.20345/) (2026)
- [Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation](https://mlsanity.com/paper/2607.21582/) (2026)
- [MaskVLA: Visual Masking Against Trajectory Overfitting of Vision-Language-Action Model](https://mlsanity.com/paper/2609.23565/) (2026)
- [Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead](https://mlsanity.com/paper/2609.37165/) (2026)
- [Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance](https://mlsanity.com/paper/2609.38616/) (2026)

## Abstract (arXiv)

Generalist robot policies trained on large-scale datasets such as Open X-Embodiment (OXE) demonstrate strong performance across a wide range of tasks. However, they often struggle to generalize beyond the distribution of their training data. In this paper, we investigate the underlying cause of this limited generalization capability. We identify shortcut learning -- the reliance on task-irrelevant features -- as a key impediment to generalization. Through comprehensive theoretical and empirical analysis, we uncover two primary contributors to shortcut learning: (1) limited diversity within individual sub-datasets, and (2) significant distributional disparities across sub-datasets, leading to dataset fragmentation. These issues arise from the inherent structure of large-scale datasets like OXE, which are typically composed of multiple sub-datasets collected independently across varied environments and embodiments. Our findings provide critical insights into dataset collection strategies that can reduce shortcut learning and enhance the generalization ability of generalist robot policies. Moreover, in scenarios where acquiring new large-scale data is impractical, we demonstrate that carefully selected robotic data augmentation strategies can effectively reduce shortcut learning in existing offline datasets, thereby improving generalization capabilities of generalist robot policies, e.g., π0, in both simulation and real-world environments. More information at https://lucky-light-sun.github.io/proj/shortcut-learning-in-grps/.