# RINAS: Training with Dataset Shuffling Can Be General and Fast

arXiv: https://arxiv.org/abs/2312.02368 · HTML: https://mlsanity.com/paper/2312.02368/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 74 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [A Unified Convergence Analysis for Shuffling-Type Gradient Methods](https://mlsanity.com/paper/2002.08246/) (2020)
- [SGD with shuffling: optimal rates without component convexity and large epoch requirements](https://mlsanity.com/paper/2006.06946/) (2020)
- [Permutation-Based SGD: Is Random Optimal?](https://mlsanity.com/paper/2102.09718/) (2021)
- [CosmoFlow: Using Deep Learning to Learn the Universe at Scale](https://mlsanity.com/paper/1808.04728/) (2018)
- [Clairvoyant Prefetching for Distributed Machine Learning I/O](https://mlsanity.com/paper/2101.08734/) (2021)
- [Deep Lake: a Lakehouse for Deep Learning](https://mlsanity.com/paper/2209.10785/) (2022)
- [Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient](https://mlsanity.com/paper/2012.08795/) (2020)
- [Exoshuffle: An Extensible Shuffle Architecture](https://mlsanity.com/paper/2203.05072/) (2022)
- [DeepCAM: A Fully CAM-based Inference Accelerator with Variable Hash Lengths for Energy-efficient Deep Neural Networks](https://mlsanity.com/paper/2302.04712/) (2023)
- [Benchmarking Apache Arrow Flight -- A wire-speed protocol for data transfer, querying and microservices](https://mlsanity.com/paper/2204.03032/) (2022)

## Built on by (most cited first, 3 of 3)

- [EVOS: Efficient Implicit Neural Training via EVOlutionary Selector](https://mlsanity.com/paper/2412.10153/) (2024)
- [Refining Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models](https://mlsanity.com/paper/2408.09621/) (2024)
- [VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets](https://mlsanity.com/paper/2608.22795/) (2026)

## Abstract (arXiv)

Deep learning datasets are expanding at an unprecedented pace, creating new challenges for data processing in model training pipelines. A crucial aspect of these pipelines is dataset shuffling, which significantly improves unbiased learning and convergence accuracy by adhering to the principles of random sampling. However, loading shuffled data for large datasets incurs significant overhead in the deep learning pipeline and severely impacts the end-to-end training throughput. To mitigate this, current deep learning systems often resort to partial dataset shuffling, sacrificing global randomness to maintain acceptable training throughput on large datasets, still leaving global shuffling efficiency issues not fully explored. In this work, we present RINAS, a data loading framework that systematically addresses the performance bottleneck of loading global shuffled datasets. Our key contribution is to offer an intra-batch unordered data fetching approach, which unleashes unexplored parallelism of data loading. We implement RINAS under the PyTorch framework for common dataset libraries HuggingFace and TorchVision. Our experimental results show that RINAS improves the throughput of general language model training and vision model training by up to 59% and 89%, respectively.