# Dataset Quantization

arXiv: https://arxiv.org/abs/2308.10524 · HTML: https://mlsanity.com/paper/2308.10524/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 67 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 30 of 52)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)

## Built on by (most cited first, 30 of 48)

- [Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content](https://mlsanity.com/paper/2410.08260/) (2024)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching](https://mlsanity.com/paper/2310.05773/) (2023)
- [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024)
- [Does Graph Distillation See Like Vision Dataset Counterpart?](https://mlsanity.com/paper/2310.09192/) (2023)
- [Elucidating the Design Space of Dataset Condensation](https://mlsanity.com/paper/2404.13733/) (2024)
- [Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning](https://mlsanity.com/paper/2502.01968/) (2025)
- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [Improving Data Efficiency via Curating LLM-Driven Rating Systems](https://mlsanity.com/paper/2410.10877/) (2024)
- [Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching](https://mlsanity.com/paper/2402.05011/) (2024)
- [SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching](https://mlsanity.com/paper/2406.18561/) (2024)
- [Rethinking Data Selection at Scale: Random Selection is Almost All You Need](https://mlsanity.com/paper/2410.09335/) (2024)
- [Dataset Condensation via Generative Model](https://mlsanity.com/paper/2309.07698/) (2023)
- [DANCE: Dual-View Distribution Alignment for Dataset Condensation](https://mlsanity.com/paper/2406.01063/) (2024)
- [CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs](https://mlsanity.com/paper/2408.02193/) (2024)
- [Dataset Quantization with Active Learning based Adaptive Sampling](https://mlsanity.com/paper/2407.07268/) (2024)
- [Can pre-trained models assist in dataset distillation?](https://mlsanity.com/paper/2310.03295/) (2023)
- [FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models](https://mlsanity.com/paper/2403.16379/) (2024)
- [Dataset Distillation in Latent Space](https://mlsanity.com/paper/2311.15547/) (2023)
- [Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning](https://mlsanity.com/paper/2509.23873/) (2025)
- [Scale Efficient Training for Large Datasets](https://mlsanity.com/paper/2503.13385/) (2025)
- [Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning](https://mlsanity.com/paper/2508.21589/) (2025)
- [One Category One Prompt: Dataset Distillation using Diffusion Models](https://mlsanity.com/paper/2403.07142/) (2024)
- [Data-efficient LLM Fine-tuning for Code Generation](https://mlsanity.com/paper/2504.12687/) (2025)
- [GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder](https://mlsanity.com/paper/2505.11882/) (2025)
- [Dataset Condensation with Color Compensation](https://mlsanity.com/paper/2508.01139/) (2025)
- [Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning](https://mlsanity.com/paper/2508.04329/) (2025)
- [ATOM: Attention Mixer for Efficient Dataset Distillation](https://mlsanity.com/paper/2405.01373/) (2024)
- [CONCORD: Concept-Informed Diffusion for Dataset Distillation](https://mlsanity.com/paper/2505.18358/) (2025)

## Abstract (arXiv)

State-of-the-art deep neural networks are trained with large amounts (millions or even billions) of data. The expensive computation and memory costs make it difficult to train them on limited hardware resources, especially for recent popular large language models (LLM) and computer vision models (CV). Recent popular dataset distillation methods are thus developed, aiming to reduce the number of training samples via synthesizing small-scale datasets via gradient matching. However, as the gradient calculation is coupled with the specific network architecture, the synthesized dataset is biased and performs poorly when used for training unseen architectures. To address these limitations, we present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets which can be used for training any neural network architectures. Extensive experiments demonstrate that DQ is able to generate condensed small datasets for training unseen network architectures with state-of-the-art compression ratios for lossless model training. To the best of our knowledge, DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio. Notably, with 60% data from ImageNet and 20% data from Alpaca's instruction tuning data, the models can be trained with negligible or no performance drop for both vision tasks (including classification, semantic segmentation, and object detection) as well as language tasks (including instruction tuning tasks such as BBH and DROP).