# Neural Spectral Decomposition for Dataset Distillation

arXiv: https://arxiv.org/abs/2408.16236 · HTML: https://mlsanity.com/paper/2408.16236/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022)
- [Caption Anything: Interactive Image Description with Diverse Multimodal Controls](https://mlsanity.com/paper/2305.02677/) (2023)
- [ScaIL: Classifier Weights Scaling for Class Incremental Learning](https://mlsanity.com/paper/2001.05755/) (2020)
- [Adaptive Frequency Filters As Efficient Global Token Mixers](https://mlsanity.com/paper/2307.14008/) (2023)
- [Curriculum Dataset Distillation](https://mlsanity.com/paper/2405.09150/) (2024)
- [Single-Image-Based Deep Learning for Segmentation of Early Esophageal Cancer Lesions](https://mlsanity.com/paper/2306.05912/) (2023)

## Built on by (most cited first, 14 of 14)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [DD-Ranking: Rethinking the Evaluation of Dataset Distillation](https://mlsanity.com/paper/2505.13300/) (2025)
- [Distilling Dataset into Neural Field](https://mlsanity.com/paper/2503.04835/) (2025)
- [ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2602.23295/) (2026)
- [TGDD: Trajectory Guided Dataset Distillation with Balanced Distribution](https://mlsanity.com/paper/2512.02469/) (2025)
- [Understanding Dataset Distillation via Spectral Filtering](https://mlsanity.com/paper/2503.01212/) (2025)
- [Dataset Distillation as Data Compression: A Rate-Utility Perspective](https://mlsanity.com/paper/2507.17221/) (2025)
- [Parameterizing Dataset Distillation via Gaussian Splatting](https://mlsanity.com/paper/2509.26219/) (2025)
- [A Discrepancy-Based Perspective on Dataset Condensation](https://mlsanity.com/paper/2509.10367/) (2025)
- [Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation](https://mlsanity.com/paper/2606.20196/) (2026)
- [Distill Video Datasets into Images](https://mlsanity.com/paper/2512.14621/) (2025)
- [Towards Principled Dataset Distillation: A Spectral Distribution Perspective](https://mlsanity.com/paper/2603.01698/) (2026)
- [Fair Dataset Distillation via Cross-Group Barycenter Alignment](https://mlsanity.com/paper/2605.00185/) (2026)

## Abstract (arXiv)

In this paper, we propose Neural Spectrum Decomposition, a generic decomposition framework for dataset distillation. Unlike previous methods, we consider the entire dataset as a high-dimensional observation that is low-rank across all dimensions. We aim to discover the low-rank representation of the entire dataset and perform distillation efficiently. Toward this end, we learn a set of spectrum tensors and transformation matrices, which, through simple matrix multiplication, reconstruct the data distribution. Specifically, a spectrum tensor can be mapped back to the image space by a transformation matrix, and efficient information sharing during the distillation learning process is achieved through pairwise combinations of different spectrum vectors and transformation matrices. Furthermore, we integrate a trajectory matching optimization method guided by a real distribution. Our experimental results demonstrate that our approach achieves state-of-the-art performance on benchmarks, including CIFAR10, CIFAR100, Tiny Imagenet, and ImageNet Subset. Our code are available at urlhttps://github.com/slyang2021/NSD.