# PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors

arXiv: https://arxiv.org/abs/2511.09905 · HTML: https://mlsanity.com/paper/2511.09905/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm](https://mlsanity.com/paper/2312.03526/) (2023)
- [D4M: Dataset Distillation via Disentangled Diffusion Model](https://mlsanity.com/paper/2407.15138/) (2024)
- [Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching](https://mlsanity.com/paper/2311.17950/) (2023)
- [Elucidating the Design Space of Dataset Condensation](https://mlsanity.com/paper/2404.13733/) (2024)
- [A Label is Worth a Thousand Images in Dataset Distillation](https://mlsanity.com/paper/2406.10485/) (2024)
- [No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"](https://mlsanity.com/paper/2209.14987/) (2022)
- [DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation](https://mlsanity.com/paper/2411.19946/) (2024)
- [Dataset Distillation via Committee Voting](https://mlsanity.com/paper/2501.07575/) (2025)
- [A Fairness Analysis on Private Aggregation of Teacher Ensembles](https://mlsanity.com/paper/2109.08630/) (2021)
- [DD-Ranking: Rethinking the Evaluation of Dataset Distillation](https://mlsanity.com/paper/2505.13300/) (2025)
- [Dataset Distillation in Latent Space](https://mlsanity.com/paper/2311.15547/) (2023)
- [Direct Ascent Synthesis: Revealing Hidden Generative Capabilities in Discriminative Models](https://mlsanity.com/paper/2502.07753/) (2025)
- [Robust Dataset Distillation by Matching Adversarial Trajectories](https://mlsanity.com/paper/2503.12069/) (2025)

## Abstract (arXiv)

Dataset distillation (DD) promises compact yet faithful synthetic data, but existing approaches often inherit the inductive bias of a single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogeneous samples, reducing intra-class diversity and limiting generalization. We present PRISM (PRIors from diverse Source Models), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and a stochastic subset for batch-normalization (BN) alignment. On ImageNet-1K, PRISM consistently and reproducibly outperforms single-teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between features. We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch formation scheme for fast parallel synthesis. Code will be released after the review period.