# Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications

arXiv: https://arxiv.org/abs/2510.27186 · HTML: https://mlsanity.com/paper/2510.27186/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 73 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [Generative Low-bitwidth Data Free Quantization](https://mlsanity.com/paper/2003.03603/) (2020)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity](https://mlsanity.com/paper/2302.06015/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [CF-ViT: A General Coarse-to-Fine Method for Vision Transformer](https://mlsanity.com/paper/2203.03821/) (2022)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)

## Built on by (most cited first, 4 of 4)

- [Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers](https://mlsanity.com/paper/2606.04373/) (2026)
- [D4C: Data-Free Quantization for Contrastive Language-Image Pre-training Models](https://mlsanity.com/paper/2511.15411/) (2025)
- [Gradient Transformer: Learning to Generate Updates for LLMs](https://mlsanity.com/paper/2605.27591/) (2026)
- [IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves](https://mlsanity.com/paper/2608.24759/) (2026)

## Abstract (arXiv)

Model inversion, which aims to reconstruct the original training data from pre-trained discriminative models, is especially useful when the original training data is unavailable due to privacy, usage rights, or size constraints. However, existing dense inversion methods attempt to reconstruct the entire image area, making them extremely inefficient when inverting high-resolution images from large-scale Vision Transformers (ViTs). We further identify two underlying causes of this inefficiency: the redundant inversion of noisy backgrounds and the unintended inversion of spurious correlations--a phenomenon we term "hallucination" in model inversion. To address these limitations, we propose a novel sparse model inversion strategy, as a plug-and-play extension to speed up existing dense inversion methods with no need for modifying their original loss functions. Specifically, we selectively invert semantic foregrounds while stopping the inversion of noisy backgrounds and potential spurious correlations. Through both theoretical and empirical studies, we validate the efficacy of our approach in achieving significant inversion acceleration (up to 3.79 faster) while maintaining comparable or even enhanced downstream performance in data-free model quantization and data-free knowledge transfer. Code is available at https://github.com/Egg-Hu/SMI.