# Model Inversion Robustness: Can Transfer Learning Help?

arXiv: https://arxiv.org/abs/2405.05588 · HTML: https://mlsanity.com/paper/2405.05588/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Task2Vec: Task Embedding for Meta-Learning](https://mlsanity.com/paper/1902.03545/) (2019)
- [Few-shot Image Generation with Elastic Weight Consolidation](https://mlsanity.com/paper/2012.02780/) (2020)
- [Variational Model Inversion Attacks](https://mlsanity.com/paper/2201.10787/) (2022)
- [Knowledge-Enriched Distributional Model Inversion Attacks](https://mlsanity.com/paper/2010.04092/) (2020)
- [Learning Meta Face Recognition in Unseen Domains](https://mlsanity.com/paper/2003.07733/) (2020)
- [Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks](https://mlsanity.com/paper/2201.12179/) (2022)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Label-Only Model Inversion Attacks via Boundary Repulsion](https://mlsanity.com/paper/2203.01925/) (2022)
- [Improving Robustness to Model Inversion Attacks via Mutual Information Regularization](https://mlsanity.com/paper/2009.05241/) (2020)
- [Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification](https://mlsanity.com/paper/2106.08808/) (2021)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)

## Built on by (most cited first, 12 of 12)

- [Model Inversion Attacks: A Survey of Approaches and Countermeasures](https://mlsanity.com/paper/2411.10023/) (2024)
- [Theoretical Insights in Model Inversion Robustness and Conditional Entropy Maximization for Collaborative Inference Systems](https://mlsanity.com/paper/2503.00383/) (2025)
- [Trap-MID: Trapdoor-based Defense against Model Inversion Attacks](https://mlsanity.com/paper/2411.08460/) (2024)
- [Rank Matters: Understanding and Defending Model Inversion Attacks via Low-Rank Feature Filtering](https://mlsanity.com/paper/2410.05814/) (2024)
- [Random Erasing vs. Model Inversion: A Promising Defense or a False Hope?](https://mlsanity.com/paper/2409.01062/) (2024)
- [Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessment](https://mlsanity.com/paper/2505.03519/) (2025)
- [Generative Model Inversion Through the Lens of the Manifold Hypothesis](https://mlsanity.com/paper/2509.20177/) (2025)
- [Do Vision-Language Models Leak What They Learn? Adaptive Token-Weighted Model Inversion Attacks](https://mlsanity.com/paper/2508.04097/) (2025)
- [Reducing information dependency does not cause training data privacy. Adversarially non-robust features do](https://mlsanity.com/paper/2607.12354/) (2026)
- [Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching](https://mlsanity.com/paper/2608.16791/) (2026)
- [On the Relationship between Model Quantization and Model Inversion Attacks](https://mlsanity.com/paper/2610.00382/) (2026)
- [Adaptive Model Inversion Attacks Generalize a Privacy-Robustness Tradeoff](https://mlsanity.com/paper/2610.07677/) (2026)

## Abstract (arXiv)

Model Inversion (MI) attacks aim to reconstruct private training data by abusing access to machine learning models. Contemporary MI attacks have achieved impressive attack performance, posing serious threats to privacy. Meanwhile, all existing MI defense methods rely on regularization that is in direct conflict with the training objective, resulting in noticeable degradation in model utility. In this work, we take a different perspective, and propose a novel and simple Transfer Learning-based Defense against Model Inversion (TL-DMI) to render MI-robust models. Particularly, by leveraging TL, we limit the number of layers encoding sensitive information from private training dataset, thereby degrading the performance of MI attack. We conduct an analysis using Fisher Information to justify our method. Our defense is remarkably simple to implement. Without bells and whistles, we show in extensive experiments that TL-DMI achieves state-of-the-art (SOTA) MI robustness. Our code, pre-trained models, demo and inverted data are available at: https://hosytuyen.github.io/projects/TL-DMI