# PEEL the Layers and Find Yourself: Revisiting Inference-time Data Leakage for Residual Neural Networks

arXiv: https://arxiv.org/abs/2504.06410 · HTML: https://mlsanity.com/paper/2504.06410/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Evaluating Gradient Inversion Attacks and Defenses in Federated Learning](https://mlsanity.com/paper/2112.00059/) (2021)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [CAFE: Catastrophic Data Leakage in Vertical Federated Learning](https://mlsanity.com/paper/2110.15122/) (2021)
- [Variational Model Inversion Attacks](https://mlsanity.com/paper/2201.10787/) (2022)
- [Split Learning for collaborative deep learning in healthcare](https://mlsanity.com/paper/1912.12115/) (2019)
- [Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks](https://mlsanity.com/paper/2310.19909/) (2023)
- [Knowledge-Enriched Distributional Model Inversion Attacks](https://mlsanity.com/paper/2010.04092/) (2020)
- [Re-thinking Model Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/2304.01669/) (2023)
- [An Inexact Augmented Lagrangian Framework for Nonconvex Optimization with Nonlinear Constraints](https://mlsanity.com/paper/1906.11357/) (2019)
- [Face.evoLVe: A High-Performance Face Recognition Library](https://mlsanity.com/paper/2107.08621/) (2021)
- [Inverting face embeddings with convolutional neural networks](https://mlsanity.com/paper/1606.04189/) (2016)
- [Rate-improved Inexact Augmented Lagrangian Method for Constrained Nonconvex Optimization](https://mlsanity.com/paper/2007.01284/) (2020)
- [Privacy and Efficiency of Communications in Federated Split Learning](https://mlsanity.com/paper/2301.01824/) (2023)
- [Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs](https://mlsanity.com/paper/2105.12049/) (2021)
- [DeepReShape: Redesigning Neural Networks for Efficient Private Inference](https://mlsanity.com/paper/2304.10593/) (2023)
- [NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning](https://mlsanity.com/paper/2111.13984/) (2021)
- [MPC-Pipe: an Efficient Pipeline Scheme for Secure Multi-party Machine Learning Inference](https://mlsanity.com/paper/2209.13643/) (2022)
- [Cached and Confused: Web Cache Deception in the Wild](https://mlsanity.com/paper/1912.10190/) (2019)

## Abstract (arXiv)

This paper explores inference-time data leakage risks of deep neural networks (NNs), where a curious and honest model service provider is interested in retrieving users' private data inputs solely based on the model inference results. Particularly, we revisit residual NNs due to their popularity in computer vision and our hypothesis that residual blocks are a primary cause of data leakage owing to the use of skip connections. By formulating inference-time data leakage as a constrained optimization problem, we propose a novel backward feature inversion method, PEEL, which can effectively recover block-wise input features from the intermediate output of residual NNs. The surprising results in high-quality input data recovery can be explained by the intuition that the output from these residual blocks can be considered as a noisy version of the input and thus the output retains sufficient information for input recovery. We demonstrate the effectiveness of our layer-by-layer feature inversion method on facial image datasets and pre-trained classifiers. Our results show that PEEL outperforms the state-of-the-art recovery methods by an order of magnitude when evaluated by mean squared error (MSE). The code is available at hrefhttps://github.com/Huzaifa-Arif/PEELhttps://github.com/Huzaifa-Arif/PEEL