# Adversarial Training Reduces Information and Improves Transferability

arXiv: https://arxiv.org/abs/2007.11259 · HTML: https://mlsanity.com/paper/2007.11259/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training](https://mlsanity.com/paper/2102.04716/) (2021)
- [A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks](https://mlsanity.com/paper/2106.02105/) (2021)
- [Survey on Adversarial Attack and Defense for Medical Image Analysis: Methods and Challenges](https://mlsanity.com/paper/2303.14133/) (2023)
- [Adversarial Perturbations Are Not So Weird: Entanglement of Robust and Non-Robust Features in Neural Network Classifiers](https://mlsanity.com/paper/2102.05110/) (2021)
- [MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier](https://mlsanity.com/paper/2209.11549/) (2022)
- [Exploring the Connection between Robust and Generative Models](https://mlsanity.com/paper/2304.04033/) (2023)
- [Representation Learning with Conditional Information Flow Maximization](https://mlsanity.com/paper/2406.05510/) (2024)
- [On the Properties of Adversarially-Trained CNNs](https://mlsanity.com/paper/2203.09243/) (2022)
- [Designing an attack-defense game: how to increase robustness of financial transaction models via a competition](https://mlsanity.com/paper/2308.11406/) (2023)
- [ForecastGAN: A Decomposition-Based Adversarial Framework for Multi-Horizon Time Series Forecasting](https://mlsanity.com/paper/2511.04445/) (2025)

## Abstract (arXiv)

Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility. The latter property may seem counter-intuitive as it is widely accepted by the community that classification models should only capture the minimal information (features) required for the task. Motivated by this discrepancy, we investigate the dual relationship between Adversarial Training and Information Theory. We show that the Adversarial Training can improve linear transferability to new tasks, from which arises a new trade-off between transferability of representations and accuracy on the source task. We validate our results employing robust networks trained on CIFAR-10, CIFAR-100 and ImageNet on several datasets. Moreover, we show that Adversarial Training reduces Fisher information of representations about the input and of the weights about the task, and we provide a theoretical argument which explains the invertibility of deterministic networks without violating the principle of minimality. Finally, we leverage our theoretical insights to remarkably improve the quality of reconstructed images through inversion.