# Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspective

arXiv: https://arxiv.org/abs/2311.01047 · HTML: https://mlsanity.com/paper/2311.01047/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [A Fourier Perspective on Model Robustness in Computer Vision](https://mlsanity.com/paper/1906.08988/) (2019)
- [Stochastic Activation Pruning for Robust Adversarial Defense](https://mlsanity.com/paper/1803.01442/) (2018)
- [Adversarial Examples Are a Natural Consequence of Test Error in Noise](https://mlsanity.com/paper/1901.10513/) (2019)
- [A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions](https://mlsanity.com/paper/1705.02498/) (2017)
- [Tilted Empirical Risk Minimization](https://mlsanity.com/paper/2007.01162/) (2020)
- [Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations](https://mlsanity.com/paper/1807.01697/) (2018)
- [On the effectiveness of adversarial training against common corruptions](https://mlsanity.com/paper/2103.02325/) (2021)
- [Improved OOD Generalization via Adversarial Training and Pre-training](https://mlsanity.com/paper/2105.11144/) (2021)
- [On Tilted Losses in Machine Learning: Theory and Applications](https://mlsanity.com/paper/2109.06141/) (2021)
- [Consistency Regularization for Adversarial Robustness](https://mlsanity.com/paper/2103.04623/) (2021)
- [Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation](https://mlsanity.com/paper/2110.07858/) (2021)
- [Defending Against Image Corruptions Through Adversarial Augmentations](https://mlsanity.com/paper/2104.01086/) (2021)
- [Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective](https://mlsanity.com/paper/1908.10920/) (2019)
- [Robustness through Data Augmentation Loss Consistency](https://mlsanity.com/paper/2110.11205/) (2021)
- [What Are Effective Labels for Augmented Data? Improving Calibration and Robustness with AutoLabel](https://mlsanity.com/paper/2302.11188/) (2023)
- [Empirical Advocacy of Bio-inspired Models for Robust Image Recognition](https://mlsanity.com/paper/2205.09037/) (2022)
- [Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations](https://mlsanity.com/paper/2202.13074/) (2022)

## Abstract (arXiv)

State-of-the-art techniques for enhancing robustness of deep networks mostly rely on empirical risk minimization with suitable data augmentation. In this paper, we propose a complementary approach motivated by communication theory, aimed at enhancing the signal-to-noise ratio at the output of a neural network layer via neural competition during learning and inference. In addition to standard empirical risk minimization, neurons compete to sparsely represent layer inputs by maximization of a tilted exponential (TEXP) objective function for the layer. TEXP learning can be interpreted as maximum likelihood estimation of matched filters under a Gaussian model for data noise. Inference in a TEXP layer is accomplished by replacing batch norm by a tilted softmax, which can be interpreted as computation of posterior probabilities for the competing signaling hypotheses represented by each neuron. After providing insights via simplified models, we show, by experimentation on standard image datasets, that TEXP learning and inference enhances robustness against noise and other common corruptions, without requiring data augmentation. Further cumulative gains in robustness against this array of distortions can be obtained by appropriately combining TEXP with data augmentation techniques. The code for all our experiments is available at https://github.com/bhagyapuranik/texp_for_robustness.