# Maximum and Leaky Maximum Propagation

arXiv: https://arxiv.org/abs/2105.10277 · HTML: https://mlsanity.com/paper/2105.10277/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 84 references that are in the map.
- First posted on arXiv 21 May 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems](https://mlsanity.com/paper/1905.04413/) (2019)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [Revisiting Batch Normalization for Improving Corruption Robustness](https://mlsanity.com/paper/2010.03630/) (2020)
- [Unsupervised Representation Learning for Gaze Estimation](https://mlsanity.com/paper/1911.06939/) (2019)
- [The AGI Containment Problem](https://mlsanity.com/paper/1604.00545/) (2016)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [PupilNet: Convolutional Neural Networks for Robust Pupil Detection](https://mlsanity.com/paper/1601.04902/) (2016)
- [Learning Collision-Free Space Detection from Stereo Images: Homography Matrix Brings Better Data Augmentation](https://mlsanity.com/paper/2012.07890/) (2020)
- [ElSe: Ellipse Selection for Robust Pupil Detection in Real-World Environments](https://mlsanity.com/paper/1511.06575/) (2015)
- [PupilNet v2.0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection](https://mlsanity.com/paper/1711.00112/) (2017)
- [Fully Convolutional Neural Networks for Raw Eye Tracking Data Segmentation, Generation, and Reconstruction](https://mlsanity.com/paper/2002.10905/) (2020)
- [Rotated Ring, Radial and Depth Wise Separable Radial Convolutions](https://mlsanity.com/paper/2010.00873/) (2020)
- [Eye movement velocity and gaze data generator for evaluation, robustness testing and assess of eye tracking software and visualization tools](https://mlsanity.com/paper/1808.09296/) (2018)
- [Learning to Validate the Quality of Detected Landmarks](https://mlsanity.com/paper/1901.10143/) (2019)
- [Weight and Gradient Centralization in Deep Neural Networks](https://mlsanity.com/paper/2010.00866/) (2020)
- [Fast camera focus estimation for gaze-based focus control](https://mlsanity.com/paper/1711.03306/) (2017)
- [Eye movement simulation and detector creation to reduce laborious parameter adjustments](https://mlsanity.com/paper/1804.00970/) (2018)
- [Training Decision Trees as Replacement for Convolution Layers](https://mlsanity.com/paper/1905.10073/) (2019)
- [The Gaze and Mouse Signal as additional Source for User Fingerprints in Browser Applications](https://mlsanity.com/paper/2101.03793/) (2021)
- [Multi Layer Neural Networks as Replacement for Pooling Operations](https://mlsanity.com/paper/2006.06969/) (2020)

## Built on by (most cited first, 4 of 4)

- [Pistol: Pupil Invisible Supportive Tool to extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D as well as 3D Gaze](https://mlsanity.com/paper/2201.06799/) (2022)
- [Tensor Normalization and Full Distribution Training](https://mlsanity.com/paper/2109.02345/) (2021)
- [GroupGazer: A Tool to Compute the Gaze per Participant in Groups with integrated Calibration to Map the Gaze Online to a Screen or Beamer Projection](https://mlsanity.com/paper/2201.07692/) (2022)
- [Technical Report: Combining knowledge from Transfer Learning during training and Wide Resnets](https://mlsanity.com/paper/2206.09697/) (2022)

## Abstract (arXiv)

In this work, we present an alternative to conventional residual connections, which is inspired by maxout nets. This means that instead of the addition in residual connections, our approach only propagates the maximum value or, in the leaky formulation, propagates a percentage of both. In our evaluation, we show on different public data sets that the presented approaches are comparable to the residual connections and have other interesting properties, such as better generalization with a constant batch normalization, faster learning, and also the possibility to generalize without additional activation functions. In addition, the proposed approaches work very well if ensembles together with residual networks are formed. https://atreus.informatik.uni-tuebingen.de/seafile/d/8e2ab8c3fdd444e1a135/?p=%2FMaximumPropagation&mode=list