# Feedback-Gated Rectified Linear Units

arXiv: https://arxiv.org/abs/2301.02610 · HTML: https://mlsanity.com/paper/2301.02610/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 20 references that are in the map.
- First posted on arXiv 6 Jan 2023.

## Builds on (most cited first, 2 of 2)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)

## Abstract (arXiv)

Feedback connections play a prominent role in the human brain but have not received much attention in artificial neural network research. Here, a biologically inspired feedback mechanism which gates rectified linear units is proposed. On the MNIST dataset, autoencoders with feedback show faster convergence, better performance, and more robustness to noise compared to their counterparts without feedback. Some benefits, although less pronounced and less consistent, can be observed when networks with feedback are applied on the CIFAR-10 dataset.