# Putting An End to End-to-End: Gradient-Isolated Learning of Representations

arXiv: https://arxiv.org/abs/1905.11786 · HTML: https://mlsanity.com/paper/1905.11786/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 30 of 86)

- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [A Good Image Generator Is What You Need for High-Resolution Video Synthesis](https://mlsanity.com/paper/2104.15069/) (2021)
- [Uncovering mesa-optimization algorithms in Transformers](https://mlsanity.com/paper/2309.05858/) (2023)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Hebbian Deep Learning Without Feedback](https://mlsanity.com/paper/2209.11883/) (2022)
- [Reconsidering Representation Alignment for Multi-view Clustering](https://mlsanity.com/paper/2103.07738/) (2021)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Masked Image Modeling with Local Multi-Scale Reconstruction](https://mlsanity.com/paper/2303.05251/) (2023)
- [CLCC: Contrastive Learning for Color Constancy](https://mlsanity.com/paper/2106.04989/) (2021)
- [Self-supervised Representation Learning with Relative Predictive Coding](https://mlsanity.com/paper/2103.11275/) (2021)
- [Can Forward Gradient Match Backpropagation?](https://mlsanity.com/paper/2306.06968/) (2023)
- [Blockwise Self-Supervised Learning at Scale](https://mlsanity.com/paper/2302.01647/) (2023)
- [FedImpro: Measuring and Improving Client Update in Federated Learning](https://mlsanity.com/paper/2402.07011/) (2024)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [The Trifecta: Three simple techniques for training deeper Forward-Forward networks](https://mlsanity.com/paper/2311.18130/) (2023)
- [Meta-Learning Bidirectional Update Rules](https://mlsanity.com/paper/2104.04657/) (2021)
- [Noisy student-teacher training for robust keyword spotting](https://mlsanity.com/paper/2106.01604/) (2021)
- [Contrastive Predictive Coding for Anomaly Detection](https://mlsanity.com/paper/2107.07820/) (2021)
- [Momentum Auxiliary Network for Supervised Local Learning](https://mlsanity.com/paper/2407.05623/) (2024)
- [Production federated keyword spotting via distillation, filtering, and joint federated-centralized training](https://mlsanity.com/paper/2204.06322/) (2022)
- [Improving equilibrium propagation without weight symmetry through Jacobian homeostasis](https://mlsanity.com/paper/2309.02214/) (2023)
- [Scaling Supervised Local Learning with Augmented Auxiliary Networks](https://mlsanity.com/paper/2402.17318/) (2024)
- [AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learning](https://mlsanity.com/paper/2112.01637/) (2021)
- [First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization](https://mlsanity.com/paper/2205.12381/) (2022)
- [A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions](https://mlsanity.com/paper/2306.02149/) (2023)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation](https://mlsanity.com/paper/2503.24322/) (2025)

## Abstract (arXiv)

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.