# The HSIC Bottleneck: Deep Learning without Back-Propagation

arXiv: https://arxiv.org/abs/1908.01580 · HTML: https://mlsanity.com/paper/1908.01580/

## Key facts

- Cited by 77 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 5 Aug 2019.

## Built on by (most cited first, 30 of 77)

- [Self-Supervised Learning with Kernel Dependence Maximization](https://mlsanity.com/paper/2106.08320/) (2021)
- [Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning](https://mlsanity.com/paper/2506.02867/) (2025)
- [Do Vision and Language Encoders Represent the World Similarly?](https://mlsanity.com/paper/2401.05224/) (2024)
- [Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts](https://mlsanity.com/paper/2002.04013/) (2020)
- [Independent Prototype Propagation for Zero-Shot Compositionality](https://mlsanity.com/paper/2106.00305/) (2021)
- [Causal Effect Inference for Structured Treatments](https://mlsanity.com/paper/2106.01939/) (2021)
- [Semi-supervised Meta-learning with Disentanglement for Domain-generalised Medical Image Segmentation](https://mlsanity.com/paper/2106.13292/) (2021)
- [Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning](https://mlsanity.com/paper/2309.05911/) (2023)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness](https://mlsanity.com/paper/2106.02734/) (2021)
- [Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models](https://mlsanity.com/paper/2402.19465/) (2024)
- [An Information Theory-inspired Strategy for Automatic Network Pruning](https://mlsanity.com/paper/2108.08532/) (2021)
- [Orthogonal Annotation Benefits Barely-supervised Medical Image Segmentation](https://mlsanity.com/paper/2303.13090/) (2023)
- [DualHSIC: HSIC-Bottleneck and Alignment for Continual Learning](https://mlsanity.com/paper/2305.00380/) (2023)
- [Pruning Adversarially Robust Neural Networks without Adversarial Examples](https://mlsanity.com/paper/2210.04311/) (2022)
- [MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness](https://mlsanity.com/paper/2312.04960/) (2023)
- [One Forward is Enough for Neural Network Training via Likelihood Ratio Method](https://mlsanity.com/paper/2305.08960/) (2023)
- [Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment](https://mlsanity.com/paper/2409.19425/) (2024)
- [Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding](https://mlsanity.com/paper/2401.09067/) (2024)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2403.08215/) (2024)
- [Data Augmentation for Graph Convolutional Network on Semi-Supervised Classification](https://mlsanity.com/paper/2106.08848/) (2021)
- [ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models](https://mlsanity.com/paper/2605.24011/) (2026)
- [FFCL: Forward-Forward Net with Cortical Loops, Training and Inference on Edge Without Backpropagation](https://mlsanity.com/paper/2405.12443/) (2024)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Expert System Gradient Descent Style Training: Development of a Defensible Artificial Intelligence Technique](https://mlsanity.com/paper/2103.04314/) (2021)
- [The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Models](https://mlsanity.com/paper/2410.16672/) (2024)
- [Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation](https://mlsanity.com/paper/2511.17579/) (2025)

## Abstract (arXiv)

We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance.