Data to 7 Oct 2026

The HSIC Bottleneck: Deep Learning without Back-Propagation

arXiv 1908.01580 ↗First posted 5 Aug 2019cs.LG stat.MLcited by 77

Not on the map. Why?

Abstract (arXiv)

We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 13; 2022: 8; 2023: 17; 2024: 17; 2025: 12; 2026: 9.

77 papers.

Sort:

By area:

2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
2021Independent Prototype Propagation for Zero-Shot Compositionalitycited by 42
2021Causal Effect Inference for Structured Treatmentscited by 38
2021Semi-supervised Meta-learning with Disentanglement for Domain-generalised Medical Image Segmentationcited by 36
2023Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learningcited by 32
2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
2021Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustnesscited by 20
2024Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Modelscited by 18
2021An Information Theory-inspired Strategy for Automatic Network Pruningcited by 17

38 more

2023Orthogonal Annotation Benefits Barely-supervised Medical Image Segmentationcited by 15
2023DualHSIC: HSIC-Bottleneck and Alignment for Continual Learningcited by 15
2022Pruning Adversarially Robust Neural Networks without Adversarial Examplescited by 12
2023MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustnesscited by 12
2023One Forward is Enough for Neural Network Training via Likelihood Ratio Methodcited by 11
2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 11
2024Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embeddingcited by 9
2024Cauchy-Schwarz Divergence Information Bottleneck for Regressioncited by 9
2023Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Surveycited by 8
2024End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Trainingcited by 8
2024LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Drivingcited by 8
2021Data Augmentation for Graph Convolutional Network on Semi-Supervised Classificationcited by 7
2026ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Modelscited by 7
2024FFCL: Forward-Forward Net with Cortical Loops, Training and Inference on Edge Without Backpropagationcited by 6
2021Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methodscited by 5
2021Expert System Gradient Descent Style Training: Development of a Defensible Artificial Intelligence Techniquecited by 5
2024The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Modelscited by 5
2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
2023Diversity-enhancing Generative Network for Few-shot Hypothesis Adaptationcited by 4
2023Module-wise Training of Neural Networks via the Minimizing Movement Schemecited by 4
2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
2023Text Representation Distillation via Information Bottleneck Principlecited by 3
2024FLOPS: Forward Learning with OPtimal Samplingcited by 3
2022Deep Layer-wise Networks Have Closed-Form Weightscited by 2
2022A Robust Backpropagation-Free Framework for Imagescited by 2
2023IB-RAR: Information Bottleneck as Regularizer for Adversarial Robustnesscited by 2
2023Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learningcited by 2
2024Exploring Heterogeneity and Uncertainty for Graph-based Cognitive Diagnosis Models in Intelligent Educationcited by 2
2025H-SPLID: HSIC-based Saliency Preserving Latent Information Decompositioncited by 2
2026Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answeringcited by 2
2021Intra-Model Collaborative Learning of Neural Networkscited by 1
2021Information Bottleneck-Based Hebbian Learning Rule Naturally Ties Working Memory and Synaptic Updatescited by 1
2022Deep Regression Ensemblescited by 1
2022HSIC-InfoGAN: Learning Unsupervised Disentangled Representations by Maximising Approximated Mutual Informationcited by 1
2022Measuring Statistical Dependencies via Maximum Norm and Characteristic Functionscited by 1
2022Block-wise Training of Residual Networks via the Minimizing Movement Schemecited by 1
2023Gate Recurrent Unit Network based on Hilbert-Schmidt Independence Criterion for State-of-Health Estimationcited by 1
2023SLM: End-to-end Feature Selection via Sparse Learnable Maskscited by 1

50 most cited of 77 All 77 (JSON)