# Unsupervised End-to-End Training with a Self-Defined Target

arXiv: https://arxiv.org/abs/2403.12116 · HTML: https://mlsanity.com/paper/2403.12116/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 76 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias](https://mlsanity.com/paper/2006.03824/) (2020)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Hebbian Deep Learning Without Feedback](https://mlsanity.com/paper/2209.11883/) (2022)
- [Unsupervised Learning by Competing Hidden Units](https://mlsanity.com/paper/1806.10181/) (2018)
- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input](https://mlsanity.com/paper/1905.13633/) (2019)
- [EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementations](https://mlsanity.com/paper/2010.07859/) (2020)
- [SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks](https://mlsanity.com/paper/2107.05747/) (2021)
- [Training an Ising Machine with Equilibrium Propagation](https://mlsanity.com/paper/2305.18321/) (2023)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Blockwise Self-Supervised Learning at Scale](https://mlsanity.com/paper/2302.01647/) (2023)
- [Hebbian learning with gradients: Hebbian convolutional neural networks with modern deep learning frameworks](https://mlsanity.com/paper/2107.01729/) (2021)
- [Improving Image Autoencoder Embeddings with Perceptual Loss](https://mlsanity.com/paper/2001.03444/) (2020)
- [Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop](https://mlsanity.com/paper/2212.04614/) (2022)
- [Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16236/) (2023)
- [Activation Learning by Local Competitions](https://mlsanity.com/paper/2209.13400/) (2022)
- [Unsupervised representation learning using convolutional and stacked auto-encoders: a domain and cross-domain feature space analysis](https://mlsanity.com/paper/1811.00473/) (2018)

## Built on by (most cited first, 1 of 1)

- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)

## Abstract (arXiv)

Designing algorithms for versatile AI hardware that can learn on the edge using both labeled and unlabeled data is challenging. Deep end-to-end training methods incorporating phases of self-supervised and supervised learning are accurate and adaptable to input data but self-supervised learning requires even more computational and memory resources than supervised learning, too high for current embedded hardware. Conversely, unsupervised layer-by-layer training, such as Hebbian learning, is more compatible with existing hardware but does not integrate well with supervised learning. To address this, we propose a method enabling networks or hardware designed for end-to-end supervised learning to also perform high-performance unsupervised learning by adding two simple elements to the output layer: Winner-Take-All (WTA) selectivity and homeostasis regularization. These mechanisms introduce a "self-defined target" for unlabeled data, allowing purely unsupervised training for both fully-connected and convolutional layers using backpropagation or equilibrium propagation on datasets like MNIST (up to 99.2%), Fashion-MNIST (up to 90.3%), and SVHN (up to 81.5%). We extend this method to semi-supervised learning, adjusting targets based on data type, achieving 96.6% accuracy with only 600 labeled MNIST samples in a multi-layer perceptron. Our results show that this approach can effectively enable networks and hardware initially dedicated to supervised learning to also perform unsupervised learning, adapting to varying availability of labeled data.