# Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning

arXiv: https://arxiv.org/abs/2208.01573 · HTML: https://mlsanity.com/paper/2208.01573/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 2 Aug 2022.

## Builds on (most cited first, 29 of 29)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Bayesian Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.03836/) (2018)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Meta-Learning Probabilistic Inference For Prediction](https://mlsanity.com/paper/1805.09921/) (2018)
- [Building machines that adapt and compute like brains](https://mlsanity.com/paper/1711.04203/) (2017)
- [Meta-Learning with Adaptive Hyperparameters](https://mlsanity.com/paper/2011.00209/) (2020)
- [Few-Shot Learning Through an Information Retrieval Lens](https://mlsanity.com/paper/1707.02610/) (2017)
- [Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation](https://mlsanity.com/paper/2109.13318/) (2021)
- [Modular Meta-Learning with Shrinkage](https://mlsanity.com/paper/1909.05557/) (2019)
- [Nonparametric Bayesian Deep Networks with Local Competition](https://mlsanity.com/paper/1805.07624/) (2018)
- [Predictive Complexity Priors](https://mlsanity.com/paper/2006.10801/) (2020)
- [Local Competition and Stochasticity for Adversarial Robustness in Deep Learning](https://mlsanity.com/paper/2101.01121/) (2021)
- [Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning](https://mlsanity.com/paper/1912.02290/) (2019)
- [Human vs. supervised machine learning: Who learns patterns faster?](https://mlsanity.com/paper/2012.03661/) (2020)
- [Gradient-EM Bayesian Meta-learning](https://mlsanity.com/paper/2006.11764/) (2020)
- [Adaptive Network Sparsification with Dependent Variational Beta-Bernoulli Dropout](https://mlsanity.com/paper/1805.10896/) (2018)
- [Fast Task-Adaptation for Tasks Labeled Using Natural Language in Reinforcement Learning](https://mlsanity.com/paper/1910.04040/) (2019)

## Built on by (most cited first, 7 of 7)

- [MetaNO: How to Transfer Your Knowledge on Learning Hidden Physics](https://mlsanity.com/paper/2301.12095/) (2023)
- [DISCOVER: Making Vision Networks Interpretable via Competition and Dissection](https://mlsanity.com/paper/2310.04929/) (2023)
- [A New Dataset for End-to-End Sign Language Translation: The Greek Elementary School Dataset](https://mlsanity.com/paper/2310.04753/) (2023)
- [Transformers with Stochastic Competition for Tabular Data Modelling](https://mlsanity.com/paper/2407.13238/) (2024)
- [EMO: Episodic Memory Optimization for Few-Shot Meta-Learning](https://mlsanity.com/paper/2306.05189/) (2023)
- [MetaFBP: Learning to Learn High-Order Predictor for Personalized Facial Beauty Prediction](https://mlsanity.com/paper/2311.13929/) (2023)
- [Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks](https://mlsanity.com/paper/2407.10758/) (2024)

## Abstract (arXiv)

This work addresses meta-learning (ML) by considering deep networks with stochastic local winner-takes-all (LWTA) activations. This type of network units results in sparse representations from each model layer, as the units are organized into blocks where only one unit generates a non-zero output. The main operating principle of the introduced units rely on stochastic principles, as the network performs posterior sampling over competing units to select the winner. Therefore, the proposed networks are explicitly designed to extract input data representations of sparse stochastic nature, as opposed to the currently standard deterministic representation paradigm. Our approach produces state-of-the-art predictive accuracy on few-shot image classification and regression experiments, as well as reduced predictive error on an active learning setting; these improvements come with an immensely reduced computational cost.