# Nonparametric Bayesian Deep Networks with Local Competition

arXiv: https://arxiv.org/abs/1805.07624 · HTML: https://mlsanity.com/paper/1805.07624/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2018.

## Built on by (most cited first, 17 of 17)

- [Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation](https://mlsanity.com/paper/2109.13318/) (2021)
- [Coarse-to-Fine Concept Bottleneck Models](https://mlsanity.com/paper/2310.02116/) (2023)
- [Sparse Linear Concept Discovery Models](https://mlsanity.com/paper/2308.10782/) (2023)
- [Local Competition and Stochasticity for Adversarial Robustness in Deep Learning](https://mlsanity.com/paper/2101.01121/) (2021)
- [Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling](https://mlsanity.com/paper/2205.14283/) (2022)
- [Stochastic Local Winner-Takes-All Networks Enable Profound Adversarial Robustness](https://mlsanity.com/paper/2112.02671/) (2021)
- [Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning](https://mlsanity.com/paper/2404.03323/) (2024)
- [DISCOVER: Making Vision Networks Interpretable via Competition and Dissection](https://mlsanity.com/paper/2310.04929/) (2023)
- [Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2208.01573/) (2022)
- [A New Dataset for End-to-End Sign Language Translation: The Greek Elementary School Dataset](https://mlsanity.com/paper/2310.04753/) (2023)
- [Competing Mutual Information Constraints with Stochastic Competition-based Activations for Learning Diversified Representations](https://mlsanity.com/paper/2201.03624/) (2022)
- [Regularization-Based Efficient Continual Learning in Deep State-Space Models](https://mlsanity.com/paper/2403.10123/) (2024)
- [Variational Bayesian Sequence-to-Sequence Networks for Memory-Efficient Sign Language Translation](https://mlsanity.com/paper/2102.06143/) (2021)
- [Transformers with Stochastic Competition for Tabular Data Modelling](https://mlsanity.com/paper/2407.13238/) (2024)
- [Uncertainty Estimation in Deep Speech Enhancement Using Complex Gaussian Mixture Models](https://mlsanity.com/paper/2212.04831/) (2022)
- [Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks](https://mlsanity.com/paper/2407.10758/) (2024)
- [Exploring Attention Mechanisms in Integration of Multi-Modal Information for Sign Language Recognition and Translation](https://mlsanity.com/paper/2309.01860/) (2023)

## Abstract (arXiv)

The aim of this work is to enable inference of deep networks that retain high accuracy for the least possible model complexity, with the latter deduced from the data during inference. To this end, we revisit deep networks that comprise competing linear units, as opposed to nonlinear units that do not entail any form of (local) competition. In this context, our main technical innovation consists in an inferential setup that leverages solid arguments from Bayesian nonparametrics. We infer both the needed set of connections or locally competing sets of units, as well as the required floating-point precision for storing the network parameters. Specifically, we introduce auxiliary discrete latent variables representing which initial network components are actually needed for modeling the data at hand, and perform Bayesian inference over them by imposing appropriate stick-breaking priors. As we experimentally show using benchmark datasets, our approach yields networks with less computational footprint than the state-of-the-art, and with no compromises in predictive accuracy.