# Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

arXiv: https://arxiv.org/abs/2310.10443 · HTML: https://mlsanity.com/paper/2310.10443/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](https://mlsanity.com/paper/1905.05897/) (2019)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [Bullseye Polytope: A Scalable Clean-Label Poisoning Attack with Improved Transferability](https://mlsanity.com/paper/2005.00191/) (2020)
- [PECOS: Prediction for Enormous and Correlated Output Spaces](https://mlsanity.com/paper/2010.05878/) (2020)
- [Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study](https://mlsanity.com/paper/2304.10909/) (2023)
- [An Exploration of Encoder-Decoder Approaches to Multi-Label Classification for Legal and Biomedical Text](https://mlsanity.com/paper/2305.05627/) (2023)
- [Stolen Probability: A Structural Weakness of Neural Language Models](https://mlsanity.com/paper/2005.02433/) (2020)
- [Node Embeddings and Exact Low-Rank Representations of Complex Networks](https://mlsanity.com/paper/2006.05592/) (2020)
- [Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice](https://mlsanity.com/paper/2203.06462/) (2022)
- [Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities](https://mlsanity.com/paper/1902.08077/) (2019)

## Built on by (most cited first, 11 of 11)

- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [On the Independence Assumption in Neurosymbolic Learning](https://mlsanity.com/paper/2404.08458/) (2024)
- [A Probabilistic Neuro-symbolic Layer for Algebraic Constraint Satisfaction](https://mlsanity.com/paper/2503.19466/) (2025)
- [Towards Reasonable Concept Bottleneck Models](https://mlsanity.com/paper/2506.05014/) (2025)
- [On the Theoretical Limitations of Embedding-based Link Prediction](https://mlsanity.com/paper/2506.22271/) (2025)
- [The Theory and Practice of MAP Inference over Non-Convex Constraints](https://mlsanity.com/paper/2602.08681/) (2026)
- [What Cosine Similarity of Label Representations Can and Cannot Tell us](https://mlsanity.com/paper/2603.29488/) (2026)
- [MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts](https://mlsanity.com/paper/2604.11575/) (2026)
- [Logit Distance Bounds Representational Similarity](https://mlsanity.com/paper/2602.15438/) (2026)
- [MultiSolSegment: Multi-channel segmentation of overlapping features in electroluminescence images of photovoltaic cells](https://mlsanity.com/paper/2603.13337/) (2026)
- [How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention](https://mlsanity.com/paper/2608.26052/) (2026)

## Abstract (arXiv)

Sigmoid output layers are widely used in multi-label classification (MLC) tasks, in which multiple labels can be assigned to any input. In many practical MLC tasks, the number of possible labels is in the thousands, often exceeding the number of input features and resulting in a low-rank output layer. In multi-class classification, it is known that such a low-rank output layer is a bottleneck that can result in unargmaxable classes: classes which cannot be predicted for any input. In this paper, we show that for MLC tasks, the analogous sigmoid bottleneck results in exponentially many unargmaxable label combinations. We explain how to detect these unargmaxable outputs and demonstrate their presence in three widely used MLC datasets. We then show that they can be prevented in practice by introducing a Discrete Fourier Transform (DFT) output layer, which guarantees that all sparse label combinations with up to k active labels are argmaxable. Our DFT layer trains faster and is more parameter efficient, matching the F1@k score of a sigmoid layer while using up to 50% fewer trainable parameters. Our code is publicly available at https://github.com/andreasgrv/sigmoid-bottleneck.