# Large Language Models Do Multi-Label Classification Differently

arXiv: https://arxiv.org/abs/2505.17510 · HTML: https://mlsanity.com/paper/2505.17510/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024)
- [On the Calibration of Large Language Models and Alignment](https://mlsanity.com/paper/2311.13240/) (2023)
- [Calibrating Language Models with Adaptive Temperature Scaling](https://mlsanity.com/paper/2409.19817/) (2024)
- [A Close Look into the Calibration of Pre-trained Language Models](https://mlsanity.com/paper/2211.00151/) (2022)
- [SpanEmo: Casting Multi-label Emotion Classification as Span-prediction](https://mlsanity.com/paper/2101.10038/) (2021)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks](https://mlsanity.com/paper/2311.09743/) (2023)
- [Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models](https://mlsanity.com/paper/2402.17124/) (2024)
- [ML-Net: multi-label classification of biomedical texts with deep neural networks](https://mlsanity.com/paper/1811.05475/) (2018)
- [Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion](https://mlsanity.com/paper/2210.15842/) (2022)
- [Joint Binary Neural Network for Multi-label Learning with Applications to Emotion Classification](https://mlsanity.com/paper/1802.00891/) (2018)
- [Can Large Language Models Serve as Effective Classifiers for Hierarchical Multi-Label Classification of Scientific Documents at Industrial Scale?](https://mlsanity.com/paper/2412.05137/) (2024)
- [The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition](https://mlsanity.com/paper/2403.17125/) (2024)
- [Noise Audits Improve Moral Foundation Classification](https://mlsanity.com/paper/2210.07415/) (2022)
- [DALLMi: Domain Adaption for LLM-based Multi-label Classifier](https://mlsanity.com/paper/2405.01883/) (2024)
- [Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities](https://mlsanity.com/paper/2311.00203/) (2023)
- [Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors](https://mlsanity.com/paper/2410.13776/) (2024)

## Built on by (most cited first, 13 of 13)

- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions](https://mlsanity.com/paper/2506.00643/) (2025)
- [Seven simple steps for log analysis in AI systems](https://mlsanity.com/paper/2604.09563/) (2026)
- [BigTokDetect: A Clinically-Informed Vision-Language Modeling Framework for Detecting Pro-Bigorexia Videos on TikTok](https://mlsanity.com/paper/2508.06515/) (2025)
- [Learning to vary: Teaching LMs to reproduce human linguistic variability in next-word prediction](https://mlsanity.com/paper/2509.17794/) (2025)
- [Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs](https://mlsanity.com/paper/2602.15436/) (2026)
- [Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights](https://mlsanity.com/paper/2605.10808/) (2026)
- [Automatic Thematic Indexing of Large Literary Corpora: A Machine Learning Approach to Voltaire's Complete Works](https://mlsanity.com/paper/2607.09316/) (2026)
- [Positional task conditioning for scalable defect detection across product families in large product catalogs](https://mlsanity.com/paper/2609.09567/) (2026)
- [Beyond Final-Token Classification: Heterogeneous Readouts for Evidence-Grounded Suicide Risk Detection](https://mlsanity.com/paper/2609.22767/) (2026)
- [Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing](https://mlsanity.com/paper/2609.28026/) (2026)
- [Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs](https://mlsanity.com/paper/2609.36115/) (2026)
- [Multi-Label Topic Assignment via LLM Distillation: A Comparative Analysis of Generative vs. Discriminative Student Models](https://mlsanity.com/paper/2610.09063/) (2026)

## Abstract (arXiv)

Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. We investigate how autoregressive LLMs perform multi-label classification, focusing on subjective tasks, by analyzing the output distributions of the models at each label generation step. We find that the initial probability distribution for the first label often does not reflect the eventual final output, even in terms of relative order and find LLMs tend to suppress all but one label at each generation step. We further observe that as model scale increases, their token distributions exhibit lower entropy and higher single-label confidence, but the internal relative ranking of the labels improves. Finetuning methods such as supervised finetuning and reinforcement learning amplify this phenomenon. We introduce the task of distribution alignment for multi-label settings: aligning LLM-derived label distributions with empirical distributions estimated from annotator responses in subjective tasks. We propose both zero-shot and supervised methods which improve both alignment and predictive performance over existing approaches. We find one method -- taking the max probability over all label generation distributions instead of just using the initial probability distribution -- improves both distribution alignment and overall F1 classification without adding any additional computation.