# MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events

arXiv: https://arxiv.org/abs/2604.15203 · HTML: https://mlsanity.com/paper/2604.15203/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 80 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [SGM: Sequence Generation Model for Multi-label Classification](https://mlsanity.com/paper/1806.04822/) (2018)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://mlsanity.com/paper/2312.12343/) (2023)
- [Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution](https://mlsanity.com/paper/2109.04712/) (2021)
- [Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency](https://mlsanity.com/paper/2502.04964/) (2025)
- [Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotations](https://mlsanity.com/paper/2204.09781/) (2022)
- [Exploring the Landscape of Natural Language Processing Research](https://mlsanity.com/paper/2307.10652/) (2023)
- [Revisiting Hierarchical Text Classification: Inference and Metrics](https://mlsanity.com/paper/2410.01305/) (2024)
- [JRC EuroVoc Indexer JEX - A freely available multi-label categorisation tool](https://mlsanity.com/paper/1309.5223/) (2013)

## Built on by (most cited first, 1 of 1)

- [Argument Graph Uncertainty: Quantifying Uncertainty from the Logical Structure of Reasoning Chains](https://mlsanity.com/paper/2610.20328/) (2026)

## Abstract (arXiv)

Machine learning in high-stakes domains such as healthcare requires not only strong predictive performance but also reliable uncertainty quantification (UQ) to support human oversight. Multi-label text classification (MLTC) is a central task in this domain, yet remains challenging due to label imbalances, dependencies, and combinatorial complexity. Existing MLTC benchmarks are increasingly saturated and may be affected by training data contamination, making it difficult to distinguish genuine reasoning capabilities from memorization. We introduce MADE, a living MLTC benchmark derived from {m}edical device {ad}verse {e}vent reports and continuously updated with newly published reports to prevent contamination. MADE features a long-tailed distribution of hierarchical labels and enables reproducible evaluation with strict temporal splits. We establish baselines across more than 20 encoder- and decoder-only models under fine-tuning and few-shot settings (instruction-tuned/reasoning variants, local/API-accessible). We systematically assess entropy-/consistency-based and self-verbalized UQ methods. Results show clear trade-offs: smaller discriminatively fine-tuned decoders achieve the strongest head-to-tail accuracy while maintaining competitive UQ; generative fine-tuning delivers the most reliable UQ; large reasoning models improve performance on rare labels yet exhibit surprisingly weak UQ; and self-verbalized confidence is not a reliable proxy for uncertainty. Our work is publicly available at https://hhi.fraunhofer.de/aml-demonstrator/made-benchmark.