# Token-Level Marginalization for Multi-Label LLM Classifiers

arXiv: https://arxiv.org/abs/2511.22312 · HTML: https://mlsanity.com/paper/2511.22312/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 10 of 10)

- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024)
- [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://mlsanity.com/paper/2406.14491/) (2024)
- [A Survey of Uncertainty Estimation Methods on Large Language Models](https://mlsanity.com/paper/2503.00172/) (2025)
- [Multicalibration for Confidence Scoring in LLMs](https://mlsanity.com/paper/2404.04689/) (2024)
- [MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty](https://mlsanity.com/paper/2408.06816/) (2024)
- [Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models](https://mlsanity.com/paper/2502.14427/) (2025)

## Built on by (most cited first, 1 of 1)

- [DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents](https://mlsanity.com/paper/2603.13791/) (2026)

## Abstract (arXiv)

This paper addresses the critical challenge of deriving interpretable confidence scores from generative language models (LLMs) when applied to multi-label content safety classification. While models like LLaMA Guard are effective for identifying unsafe content and its categories, their generative architecture inherently lacks direct class-level probabilities, which hinders model confidence assessment and performance interpretation. This limitation complicates the setting of dynamic thresholds for content moderation and impedes fine-grained error analysis. This research proposes and evaluates three novel token-level probability estimation approaches to bridge this gap. The aim is to enhance model interpretability and accuracy, and evaluate the generalizability of this framework across different instruction-tuned models. Through extensive experimentation on a synthetically generated, rigorously annotated dataset, it is demonstrated that leveraging token logits significantly improves the interpretability and reliability of generative classifiers, enabling more nuanced content safety moderation.