# Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries

arXiv: https://arxiv.org/abs/2509.21633 · HTML: https://mlsanity.com/paper/2509.21633/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Recent Advances in Optimal Transport for Machine Learning](https://mlsanity.com/paper/2306.16156/) (2023)
- [EmoBench: Evaluating the Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2402.12071/) (2024)
- [Entity Tracking in Language Models](https://mlsanity.com/paper/2305.02363/) (2023)
- [AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2501.16566/) (2025)
- [SpanEmo: Casting Multi-label Emotion Classification as Span-prediction](https://mlsanity.com/paper/2101.10038/) (2021)
- [sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification](https://mlsanity.com/paper/2108.10566/) (2021)
- [The Moral Foundations Reddit Corpus](https://mlsanity.com/paper/2208.05545/) (2022)
- [Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion](https://mlsanity.com/paper/2210.15842/) (2022)
- [Joint Binary Neural Network for Multi-label Learning with Applications to Emotion Classification](https://mlsanity.com/paper/1802.00891/) (2018)
- [CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification](https://mlsanity.com/paper/2109.04853/) (2021)
- [Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats](https://mlsanity.com/paper/2211.00171/) (2022)
- [LC-Protonets: Multi-Label Few-Shot Learning for World Music Audio Tagging](https://mlsanity.com/paper/2409.11264/) (2024)
- [Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts](https://mlsanity.com/paper/2505.17222/) (2025)
- [Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors](https://mlsanity.com/paper/2410.13776/) (2024)
- [Similarity-based Multi-label Learning](https://mlsanity.com/paper/1710.10335/) (2017)
- [Tree-Based Dynamic Classifier Chains](https://mlsanity.com/paper/2112.06672/) (2021)
- [Dimensionality-Reduction Techniques for Approximate Nearest Neighbor Search: A Survey and Evaluation](https://mlsanity.com/paper/2403.13491/) (2024)
- [Similarity-Dissimilarity Loss for Multi-label Supervised Contrastive Learning](https://mlsanity.com/paper/2410.13439/) (2024)
- [Knowledge-Based Construction of Confusion Matrices for Multi-Label Classification Algorithms using Semantic Similarity Measures](https://mlsanity.com/paper/2011.00109/) (2020)

## Abstract (arXiv)

We propose Semantic F1 Scores, novel evaluation metrics for subjective or fuzzy multi-label classification that quantify semantic relatedness between predicted and gold labels. Unlike the conventional F1 metrics that treat semantically related predictions as complete failures, Semantic F1 incorporates a label similarity matrix to compute soft precision-like and recall-like scores, from which the Semantic F1 scores are derived. Unlike existing similarity-based metrics, our novel two-step precision-recall formulation enables the comparison of label sets of arbitrary sizes without discarding labels or forcing matches between dissimilar labels. By granting partial credit for semantically related but nonidentical labels, Semantic F1 better reflects the realities of domains marked by human disagreement or fuzzy category boundaries. In this way, it provides fairer evaluations: it recognizes that categories overlap, that annotators disagree, and that downstream decisions based on similar predictions lead to similar outcomes. Through theoretical justification and extensive empirical validation on synthetic and real data, we show that Semantic F1 demonstrates greater interpretability and ecological validity. Because it requires only a domain-appropriate similarity matrix, which is robust to misspecification, and not a rigid ontology, it is applicable across tasks and modalities.