# Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models

arXiv: https://arxiv.org/abs/2510.11222 · HTML: https://mlsanity.com/paper/2510.11222/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution](https://mlsanity.com/paper/2109.04712/) (2021)
- [Self-Distilled Vision Transformer for Domain Generalization](https://mlsanity.com/paper/2207.12392/) (2022)
- [MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussions](https://mlsanity.com/paper/2403.07678/) (2024)
- [The Moral Foundations Reddit Corpus](https://mlsanity.com/paper/2208.05545/) (2022)
- [A Data Fusion Framework for Multi-Domain Morality Learning](https://mlsanity.com/paper/2304.02144/) (2023)
- [Values, Ethics, Morals? On the Use of Moral Concepts in NLP Research](https://mlsanity.com/paper/2310.13915/) (2023)
- [The harms of class imbalance corrections for machine learning based prediction models: a simulation study](https://mlsanity.com/paper/2404.19494/) (2024)
- [A Survey on Bias and Fairness in Natural Language Processing](https://mlsanity.com/paper/2204.09591/) (2022)
- [Morality is Non-Binary: Building a Pluralist Moral Sentence Embedding Space using Contrastive Learning](https://mlsanity.com/paper/2401.17228/) (2024)
- [Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT](https://mlsanity.com/paper/2407.21058/) (2024)
- [A Survey on Moral Foundation Theory and Pre-Trained Language Models: Current Advances and Challenges](https://mlsanity.com/paper/2409.13521/) (2024)
- [Advancing Vulnerability Classification with BERT: A Multi-Objective Learning Model](https://mlsanity.com/paper/2503.20831/) (2025)

## Built on by (most cited first, 2 of 2)

- [Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings](https://mlsanity.com/paper/2602.01363/) (2026)
- [Integrating Fairness and Explainability in a Multiple Instance Reinforcement Learning System](https://mlsanity.com/paper/2610.00035/) (2026)

## Abstract (arXiv)

Ensuring fairness in natural language processing for moral sentiment classification is challenging, particularly under cross-domain shifts where transformer models are increasingly deployed. Using the Moral Foundations Twitter Corpus (MFTC) and Moral Foundations Reddit Corpus (MFRC), this work evaluates BERT and DistilBERT in a multi-label setting with in-domain and cross-domain protocols. Aggregate performance can mask disparities: we observe pronounced asymmetry in transfer, with Twitter->Reddit degrading micro-F1 by 14.9% versus only 1.5% for Reddit->Twitter. Per-label analysis reveals fairness violations hidden by overall scores; notably, the authority label exhibits Demographic Parity Differences of 0.22-0.23 and Equalized Odds Differences of 0.40-0.41. To address this gap, we introduce the Moral Fairness Consistency (MFC) metric, which quantifies the cross-domain stability of moral foundation detection. MFC shows strong empirical validity, achieving a perfect negative correlation with Demographic Parity Difference (rho = -1.000, p < 0.001) while remaining independent of standard performance metrics. Across labels, loyalty demonstrates the highest consistency (MFC = 0.96) and authority the lowest (MFC = 0.78). These findings establish MFC as a complementary, diagnosis-oriented metric for fairness-aware evaluation of moral reasoning models, enabling more reliable deployment across heterogeneous linguistic contexts. .