mlsanityData to 7 Oct 2026

Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models

arXiv 2510.11222 ↗First posted 13 Oct 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Ensuring fairness in natural language processing for moral sentiment classification is challenging, particularly under cross-domain shifts where transformer models are increasingly deployed. Using the Moral Foundations Twitter Corpus (MFTC) and Moral Foundations Reddit Corpus (MFRC), this work evaluates BERT and DistilBERT in a multi-label setting with in-domain and cross-domain protocols. Aggregate performance can mask disparities: we observe pronounced asymmetry in transfer, with Twitter->Reddit degrading micro-F1 by 14.9% versus only 1.5% for Reddit->Twitter. Per-label analysis reveals fairness violations hidden by overall scores; notably, the authority label exhibits Demographic Parity Differences of 0.22-0.23 and Equalized Odds Differences of 0.40-0.41. To address this gap, we introduce the Moral Fairness Consistency (MFC) metric, which quantifies the cross-domain stability of moral foundation detection. MFC shows strong empirical validity, achieving a perfect negative correlation with Demographic Parity Difference (rho = -1.000, p < 0.001) while remaining independent of standard performance metrics. Across labels, loyalty demonstrates the highest consistency (MFC = 0.96) and authority the lowest (MFC = 0.78). These findings establish MFC as a complementary, diagnosis-oriented metric for fairness-aware evaluation of moral reasoning models, enabling more reliable deployment across heterogeneous linguistic contexts. .

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

29 of 45 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  11. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  12. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
17 more
  1. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  2. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  3. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  4. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  5. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 61
  6. 2022Self-Distilled Vision Transformer for Domain Generalizationcited by 35
  7. 2021Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distributioncited by 34
  8. 2024MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussionscited by 23
  9. 2022The Moral Foundations Reddit Corpuscited by 20
  10. 2023A Data Fusion Framework for Multi-Domain Morality Learningcited by 14
  11. 2023Values, Ethics, Morals? On the Use of Moral Concepts in NLP Researchcited by 13
  12. 2024The harms of class imbalance corrections for machine learning based prediction models: a simulation studycited by 10
  13. 2022A Survey on Bias and Fairness in Natural Language Processingcited by 9
  14. 2024Morality is Non-Binary: Building a Pluralist Moral Sentence Embedding Space using Contrastive Learningcited by 7
  15. 2024Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERTcited by 5
  16. 2024A Survey on Moral Foundation Theory and Pre-Trained Language Models: Current Advances and Challengescited by 3
  17. 2025Advancing Vulnerability Classification with BERT: A Multi-Objective Learning Modelcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddingscited by 2
  2. 2026Integrating Fairness and Explainability in a Multiple Instance Reinforcement Learning Systemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MFTCXplain: A Multilingual Benchmark Dataset for Evaluating the Moral Reasoning of LLMs through Multi-hop Hate Speech Explanation
  2. 2026Unsupervised Elicitation of Moral Values from Language Models
  3. 2025EvalMORAAL: Interpretable Chain-of-Thought and LLM-as-Judge Evaluation for Moral Alignment in Large Language Models
  4. 2026Tracing Moral Foundations in Large Language Models
  5. 2025Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models
  6. 2026PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
  7. 2024Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback
  8. 2026Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking