# Fairness Evaluation and Inference Level Mitigation in LLMs

arXiv: https://arxiv.org/abs/2510.18914 · HTML: https://mlsanity.com/paper/2510.18914/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 29 of 29)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings](https://mlsanity.com/paper/1904.01596/) (2019)
- [Conditional probing: measuring usable information beyond a baseline](https://mlsanity.com/paper/2109.09234/) (2021)
- [UP5: Unbiased Foundation Model for Fairness-aware Recommendation](https://mlsanity.com/paper/2305.12090/) (2023)
- [FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs](https://mlsanity.com/paper/2410.19317/) (2024)
- [Investigating Non-Transitivity in LLM-as-a-Judge](https://mlsanity.com/paper/2502.14074/) (2025)
- [Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior](https://mlsanity.com/paper/2010.01791/) (2020)
- [Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination](https://mlsanity.com/paper/2311.09627/) (2023)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs](https://mlsanity.com/paper/2503.05371/) (2025)
- [Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach](https://mlsanity.com/paper/2412.11679/) (2024)
- [What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models](https://mlsanity.com/paper/2406.12277/) (2024)
- [Task-specific Compression for Multi-task Language Models using Attribution-based Pruning](https://mlsanity.com/paper/2205.04157/) (2022)
- [Steering Towards Fairness: Mitigating Political Bias in LLMs](https://mlsanity.com/paper/2508.08846/) (2025)
- [Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons](https://mlsanity.com/paper/2403.07688/) (2024)
- [LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases](https://mlsanity.com/paper/2501.03112/) (2025)
- [Framing Political Bias in Multilingual LLMs Across Pakistani Languages](https://mlsanity.com/paper/2506.00068/) (2025)
- [AdvSumm: Adversarial Training for Bias Mitigation in Text Summarization](https://mlsanity.com/paper/2506.06273/) (2025)
- [No Training Wheels: Steering Vectors for Bias Correction at Inference Time](https://mlsanity.com/paper/2506.18598/) (2025)
- [BiMi Sheets: Infosheets for bias mitigation methods](https://mlsanity.com/paper/2505.22114/) (2025)

## Built on by (most cited first, 2 of 2)

- [Bias Beyond Borders: Political Ideology Evaluation and Steering in Multilingual LLMs](https://mlsanity.com/paper/2601.23001/) (2026)
- [Do Clinical Question Answering Systems Really Need Specialised Medical Fine Tuning?](https://mlsanity.com/paper/2601.12812/) (2026)

## Abstract (arXiv)

Large language models often display undesirable behaviors embedded in their internal representations, undermining fairness, inconsistency drift, amplification of harmful content, and the propagation of unwanted patterns during extended dialogue and conversations. Although training-time or data-centric methods attempt to reduce these effects, they are computationally expensive, irreversible once deployed, and slow to adapt to new conversational contexts. Pruning-based methods provide a flexible and transparent way to reduce bias by adjusting the neurons responsible for certain behaviors. However, most existing approaches are static; once a neuron is removed, the model loses the ability to adapt when the conversation or context changes. To address this, we propose a dynamic, reversible, pruning-based framework that detects context-aware neuron activations and applies adaptive masking to modulate their influence during generation. Our inference-time solution provides fine-grained, memory-aware mitigation with knowledge-preserved, more coherent behavior across multilingual single- and multi-turn dialogues, enabling dynamic fairness control in real-world conversational AI.