# UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation

arXiv: https://arxiv.org/abs/2405.20612 · HTML: https://mlsanity.com/paper/2405.20612/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [Prototypical Calibration for Few-shot Learning of Language Models](https://mlsanity.com/paper/2205.10183/) (2022)
- [Characterizing Mechanisms for Factual Recall in Language Models](https://mlsanity.com/paper/2310.15910/) (2023)
- [LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction](https://mlsanity.com/paper/2311.06555/) (2023)
- [Unveiling and Manipulating Prompt Influence in Large Language Models](https://mlsanity.com/paper/2405.11891/) (2024)

## Built on by (most cited first, 30 of 31)

- [ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability](https://mlsanity.com/paper/2410.11414/) (2024)
- [Grounded Chain-of-Thought for Multimodal Large Language Models](https://mlsanity.com/paper/2503.12799/) (2025)
- [Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2505.20272/) (2025)
- [How Many Instructions Can LLMs Follow at Once?](https://mlsanity.com/paper/2507.11538/) (2025)
- [NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models](https://mlsanity.com/paper/2410.08970/) (2024)
- [Shortcut Learning in In-Context Learning: A Survey](https://mlsanity.com/paper/2411.02018/) (2024)
- [Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction](https://mlsanity.com/paper/2504.03159/) (2025)
- [Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction](https://mlsanity.com/paper/2512.18813/) (2025)
- [Semantic and Structural Analysis of Implicit Biases in Large Language Models: An Interpretable Approach](https://mlsanity.com/paper/2508.06155/) (2025)
- [Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models](https://mlsanity.com/paper/2509.16805/) (2025)
- [Robustness is Important: Limitations of LLMs for Data Fitting](https://mlsanity.com/paper/2508.19563/) (2025)
- [Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulations](https://mlsanity.com/paper/2510.13982/) (2025)
- [When Numbers Start Talking: Implicit Numerical Coordination Among LLM-Based Agents](https://mlsanity.com/paper/2601.03846/) (2026)
- [Understanding the Dynamics of Demonstration Conflict in In-Context Learning](https://mlsanity.com/paper/2603.04464/) (2026)
- [Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement](https://mlsanity.com/paper/2504.15630/) (2025)
- [In-Context Bias Propagation in LLM-Based Tabular Data Generation](https://mlsanity.com/paper/2506.09630/) (2025)
- [BiasGym: A Simple and Generalizable Framework for Analyzing and Removing Biases through Injection](https://mlsanity.com/paper/2508.08855/) (2025)
- [Ensembling Multilingual Transformers for Robust Sentiment Analysis of Tweets](https://mlsanity.com/paper/2509.24080/) (2025)
- [Identity-Robust Language Model Generation via Content Integrity Preservation](https://mlsanity.com/paper/2601.09141/) (2026)
- [Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues](https://mlsanity.com/paper/2601.12868/) (2026)
- [Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization](https://mlsanity.com/paper/2605.11974/) (2026)
- [Large Language Models Are Still Misled by Simple Bias Ensembles](https://mlsanity.com/paper/2505.16522/) (2025)
- [Fair-GPTQ: Bias-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2509.15206/) (2025)
- [Beyond Hidden-Layer Manipulation: Semantically-Aware Logit Interventions for Debiasing LLMs](https://mlsanity.com/paper/2510.23650/) (2025)
- [CBMAS: Cognitive Behavioral Modeling via Activation Steering](https://mlsanity.com/paper/2601.06109/) (2026)
- [UGID: Unified Graph Isomorphism for Debiasing Large Language Models](https://mlsanity.com/paper/2603.19144/) (2026)
- [Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation](https://mlsanity.com/paper/2604.03257/) (2026)
- [FairNVT: Fair Classification via Noise Injection in Vision Transformers](https://mlsanity.com/paper/2604.16780/) (2026)
- [Counting Worlds Branching Time Semantics for post-hoc Bias Mitigation in generative AI](https://mlsanity.com/paper/2604.19431/) (2026)
- [When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation](https://mlsanity.com/paper/2607.07937/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in various tasks using the in-context learning (ICL) paradigm. However, their effectiveness is often compromised by inherent bias, leading to prompt brittleness, i.e., sensitivity to design settings such as example selection, order, and prompt formatting. Previous studies have addressed LLM bias through external adjustment of model outputs, but the internal mechanisms that lead to such bias remain unexplored. Our work delves into these mechanisms, particularly investigating how feedforward neural networks (FFNs) and attention heads result in the bias of LLMs. By Interpreting the contribution of individual FFN vectors and attention heads, we identify the biased LLM components that skew LLMs' prediction toward specific labels. To mitigate these biases, we introduce UniBias, an inference-only method that effectively identifies and eliminates biased FFN vectors and attention heads. Extensive experiments across 12 NLP datasets demonstrate that UniBias significantly enhances ICL performance and alleviates prompt brittleness of LLMs.