# Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination

arXiv: https://arxiv.org/abs/2311.09627 · HTML: https://mlsanity.com/paper/2311.09627/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [Task-Specific Skill Localization in Fine-tuned Language Models](https://mlsanity.com/paper/2302.06600/) (2023)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages](https://mlsanity.com/paper/1903.06400/) (2019)
- [Calibration, Entropy Rates, and Memory in Language Models](https://mlsanity.com/paper/1906.05664/) (2019)
- [Task-specific Compression for Multi-task Language Models using Attribution-based Pruning](https://mlsanity.com/paper/2205.04157/) (2022)

## Built on by (most cited first, 23 of 23)

- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric](https://mlsanity.com/paper/2402.06900/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics](https://mlsanity.com/paper/2412.01113/) (2024)
- [Shortcut Learning in In-Context Learning: A Survey](https://mlsanity.com/paper/2411.02018/) (2024)
- [Towards Robust ESG Analysis Against Greenwashing Risks: Aspect-Action Analysis with Cross-Category Generalization](https://mlsanity.com/paper/2502.15821/) (2025)
- [Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models](https://mlsanity.com/paper/2509.16805/) (2025)
- [Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning](https://mlsanity.com/paper/2509.22263/) (2025)
- [Are Bias Evaluation Methods Biased ?](https://mlsanity.com/paper/2506.17111/) (2025)
- [Breaking Down Bias: On The Limits of Generalizable Pruning Strategies](https://mlsanity.com/paper/2502.07771/) (2025)
- [CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing](https://mlsanity.com/paper/2504.21228/) (2025)
- [REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting](https://mlsanity.com/paper/2509.15723/) (2025)
- [Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset](https://mlsanity.com/paper/2509.24468/) (2025)
- [Fairness Evaluation and Inference Level Mitigation in LLMs](https://mlsanity.com/paper/2510.18914/) (2025)
- [Neuron-Level Interventions for Gendered and Gender-Neutral Generation in Language Models](https://mlsanity.com/paper/2605.30717/) (2026)
- [From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling](https://mlsanity.com/paper/2505.12381/) (2025)
- [Neuron-Level Analysis of Cultural Understanding in Large Language Models](https://mlsanity.com/paper/2510.08284/) (2025)
- [Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP](https://mlsanity.com/paper/2510.22823/) (2025)
- [DeFrame: Debiasing Large Language Models Against Framing Effects](https://mlsanity.com/paper/2602.04306/) (2026)
- [Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation](https://mlsanity.com/paper/2603.15134/) (2026)
- [Unplug and Play Language Models: Decomposing Experts in Language Models at Inference Time](https://mlsanity.com/paper/2404.11916/) (2024)
- [Information Gain-Guided Causal Intervention for Autonomous Debiasing Large Language Models](https://mlsanity.com/paper/2504.12898/) (2025)
- [Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement](https://mlsanity.com/paper/2601.21864/) (2026)

## Abstract (arXiv)

Instruction-following language models often show undesirable biases. These undesirable biases may be accelerated in the real-world usage of language models, where a wide range of instructions is used through zero-shot example prompting. To solve this problem, we first define the bias neuron, which significantly affects biased outputs, and prove its existence empirically. Furthermore, we propose a novel and practical bias mitigation method, CRISPR, to eliminate bias neurons of language models in instruction-following settings. CRISPR automatically determines biased outputs and categorizes neurons that affect the biased outputs as bias neurons using an explainability method. Experimental results demonstrate the effectiveness of our method in mitigating biases under zero-shot instruction-following settings without losing the model's task performance and existing knowledge. The experimental results reveal the generalizability of our method as it shows robustness under various instructions and datasets. Surprisingly, our method can mitigate the bias in language models by eliminating only a few neurons (at least three).