# From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling

arXiv: https://arxiv.org/abs/2505.12381 · HTML: https://mlsanity.com/paper/2505.12381/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 18 May 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'](https://mlsanity.com/paper/2410.12954/) (2024)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs](https://mlsanity.com/paper/1801.05453/) (2018)
- [Fairness via Representation Neutralization](https://mlsanity.com/paper/2106.12674/) (2021)
- [Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions](https://mlsanity.com/paper/2207.14251/) (2022)
- [A Challenge Set Approach to Evaluating Machine Translation](https://mlsanity.com/paper/1704.07431/) (2017)
- [IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context](https://mlsanity.com/paper/2403.20147/) (2024)
- [Transformers Can Represent n-gram Language Models](https://mlsanity.com/paper/2404.14994/) (2024)
- [On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2406.14197/) (2024)
- [Understanding Transformers via N-gram Statistics](https://mlsanity.com/paper/2407.12034/) (2024)
- [Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.08045/) (2025)
- [Parameter-efficient Modularised Bias Mitigation via AdapterFusion](https://mlsanity.com/paper/2302.06321/) (2023)
- [Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination](https://mlsanity.com/paper/2311.09627/) (2023)

## Built on by (most cited first, 1 of 1)

- [XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad](https://mlsanity.com/paper/2601.14063/) (2026)

## Abstract (arXiv)

Current research on bias in language models (LMs) predominantly focuses on data quality, with significantly less attention paid to model architecture and temporal influences of data. Even more critically, few studies systematically investigate the origins of bias. We propose a methodology grounded in comparative behavioral theory to interpret the complex interaction between training data and model architecture in bias propagation during language modeling. Building on recent work that relates transformers to n-gram LMs, we evaluate how data, model design choices, and temporal dynamics affect bias propagation. Our findings reveal that: (1) n-gram LMs are highly sensitive to context window size in bias propagation, while transformers demonstrate architectural robustness; (2) the temporal provenance of training data significantly affects bias; and (3) different model architectures respond differentially to controlled bias injection, with certain biases (e.g. sexual orientation) being disproportionately amplified. As language models become ubiquitous, our findings highlight the need for a holistic approach -- tracing bias to its origins across both data and model dimensions, not just symptoms, to mitigate harm.