# Finding Words Associated with DIF: Predicting Differential Item Functioning using LLMs and Explainable AI

arXiv: https://arxiv.org/abs/2502.07017 · HTML: https://mlsanity.com/paper/2502.07017/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 37 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 6 of 6)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)

## Built on by (most cited first, 3 of 3)

- [Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?](https://mlsanity.com/paper/2609.00482/) (2026)
- [Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation](https://mlsanity.com/paper/2606.08988/) (2026)
- [Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques](https://mlsanity.com/paper/2608.06609/) (2026)

## Abstract (arXiv)

We fine-tuned and compared several encoder-based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to these models to identify specific words associated with DIF. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction R2 ranged from .04 to .32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor sub-domains included in the test blueprint by design, rather than construct-irrelevant item content that should be removed from assessments. This may explain why qualitative reviews of DIF items often yield confusing or inconclusive results. Our approach can be used to screen words associated with DIF during the item-writing process for immediate revision, or help review traditional DIF analysis results by highlighting key words in the text. Extensions of this research can enhance the fairness of assessment programs, especially those that lack resources to build high-quality items, and among smaller subpopulations where we do not have sufficient sample sizes for traditional DIF analyses.