# Addressing Selection Bias in Computerized Adaptive Testing: A User-Wise Aggregate Influence Function Approach

arXiv: https://arxiv.org/abs/2308.11912 · HTML: https://mlsanity.com/paper/2308.11912/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Instructions and Guide for Diagnostic Questions: The NeurIPS 2020 Education Challenge](https://mlsanity.com/paper/2007.12061/) (2020)
- [Correcting for Selection Bias in Learning-to-rank Systems](https://mlsanity.com/paper/2001.11358/) (2020)
- [BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing](https://mlsanity.com/paper/2108.07386/) (2021)
- [Quality meets Diversity: A Model-Agnostic Framework for Computerized Adaptive Testing](https://mlsanity.com/paper/2101.05986/) (2021)
- [Cross-Loss Influence Functions to Explain Deep Network Representations](https://mlsanity.com/paper/2012.01685/) (2020)

## Built on by (most cited first, 1 of 1)

- [Facets of Disparate Impact: Evaluating Legally Consistent Bias in Machine Learning](https://mlsanity.com/paper/2505.05471/) (2025)

## Abstract (arXiv)

Computerized Adaptive Testing (CAT) is a widely used, efficient test mode that adapts to the examinee's proficiency level in the test domain. CAT requires pre-trained item profiles, for CAT iteratively assesses the student real-time based on the registered items' profiles, and selects the next item to administer using candidate items' profiles. However, obtaining such item profiles is a costly process that involves gathering a large, dense item-response data, then training a diagnostic model on the collected data. In this paper, we explore the possibility of leveraging response data collected in the CAT service. We first show that this poses a unique challenge due to the inherent selection bias introduced by CAT, i.e., more proficient students will receive harder questions. Indeed, when naively training the diagnostic model using CAT response data, we observe that item profiles deviate significantly from the ground-truth. To tackle the selection bias issue, we propose the user-wise aggregate influence function method. Our intuition is to filter out users whose response data is heavily biased in an aggregate manner, as judged by how much perturbation the added data will introduce during parameter estimation. This way, we may enhance the performance of CAT while introducing minimal bias to the item profiles. We provide extensive experiments to demonstrate the superiority of our proposed method based on the three public datasets and one dataset that contains real-world CAT response data.