# Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds

arXiv: https://arxiv.org/abs/1908.11421 · HTML: https://mlsanity.com/paper/1908.11421/

## Key facts

- Cited by 40 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 29 Aug 2019.

## Built on by (most cited first, 30 of 40)

- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Better than Average: Paired Evaluation of NLP Systems](https://mlsanity.com/paper/2110.10746/) (2021)
- [Fluid Language Model Benchmarking](https://mlsanity.com/paper/2509.11106/) (2025)
- [Reliable and Efficient Amortized Model-based Evaluation](https://mlsanity.com/paper/2503.13335/) (2025)
- [What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?](https://mlsanity.com/paper/2106.00794/) (2021)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [Rissanen Data Analysis: Examining Dataset Characteristics via Description Length](https://mlsanity.com/paper/2103.03872/) (2021)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [py-irt: A Scalable Item Response Theory Library for Python](https://mlsanity.com/paper/2203.01282/) (2022)
- [Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models](https://mlsanity.com/paper/2407.15645/) (2024)
- [Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing](https://mlsanity.com/paper/2406.14230/) (2024)
- [Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?](https://mlsanity.com/paper/2506.09796/) (2025)
- [Can Transformer Language Models Predict Psychometric Properties?](https://mlsanity.com/paper/2106.06849/) (2021)
- [Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness](https://mlsanity.com/paper/2406.16342/) (2024)
- [A Psychology-based Unified Dynamic Framework for Curriculum Learning](https://mlsanity.com/paper/2408.05326/) (2024)
- [BRIDGE: Predicting Human Task Completion Time From Model Performance](https://mlsanity.com/paper/2602.07267/) (2026)
- [Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models](https://mlsanity.com/paper/2404.07720/) (2024)
- [A Spectral Approach to Item Response Theory](https://mlsanity.com/paper/2210.04317/) (2022)
- [JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory](https://mlsanity.com/paper/2509.22888/) (2025)
- [Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?](https://mlsanity.com/paper/2510.25064/) (2025)
- [Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores](https://mlsanity.com/paper/2601.13885/) (2026)
- [Aligning Teacher with Student Preferences for Tailored Training Data Generation](https://mlsanity.com/paper/2406.19227/) (2024)
- [QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulation](https://mlsanity.com/paper/2503.05888/) (2025)
- [Can We Trust Item Response Theory for AI Evaluation?](https://mlsanity.com/paper/2607.15190/) (2026)
- [Computational modeling of semantic change](https://mlsanity.com/paper/2304.06337/) (2023)
- [How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation](https://mlsanity.com/paper/2401.11185/) (2024)
- [PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics](https://mlsanity.com/paper/2404.01799/) (2024)
- [AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA](https://mlsanity.com/paper/2604.21766/) (2026)
- [Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits](https://mlsanity.com/paper/2607.02586/) (2026)
- [Item Response Theory for AI Safety](https://mlsanity.com/paper/2608.05086/) (2026)

## Abstract (arXiv)

Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.