# py-irt: A Scalable Item Response Theory Library for Python

arXiv: https://arxiv.org/abs/2203.01282 · HTML: https://mlsanity.com/paper/2203.01282/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 50 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 10 of 10)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds](https://mlsanity.com/paper/1908.11421/) (2019)
- [Variational Item Response Theory: Fast, Accurate, and Expressive](https://mlsanity.com/paper/2002.00276/) (2020)
- [Dynamic Data Selection for Curriculum Learning via Ability Estimation](https://mlsanity.com/paper/2011.00080/) (2020)
- [A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis](https://mlsanity.com/paper/2001.07859/) (2020)

## Built on by (most cited first, 19 of 19)

- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [TimeSeriesExam: A time series understanding exam](https://mlsanity.com/paper/2410.14752/) (2024)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUs](https://mlsanity.com/paper/2502.10436/) (2025)
- [How Reliable is Language Model Micro-Benchmarking?](https://mlsanity.com/paper/2510.08730/) (2025)
- [A Psychology-based Unified Dynamic Framework for Curriculum Learning](https://mlsanity.com/paper/2408.05326/) (2024)
- [Investigating the Zone of Proximal Development of Language Models for In-Context Learning](https://mlsanity.com/paper/2502.06990/) (2025)
- [BRIDGE: Predicting Human Task Completion Time From Model Performance](https://mlsanity.com/paper/2602.07267/) (2026)
- [Revisiting Generalization Across Difficulty Levels: It's Not So Easy](https://mlsanity.com/paper/2511.21692/) (2025)
- [Can We Trust Item Response Theory for AI Evaluation?](https://mlsanity.com/paper/2607.15190/) (2026)
- [Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results](https://mlsanity.com/paper/2606.14516/) (2026)
- [Computational modeling of semantic change](https://mlsanity.com/paper/2304.06337/) (2023)
- [LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation](https://mlsanity.com/paper/2511.14531/) (2025)
- [Controlling Cloze-test Question Item Difficulty with PLM-based Surrogate Models for IRT Assessment](https://mlsanity.com/paper/2403.01456/) (2024)
- [The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains](https://mlsanity.com/paper/2605.11205/) (2026)
- [Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment](https://mlsanity.com/paper/2605.00022/) (2026)
- [An Interpretable and Scalable Framework for Evaluating Large Language Models](https://mlsanity.com/paper/2605.07046/) (2026)
- [Efficient Safety Benchmarking via Item Response Theory](https://mlsanity.com/paper/2606.20626/) (2026)
- [What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language](https://mlsanity.com/paper/2610.01627/) (2026)

## Abstract (arXiv)

py-irt is a Python library for fitting Bayesian Item Response Theory (IRT) models. py-irt estimates latent traits of subjects and items, making it appropriate for use in IRT tasks as well as ideal-point models. py-irt is built on top of the Pyro and PyTorch frameworks and uses GPU-accelerated training to scale to large data sets. Code, documentation, and examples can be found at https://github.com/nd-ball/py-irt. py-irt can be installed from the GitHub page or the Python Package Index (PyPI).