mlsanityData to 14 Oct 2026

LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model

arXiv 2608.22295 ↗First posted 23 Aug 2026cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Evaluation of large language models (LLMs) increasingly requires predicting how a model will perform on new questions or tasks before collecting large amounts of new annotations. This problem is challenging because question difficulty, scenario, and underlying capability demands can vary substantially. Simple retrospective averages may confound model ability with item characteristics. In this paper, we study a model-based evaluation framework that combines multidimensional item response theory model with question contexts to predict LLM performance on unseen questions. The framework represents LLMs through latent capability profiles while using question content to inform item characteristics, allowing information to transfer beyond previously observed items. Empirically, we find that for within-scenario evaluation, incorporating question embeddings improves prediction relative to model-free baselines, and that multidimensional latent structure provides a richer description of capability variation than unidimensional alternatives. At the same time, our results reveal an important limitation that the generalizability does not necessarily translate into reliable prediction under cross-scenario shift. These findings suggest that context-aware psychometric modeling is a promising direction for efficient and interpretable LLM evaluation, while also highlighting cross-scenario generalization as a central open challenge.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

22 of 32 references.

References · 22

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  3. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  4. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  6. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  7. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  8. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 45
  9. 2019Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowdscited by 40
  10. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  11. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
  12. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 31
10 more
  1. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 31
  2. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24
  3. 2021Item Response Theory -- A Statistical Framework for Educational and Psychological Measurementcited by 17
  4. 2025Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?cited by 8
  5. 2025Efficient Evaluation of Large Language Models via Collaborative Filteringcited by 5
  6. 2025JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theorycited by 4
  7. 2025A Latent Variable Framework for Scaling Laws in Large Language Modelscited by 2
  8. 2026ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systemscited by 2
  9. 2025Just Read the Question: Enabling Generalization to New Assessment Items with Text Awarenesscited by 1
  10. 2026Knowledge-Embedded Latent Projection for Robust Representation Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation
  2. 2026An Interpretable and Scalable Framework for Evaluating Large Language Models
  3. 2024Enhancing LLM Evaluations: The Garbling Trick
  4. 2026Beyond Questions: Evaluating the Stated Knowledge of LLMs
  5. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
  6. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  7. 2025Learning Compact Representations of LLM Abilities via Item Response Theory
  8. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehension