# How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench

arXiv: https://arxiv.org/abs/2305.14947 · HTML: https://mlsanity.com/paper/2305.14947/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Predicting Performance for Natural Language Processing Tasks](https://mlsanity.com/paper/2005.00870/) (2020)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [Do Question Answering Modeling Improvements Hold Across Benchmarks?](https://mlsanity.com/paper/2102.01065/) (2021)
- [Predicting Fine-Tuning Performance with Probing](https://mlsanity.com/paper/2210.07352/) (2022)

## Built on by (most cited first, 30 of 34)

- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [Active Evaluation Acquisition for Efficient LLM Benchmarking](https://mlsanity.com/paper/2410.05952/) (2024)
- [100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances](https://mlsanity.com/paper/2409.03563/) (2024)
- [LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services](https://mlsanity.com/paper/2410.02425/) (2024)
- [Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings](https://mlsanity.com/paper/2510.26384/) (2025)
- [U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2410.01692/) (2024)
- [What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement](https://mlsanity.com/paper/2402.01865/) (2024)
- [AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems](https://mlsanity.com/paper/2408.14972/) (2024)
- [Predicting the Performance of Black-box LLMs through Follow-up Queries](https://mlsanity.com/paper/2501.01558/) (2025)
- [How Reliable is Language Model Micro-Benchmarking?](https://mlsanity.com/paper/2510.08730/) (2025)
- [Demystifying Language Model Forgetting with Low-rank Example Associations](https://mlsanity.com/paper/2406.14026/) (2024)
- [Accelerating Unbiased LLM Evaluation via Synthetic Feedback](https://mlsanity.com/paper/2502.10563/) (2025)
- [PredictaBoard: Benchmarking LLM Score Predictability](https://mlsanity.com/paper/2502.14445/) (2025)
- [Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspective](https://mlsanity.com/paper/2502.17262/) (2025)
- [Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery](https://mlsanity.com/paper/2505.16477/) (2025)
- [Performance Law of Large Language Models](https://mlsanity.com/paper/2408.09895/) (2024)
- [SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone](https://mlsanity.com/paper/2510.17998/) (2025)
- [BenTo: Benchmark Task Reduction with In-Context Transferability](https://mlsanity.com/paper/2410.13804/) (2024)
- [Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory](https://mlsanity.com/paper/2603.02663/) (2026)
- [You Don't Need to Run Every Eval](https://mlsanity.com/paper/2606.24020/) (2026)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [Predicting Task Performance with Context-aware Scaling Laws](https://mlsanity.com/paper/2510.14919/) (2025)
- [Cost-Efficient Estimation of General Abilities Across Benchmarks](https://mlsanity.com/paper/2604.01418/) (2026)
- [Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Performance](https://mlsanity.com/paper/2504.19811/) (2025)
- [Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law](https://mlsanity.com/paper/2506.13216/) (2025)
- [ZPD Detector: Data Selection via Capability-Difficulty Alignment for Large Language Models](https://mlsanity.com/paper/2601.10986/) (2026)
- [SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities](https://mlsanity.com/paper/2609.33763/) (2026)

## Abstract (arXiv)

We investigate the predictability of large language model (LLM) capabilities: given records of past experiments using different model families, numbers of parameters, tasks, and numbers of in-context examples, can we accurately predict LLM performance on new experiment configurations? Answering this question has practical implications for LLM users (e.g., deciding which models to try), developers (e.g., prioritizing evaluation on representative tasks), and the research community (e.g., identifying hard-to-predict capabilities that warrant further investigation). We study the performance prediction problem on experiment records from BIG-bench. On a random train-test split, an MLP-based predictor achieves an R2 score greater than 95%, indicating the presence of learnable patterns within the experiment records. We then formulate the problem of searching for "small-bench," an informative subset of BIG-bench tasks from which the performance on the full set can be maximally recovered. We find a subset as informative as BIG-bench Hard for evaluating new model families, while being 3× smaller. Additionally, we find competitive subsets by clustering task representations learned by our MLP-based predictor and selecting tasks close to cluster centroids, highlighting the importance of task diversity in constructing "small-bench."