# Collaborative Performance Prediction for Large Language Models

arXiv: https://arxiv.org/abs/2407.01300 · HTML: https://mlsanity.com/paper/2407.01300/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://mlsanity.com/paper/2205.10487/) (2022)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [FastSHAP: Real-Time Shapley Value Estimation](https://mlsanity.com/paper/2107.07436/) (2021)
- [A Theory for Emergence of Complex Skills in Language Models](https://mlsanity.com/paper/2307.15936/) (2023)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation](https://mlsanity.com/paper/2311.08877/) (2023)
- [How predictable is language model benchmark performance?](https://mlsanity.com/paper/2401.04757/) (2024)
- [OBOE: Collaborative Filtering for AutoML Model Selection](https://mlsanity.com/paper/1808.03233/) (2018)
- [Predicting Emergent Abilities with Infinite Resolution Evaluation](https://mlsanity.com/paper/2310.03262/) (2023)
- [Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?](https://mlsanity.com/paper/2310.11616/) (2023)
- [Do Question Answering Modeling Improvements Hold Across Benchmarks?](https://mlsanity.com/paper/2102.01065/) (2021)

## Built on by (most cited first, 9 of 9)

- [Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs](https://mlsanity.com/paper/2505.03814/) (2025)
- [Can We Predict Performance of Large Models across Vision-Language Tasks?](https://mlsanity.com/paper/2410.10112/) (2024)
- [Forecasting Frontier Language Model Agent Capabilities](https://mlsanity.com/paper/2502.15850/) (2025)
- [PACE: A Proxy for Agentic Capability Evaluation](https://mlsanity.com/paper/2607.02032/) (2026)
- [You Don't Need to Run Every Eval](https://mlsanity.com/paper/2606.24020/) (2026)
- [STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction](https://mlsanity.com/paper/2602.12143/) (2026)
- [Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Performance](https://mlsanity.com/paper/2504.19811/) (2025)
- [Collaborative Prediction: To Join or To Disjoin Datasets](https://mlsanity.com/paper/2506.11271/) (2025)
- [Beyond Independent Labels: Schwartz-Geometry Decoding for Human Value Detection](https://mlsanity.com/paper/2607.05052/) (2026)

## Abstract (arXiv)

Comprehensively understanding and accurately predicting the performance of large language models across diverse downstream tasks has emerged as a pivotal challenge in NLP research. The pioneering scaling law on downstream works demonstrated intrinsic similarities within model families and utilized such similarities for performance prediction. However, they tend to overlook the similarities between model families and only consider design factors listed in the original scaling law. To overcome these limitations, we introduce a novel framework, Collaborative Performance Prediction (CPP), which significantly enhances prediction accuracy by leveraging the historical performance of various models on downstream tasks and other design factors for both model and task. We also collect a collaborative data sourced from online platforms containing both historical performance and additional design factors. With the support of the collaborative data, CPP not only surpasses traditional scaling laws in predicting the performance of scaled LLMs but also facilitates a detailed analysis of factor importance, an area previously overlooked.