mlsanityData to 7 Oct 2026

Prompt-to-Leaderboard

arXiv 2502.14855 ↗First posted 20 Feb 2025cs.LG cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leaderboard. Our code is available on GitHub at https://github.com/lmarena/p2l.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

22 of 42 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
10 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  4. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  5. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  6. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  8. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  9. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  10. 2024A Statistical Framework for Ranking LLM-Based Chatbotscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 12.

27 papers.

Sort:

By area:

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  3. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  4. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  5. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 9
  6. 2025Don't Throw Away Your Pretrained Modelcited by 9
  7. 2026Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefscited by 9
  8. 2025INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profilingcited by 8
  9. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 5
  10. 2025Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networkscited by 4
  11. 2025Learnware of Language Models: Specialized Small Language Models Can Do Bigcited by 3
  12. 2025PromptWise: Online Learning for Cost-Aware Prompt Assignment in Generative Modelscited by 3
15 more
  1. 2025Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inferencecited by 3
  2. 2025Reward Model Routing in Alignmentcited by 3
  3. 2025EigenBench: A Comparative Behavioral Measure of Value Alignmentcited by 2
  4. 2026DAK-UCB: Diversity-Aware Prompt Routing for LLMs and Generative Modelscited by 2
  5. 2026Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judgecited by 2
  6. 2025Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routingcited by 1
  7. 2026The Single-Multi Evolution Loop for Self-Improving Model Collaboration Systemscited by 1
  8. 2026Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorizationcited by 1
  9. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 1
  10. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Modelscited by 0
  11. 2026Probing the Knowledge Boundary: An Interactive Agentic Framework for Deep Knowledge Extractioncited by 0
  12. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Modelscited by 0
  13. 2026The Routing Plateau: Understanding the Accuracy Limits of LLM Routerscited by 0
  14. 2026SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Taskscited by 0
  15. 2026Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
  2. 2025Structured Prompts Improve Evaluation of Language Models
  3. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmark
  4. 2025Inclusion Arena: An Open Platform for Evaluating Large Foundation Models with Real-World Apps
  5. 2025User-centric Subjective Leaderboard by Customizable Reward Modeling
  6. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  8. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning