Aligning Language Model Benchmarks with Pairwise Preferences
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Language model benchmarks are pervasive and computationally-efficient proxies for real-world downstream performance. However, many recent works find that benchmarks often fail to predict downstream utility. While some works have begun diagnosing sources of misalignment, there remain no ways to systematically update benchmarks to align their scores with downstream usage. Towards bridging this gap, we introduce and study textitbenchmark alignment, where we use information about downstream model performance to automatically update benchmarks, specifically aiming to update static benchmarks so they generalizably rank models according to new pairwise preferences. Our experiments involving 4576 language models and 6 benchmarks show that reweighting benchmark items can successfully rank unseen models, even generalizing across model scales in most cases. And while naive alignment unsurprisingly requires large numbers of models and benchmark questions, an oracle experiment suggests this could be reduced to as few as 20 well-chosen models. Overall, our work takes a step towards efficiently aligning benchmark development with downstream tasks.footnoteAll of our code, models, and data are publicly-available.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 52 references that are in the map. First posted on arXiv 2 Feb 2026.
Builds on
38 of 52 references.
Closest ancestors · 2
Same direction, newest first
- 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
- 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
Other references · 36
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
- 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
24 more
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
- 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
- 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
- 2023Generative Judge for Evaluating Alignmentcited by 163
- 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
- 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
- 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
- 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
- 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 77
- 2021Comparing Test Sets with Item Response Theorycited by 55
- 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
- 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Modelscited by 35
- 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
- 2025Fluid Language Model Benchmarkingcited by 30
- 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
- 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
- 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
- 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performancecited by 17
- 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
- 2024ImageNot: A contrast with ImageNet preserves model rankingscited by 14
- 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
- 2025Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomicscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
- 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
- 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
- 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
- 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
- 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
- 2024CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models
- 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger