mlsanityData to 7 Oct 2026

PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

arXiv 2609.36086 ↗First posted 28 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

also in 2 more
Abstract (arXiv)

Language models are frequently employed to evaluate other language models. An LM evaluator scoring agentic behaviors across multiple criteria is valuable, provided that its decisions align with human judgment. We call the problem of evaluating this alignment Meta-Evaluation. Tackling it directly is difficult: collecting human data is expensive, absolute scoring is hard to align, and using an LM meta-evaluator recurses the question of trustworthiness. We adopt a reformulation of meta-evaluation as a preference judgment problem: rather than comparing human and LM evaluator scores of a trajectory, we ask whether their implied preferences align. Building on this, we introduce PADM'E, a data synthesis method that generates reliable criterion-based meta-evaluation data for agentic settings. PADM'E uses only small language models, requires no human involvement during evaluations, and operates under a low computational budget. We build a prototype of PADM'E and synthesize a dataset of 1,000 samples across four agentic domains and three evaluation criteria. Human validation on a 150-sample subset demonstrates that PADM'E improves agreement with human judgment from 73% to 85% over a naive baseline. Meta-evaluating 25 common models with our dataset demonstrates the correlations between evaluation performance and scoring granularity, leniency, and model size, among other factors.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 34 of its 39 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

34 of 39 references.

References · 34

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  5. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  7. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  8. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  11. 2025Small Language Models are the Future of Agentic AIcited by 254
  12. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
22 more
  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  5. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  6. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  7. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  8. 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 77
  9. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  10. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  11. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 37
  12. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  13. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  14. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  15. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 19
  16. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 10
  17. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  18. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  19. 2026Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judgecited by 6
  20. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judgecited by 5
  21. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  22. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  2. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  3. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  4. 2025LLMs Judge Themselves: A Game-Theoretic Framework for Human-Aligned Evaluation
  5. 2024Dissecting Human and LLM Preferences
  6. 2026Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
  7. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  8. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation