mlsanityData to 14 Oct 2026

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

arXiv 2606.03650 ↗First posted 2 Jun 2026cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Selecting a pretrained language model, or evaluating a fine-tuned one, for a specific application is a high-value decision, yet the public benchmarks used to make it are poorly suited: a generic benchmark need not reflect a particular sub-domain or sub-task, and its scores are suspect when its items have leaked into pretraining and are recalled rather than solved. We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation: from a task or domain description, a pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark, answer it, and score one another, with no human labels or raters. Because every model also answers as a student, the responses are the data that weight each question by its discriminative power and each judge by its consensus with the panel. Where ground truth exists, CoEval recovers the true ranking and tracks objective correctness at rho=0.86, and the weighting recovers the gold ranking of thirteen models at Spearman 0.95. Reliability comes from panel composition, not size: this label-free weighting zeroes out broken judges and down-weights saturated questions, so neither distorts the ranking. Generated items show zero verbatim overlap with five public benchmarks, the panel cancels verbosity bias and precludes same-family self-preference, and rankings are domain-specific: three different models top four de-novo domains, so a generic leaderboard misdirects most practitioners. The same pipeline reruns on each model release, giving any team a contamination-free leaderboard for its application.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 35 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

31 of 35 references.

References · 31

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  9. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  11. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  12. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
19 more
  1. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  2. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  3. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  4. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  5. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  6. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  7. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  8. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 44
  9. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  10. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  11. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  12. 2024Constructing Domain-Specific Evaluation Sets for LLM-as-a-judgecited by 30
  13. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
  14. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgmentscited by 16
  15. 2025Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorganacited by 13
  16. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
  17. 2025How to Get Your LLM to Generate Challenging Problems for Evaluationcited by 11
  18. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truthcited by 7
  19. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studiescited by 0
  2. 2026Clinical Intent Extraction: A FHIR-Aligned Representation and the CIRCA Benchmarkcited by 0