mlsanityData to 7 Oct 2026

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

arXiv 2603.09403 ↗First posted 10 Mar 2026cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 12 of its 78 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

12 of 78 references.

References · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  8. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  9. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  10. 2025Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shotcited by 12
  11. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  12. 2025Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  2. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  3. 2024METAL: Towards Multilingual Meta-Evaluation
  4. 2025Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering
  5. 2026BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
  6. 2024Efficacy of Synthetic Data as a Benchmark
  7. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  8. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora