mlsanityData to 7 Oct 2026

AutoEval Done Right: Using Synthetic Data for Model Evaluation

arXiv 2403.07008 ↗First posted 9 Mar 2024cs.LG cs.AI cs.CLcited by 53

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of human annotations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this purpose that improve sample efficiency while remaining unbiased. These algorithms increase the effective human-labeled sample size by up to 50% on experiments with GPT-4.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 16 of its 44 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

16 of 44 references.

References · 16

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  6. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  7. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  8. 2014Learning with Pseudo-Ensemblescited by 241
  9. 2023Prediction-Powered Inferencecited by 214
  10. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  11. 2023PPI++: Efficient Prediction-Powered Inferencecited by 103
  12. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
4 more
  1. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  2. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  3. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  4. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 18; 2026: 24.

53 papers.

Sort:

By area:

  1. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  2. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  3. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  4. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 42
  5. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  6. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 25
  7. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  8. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  9. 2025Prediction-Powered E-Valuescited by 16
  10. 2025Cost-Optimal Active AI Model Evaluationcited by 14
  11. 2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
  12. 2024A Note on the Prediction-Powered Bootstrapcited by 10
38 more
  1. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 9
  2. 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guaranteescited by 9
  3. 2026Multiple-Prediction-Powered Inferencecited by 9
  4. 2025Multimodal Cultural Safety: Evaluation Framework and Alignment Strategiescited by 8
  5. 2025No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inferencecited by 8
  6. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 7
  7. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  8. 2024Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regressioncited by 6
  9. 2024Another look at statistical inference with machine learning-imputed datacited by 6
  10. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedbackcited by 6
  11. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimationcited by 6
  12. 2025General Synthetic-Powered Inferencecited by 5
  13. 2025Evaluating multiple models using labeled and unlabeled datacited by 4
  14. 2025Statistical Methods in Generative AIcited by 4
  15. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  16. 2026Revisiting Active Sequential Prediction-Powered Mean Estimationcited by 3
  17. 2024Prediction-Enhanced Monte Carlo: A Machine Learning View on Control Variatecited by 2
  18. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  19. 2026An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Datacited by 2
  20. 2026Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Boundscited by 2
  21. 2025QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictionscited by 1
  22. 2025Towards a rigorous evaluation of RAG systems: the challenge of due diligencecited by 1
  23. 2025Extending Prediction-Powered Inference through Conformal Predictioncited by 1
  24. 2026Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Modelscited by 1
  25. 2026Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signalscited by 1
  26. 2026Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorizationcited by 1
  27. 2026Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learningcited by 1
  28. 2026When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judgescited by 1
  29. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1
  30. 2026CAFE: A Compound-AI Factorial Evaluation Frameworkcited by 1
  31. 2026Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluationcited by 1
  32. 2026Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluationcited by 0
  33. 2026ReSAGE-PAR: Representational Similarity Assessment for Generative Expansion in Pedestrian Attribute Recognitioncited by 0
  34. 2026Beyond Independent Labels: Schwartz-Geometry Decoding for Human Value Detectioncited by 0
  35. 2026Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluationcited by 0
  36. 2026Accounting for Bias Enables Sustainable LLM Evaluationcited by 0
  37. 2026DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibrationcited by 0
  38. 2026Certified Selective Automation of LLM Agent Evaluationcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
  2. 2024Self-Taught Evaluators
  3. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models
  4. 2026Valid Inference with Synthetic Data via Task Exchangeability
  5. 2025Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judges
  6. 2024A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models
  7. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory
  8. 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets