mlsanityData to 7 Oct 2026

The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs

arXiv 2501.10970 ↗First posted 19 Jan 2025cs.CL cs.AI cs.HCcited by 72

Not on the map. Why?

Abstract (arXiv)

The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are widely used, not only in NLP research but also in fields like medicine, psychology, and social science. Despite their role in shaping study results and insights, there is no standard or rigorous procedure to determine whether LLMs can replace human annotators. In this paper, we propose a novel statistical procedure, the Alternative Annotator Test (alt-test), that requires only a modest subset of annotated examples to justify using LLM annotations. Additionally, we introduce a versatile and interpretable measure for comparing LLM annotators and judges. To demonstrate our procedure, we curated a diverse collection of ten datasets, consisting of language and vision-language tasks, and conducted experiments with six LLMs and four prompting techniques. Our results show that LLMs can sometimes replace humans with closed-source LLMs (such as GPT-4o), outperforming the open-source LLMs we examine, and that prompting techniques yield judges of varying quality. We hope this study encourages more rigorous and reliable practices.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 47 of its 79 references that are in the map. First posted on arXiv 19 Jan 2025.

Builds on

47 of 79 references.

References · 47

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  10. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
35 more
  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  7. 2023Prediction-Powered Inferencecited by 214
  8. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  9. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  10. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  11. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 102
  12. 2024Can LLM be a Personalized Judge?cited by 99
  13. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  14. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  15. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  16. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  17. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  18. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  19. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  20. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
  21. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  22. 2023Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languagescited by 36
  23. 2017Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasetscited by 32
  24. 2024Large Language Models: An Applied Econometric Frameworkcited by 30
  25. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  26. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  27. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  28. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  29. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  30. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  31. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  32. 2023Conflicts, Villains, Resolutions: Towards models of Narrative Media Framingcited by 14
  33. 2018Crowd disagreement about medical images is informativecited by 13
  34. 2023Self-Consistency of Large Language Models under Ambiguitycited by 12
  35. 2023CARE: Extracting Experimental Findings From Clinical Literaturecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 41.

72 papers.

Sort:

By area:

  1. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 52
  2. 2025Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Modelscited by 32
  3. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
  4. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 17
  5. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  6. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  7. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  8. 2026Benchmarks Saturate When The Model Gets Smarter Than The Judgecited by 10
  9. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  10. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9
  11. 2025Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?cited by 9
  12. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 8
38 more
  1. 2025LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderatorscited by 7
  2. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 7
  3. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 5
  4. 2025Efficient MAP Estimation of LLM Judgment Performance with Prior Transfercited by 4
  5. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  6. 2025Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketingcited by 4
  7. 2025But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectorscited by 3
  8. 2025ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clusteringcited by 3
  9. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  10. 2025TextBO: Bayesian Optimization in Language Space for Eval-Efficient Self-Improving AIcited by 3
  11. 2026STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systemscited by 3
  12. 2025Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awarenesscited by 2
  13. 2025Multi-Domain Explainability of Preferencescited by 2
  14. 2025Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gapscited by 2
  15. 2025Reward Models are Metrics in a Trench Coatcited by 2
  16. 2026Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreementcited by 2
  17. 2026Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfercited by 2
  18. 2026AI for Games in the Foundation Model Eracited by 2
  19. 2025Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Modelscited by 1
  20. 2025ASTRA: An Automated Framework for Strategy Discovery, Retrieval, and Evolution for Jailbreaking LLMscited by 1
  21. 2025GRAPH-GRPO-LEX: Contract Graph Modeling and Reinforcement Learning with Group Relative Policy Optimizationcited by 1
  22. 2025Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lenscited by 1
  23. 2026Judge Model for Large-scale Multimodality Benchmarkscited by 1
  24. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  25. 2026Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discoverycited by 1
  26. 2026Disentangling MLP Neuron Weights in Vocabulary Spacecited by 1
  27. 2026How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Modelscited by 1
  28. 2026Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judgecited by 1
  29. 2026The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignmentcited by 1
  30. 2026PhantomBench: Benchmarking the Non-existential Threat of Language Modelscited by 1
  31. 2026Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performancecited by 1
  32. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  33. 2026Conduct Under Pressure: What Sixty Language Models Do When a User Pushescited by 1
  34. 2025Embracing Dialectic Intersubjectivity: Coordination of Different Perspectives in Content Analysis with LLM Persona Simulationcited by 0
  35. 2025The Art of Audience Engagement: LLM-Based Thin-Slicing of Scientific Talkscited by 0
  36. 2025Recalibrating the Compass: Integrating Large Language Models into Classical Research Methodscited by 0
  37. 2025EduCoder: An Open-Source Annotation System for Education Transcript Datacited by 0
  38. 2025WeatherArchive-Bench: Benchmarking Retrieval-Augmented Reasoning for Historical Weather Archivescited by 0

50 most cited of 72 All 72 (JSON)