mlsanityData to 7 Oct 2026

Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA

arXiv 2408.09235 ↗First posted 17 Aug 2024cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 17 Aug 2024.

Builds on

21 of 38 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  12. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
9 more
  1. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  2. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  3. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  4. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  5. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 88
  6. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  7. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  8. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  9. 2024Latent Concept-based Explanation of NLP Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 14.

25 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Contextcited by 25
  3. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  4. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 12
  5. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  6. 2025CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answeringcited by 6
  7. 2025Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoningcited by 4
  8. 2025M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Modelscited by 4
  9. 2024AT-RAG: An Adaptive RAG Model Enhancing Query Efficiency with Topic Filtering and Iterative Reasoningcited by 3
  10. 2025CARGO: A Framework for Confidence-Aware Routing of Large Language Modelscited by 3
  11. 2025SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QAcited by 2
  12. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
13 more
  1. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guaranteescited by 1
  2. 2026COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learningcited by 1
  3. 2025Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoringcited by 0
  4. 2025Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddlescited by 0
  5. 2026TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stoppingcited by 0
  6. 2026NEURON: A Neuro-symbolic System for Grounded Clinical Explainabilitycited by 0
  7. 2026Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Taskscited by 0
  8. 2026CIRCLED: A Multi-turn CIR Dataset with Consistent Dialogues across Domainscited by 0
  9. 2026Storyline Trees: Hierarchical Representations for Long-Form Narrativescited by 0
  10. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  11. 2026BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Servingcited by 0
  12. 2026Automatic Hard Example Synthesis with Multi-Level Agentic Data Curationcited by 0
  13. 2026VikingRAG: Accurate and Token-efficient Retrieval-augmented Generation over Structured Documentscited by 0