mlsanityData to 13 Oct 2026

Calibrating LLM-Based Evaluator

arXiv 2309.13308 ↗First posted 23 Sep 2023cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) on language modeling and emergent capabilities make them a promising reference-free evaluator of natural language generation quality, and a competent alternative to human evaluation. However, hindered by the closed-source or high computational demand to host and tune, there is a lack of practice to further calibrate an off-the-shelf LLM-based evaluator towards better human alignment. In this work, we propose AutoCalibrate, a multi-stage, gradient-free approach to automatically calibrate and align an LLM-based evaluator toward human preference. Instead of explicitly modeling human preferences, we first implicitly encompass them within a set of human labels. Then, an initial set of scoring criteria is drafted by the language model itself, leveraging in-context learning on different few-shot examples. To further calibrate this set of criteria, we select the best performers and re-draft them with self-refinement. Our experiments on multiple text quality evaluation datasets illustrate a significant improvement in correlation with expert evaluation through calibration. Our comprehensive qualitative analysis conveys insightful intuitions and observations on the essence of effective scoring criteria.

Key facts. Cited by 53 papers in the map (data through 13 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

32 of 40 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,068
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
20 more
  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,054
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,027
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 745
  5. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 720
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 668
  7. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 421
  8. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 416
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 377
  11. 2023GPTScore: Evaluate as You Desirecited by 361
  12. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 293
  13. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  14. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  15. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  16. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  17. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  18. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  19. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  20. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 22; 2025: 17; 2026: 12.

53 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 457
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 422
  3. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 224
  4. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 201
  5. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 147
  6. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 147
  7. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 117
  8. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 87
  9. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 72
  10. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 67
  11. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 61
  12. 2024Multicalibration for Confidence Scoring in LLMscited by 45
38 more
  1. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 41
  2. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  3. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  4. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  5. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 27
  7. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  8. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  9. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  10. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  11. 2024Automated radiotherapy treatment planning guided by GPT-4Visioncited by 12
  12. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  13. 2024Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factorscited by 9
  14. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  15. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applicationscited by 7
  16. 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluationcited by 7
  17. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  18. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  19. 2024Visual Prompting with Iterative Refinement for Design Critique Generationcited by 5
  20. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  21. 2026When to Act, Ask, or Learn: Uncertainty-Aware Policy Steeringcited by 5
  22. 2026A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judgescited by 4
  23. 2024Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarizationcited by 3
  24. 2024FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Modelscited by 2
  25. 2025Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQLcited by 2
  26. 2025EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generationcited by 2
  27. 2026BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectoriescited by 2
  28. 2025HypoEval: Hypothesis-Guided Evaluation for Natural Language Generationcited by 1
  29. 2025SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Modelscited by 1
  30. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAGcited by 1
  31. 2026The Paradox of Robustness: Decoupling Rule-Based Logic from Affective Noise in High-Stakes Decision-Makingcited by 1
  32. 2026ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awarenesscited by 1
  33. 2026Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judgescited by 1
  34. 2024Realizing Video Summarization from the Path of Language-based Semantic Understandingcited by 0
  35. 2024VARS: Vision-based Assessment of Risk in Security Systemscited by 0
  36. 2024Towards unearthing neglected climate innovations from scientific literature using Large Language Modelscited by 0
  37. 2026TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfictioncited by 0
  38. 2026Riazi-8B: An Urdu Large Language Model for Mathematical Reasoningcited by 0

50 most cited of 53 All 53 (JSON)