mlsanityData to 14 Oct 2026

Benchmarking Foundation Models with Language-Model-as-an-Examiner

arXiv 2306.04181 ↗First posted 7 Jun 2023cs.CL cs.LGcited by 172

Not on the map. Why?

Cited by 27 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model's ability to understand and generate language in a manner similar to humans. Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and evaluation automation. In this paper, we propose a novel benchmarking framework, Language-Model-as-an-Examiner, where the LM serves as a knowledgeable examiner that formulates questions based on its knowledge and evaluates responses in a reference-free manner. Our framework allows for effortless extensibility as various LMs can be adopted as the examiner, and the questions can be constantly updated given more diverse trigger topics. For a more comprehensive and equitable evaluation, we devise three strategies: (1) We instruct the LM examiner to generate questions across a multitude of domains to probe for a broad acquisition, and raise follow-up questions to engage in a more in-depth assessment. (2) Upon evaluation, the examiner combines both scoring and ranking measurements, providing a reliable result as it aligns closely with human annotations. (3) We additionally propose a decentralized Peer-examination method to address the biases in a single examiner. Our data and benchmarking results are available at: http://lmexam.xlore.cn.

Key facts. Cited by 172 papers in the map (data through 14 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

40 of 62 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
28 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  12. 2018CoQA: A Conversational Question Answering Challengecited by 683
  13. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  14. 2019ELI5: Long Form Question Answeringcited by 575
  15. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  16. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  17. 2018QuAC : Question Answering in Contextcited by 381
  18. 2023GPTScore: Evaluate as You Desirecited by 361
  19. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  20. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  21. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  22. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  23. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  24. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  25. 2020DoQA -- Accessing Domain-Specific FAQs via Conversational QAcited by 42
  26. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  27. 2021What's in a Name? Answer Equivalence For Open-Domain Question Answeringcited by 21
  28. 2022StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoningcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 76; 2025: 52; 2026: 27.

172 papers.

Sort:

By area:

  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  5. 2024Self-Rewarding Language Modelscited by 681
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  7. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  8. 2023Large Language Model Alignment: A Surveycited by 243
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  12. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
38 more
  1. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  2. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  3. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  4. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  5. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  6. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  7. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  8. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  9. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  10. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  11. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  12. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  13. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  14. 2024Self-Taught Evaluatorscited by 72
  15. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  16. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  17. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generationcited by 57
  18. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  19. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  20. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  21. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 44
  22. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  23. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 42
  24. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  25. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  26. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  27. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  28. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  29. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  30. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 34
  31. 2024Direct Judgement Preference Optimizationcited by 33
  32. 2023Aligning Large Language Models with Human: A Surveycited by 32
  33. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 31
  34. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generationcited by 29
  35. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  36. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  37. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  38. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27

50 most cited of 172 All 172 (JSON)