mlsanityData to 14 Oct 2026

PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

arXiv 2306.05087 ↗First posted 8 Jun 2023cs.CL cs.AIcited by 316

Not on the map. Why?

Cited by 44 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned models. To determine the optimal hyperparameters, an automatic, robust, and reliable evaluation benchmark is essential. However, establishing such a benchmark is not a trivial task due to the challenges associated with evaluation accuracy and privacy protection. In response to these challenges, we introduce a judge large language model, named PandaLM, which is trained to distinguish the superior model given several LLMs. PandaLM's focus extends beyond just the objective correctness of responses, which is the main focus of traditional evaluation datasets. It addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. To ensure the reliability of PandaLM, we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences. Our results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM enables the evaluation of LLM to be fairer but with less cost, evidenced by significant improvements achieved by models tuned through PandaLM compared to their counterparts trained with default Alpaca's hyperparameters. In addition, PandaLM does not depend on API-based evaluations, thus avoiding potential data leakage. All resources of PandaLM are released at https://github.com/WeOpenML/PandaLM.

Key facts. Cited by 316 papers in the map (data through 14 Oct 2026). Builds on 55 of its 90 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

55 of 90 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  9. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  10. 2023Instruction Tuning with GPT-4cited by 686
  11. 2018CoQA: A Conversational Question Answering Challengecited by 683
  12. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  13. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  14. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  15. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  16. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  17. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  18. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  19. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  20. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  21. 2023SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learningcited by 133
  22. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  23. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  24. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  25. 2019Neural Machine Translation: A Review and Surveycited by 100
  26. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  27. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  28. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  29. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  30. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  31. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  32. 2019Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanationcited by 43
  33. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  34. 2023Exploring Vision-Language Models for Imbalanced Learningcited by 35
  35. 2023Evaluating Open-QA Evaluationcited by 35
  36. 2021Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysiscited by 30
  37. 2022A Rationale-Centric Framework for Human-in-the-loop Machine Learningcited by 27
  38. 2021The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLPcited by 19

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 40; 2024: 116; 2025: 89; 2026: 71.

316 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  5. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  12. 2023Challenges and Applications of Large Language Modelscited by 254
38 more
  1. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  3. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  4. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  5. 2023Generative Judge for Evaluating Alignmentcited by 169
  6. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  7. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  8. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  9. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  10. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  11. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  12. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  13. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  14. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  15. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  16. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  17. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  18. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
  19. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  20. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  21. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 70
  22. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  23. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  24. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  25. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  26. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  27. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  28. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  29. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  30. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  31. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 51
  32. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  33. 2024Improving Reward Models with Synthetic Critiquescited by 44
  34. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  35. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 42
  36. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  37. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  38. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40

50 most cited of 316 All 316 (JSON)