mlsanityData to 14 Oct 2026

The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models

arXiv 2406.05761 ↗First posted 9 Jun 2024cs.CLcited by 93

Not on the map. Why?

Abstract (arXiv)

As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria like helpfulness and harmlessness, which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on specific capabilities such as instruction following, leading to coverage bias. To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation. We apply this benchmark to assess 103 frontier LMs using five evaluator LMs. Our code, data, and evaluation results are all publicly available at https://github.com/prometheus-eval/prometheus-eval/tree/main/BiGGen-Bench.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 129 of its 183 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

129 of 183 references.

References · 129

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2023Let's Verify Step by Stepcited by 3,789
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2024Mixtral of Expertscited by 1,823
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  14. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  15. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  16. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  17. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  18. 2023LIMA: Less Is More for Alignmentcited by 1,100
  19. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  20. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  21. 2019How Can We Know What Language Models Know?cited by 959
  22. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  23. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  24. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  25. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  26. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  27. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  28. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  29. 2022Large Language Models Can Self-Improvecited by 664
  30. 2022PAL: Program-aided Language Modelscited by 652
  31. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  32. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  33. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  34. 2019ELI5: Long Form Question Answeringcited by 575
  35. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  36. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  37. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  38. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523

50 most cited of 129 All 129 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 48; 2026: 27.

93 papers.

Sort:

By area:

  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  2. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  3. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  5. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  6. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  7. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  8. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  9. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  10. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  11. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  12. 2024Direct Judgement Preference Optimizationcited by 33
38 more
  1. 2024Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoningcited by 33
  2. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  3. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 31
  4. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
  5. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  6. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  7. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  8. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  9. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 19
  10. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 18
  11. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  12. 2025ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memorycited by 14
  13. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  14. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  15. 2025Set-LLM: A Permutation-Invariant LLMcited by 11
  16. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  17. 2024PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluationcited by 10
  18. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  19. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  20. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  21. 2025INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profilingcited by 8
  22. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 8
  23. 2025SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoderscited by 7
  24. 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gapcited by 7
  25. 2025Quantitative LLM Judgescited by 6
  26. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  27. 2025SPRI: Aligning Large Language Models with Context-Situated Principlescited by 5
  28. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  29. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Toolcited by 5
  30. 2025DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimizationcited by 5
  31. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  32. 2025Evaluating Legal Reasoning Traces with Legal Issue Tree Rubricscited by 5
  33. 2024Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Educationcited by 4
  34. 2025MetaSC: Test-Time Safety Specification Optimization for Language Modelscited by 4
  35. 2025Trillion 7B Technical Reportcited by 4
  36. 2025Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Modelscited by 4
  37. 2025VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understandingcited by 4
  38. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4

50 most cited of 93 All 93 (JSON)