mlsanityData to 14 Oct 2026

Benchmarking Large Language Model Capabilities for Conditional Generation

arXiv 2306.16793 ↗First posted 29 Jun 2023cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Pre-trained large language models (PLMs) underlie most new developments in natural language processing. They have shifted the field from application-specific model pipelines to a single model that is adapted to a wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside techniques like few-shot learning, have additionally shifted the output modality to generation instead of classification or regression. Despite their ubiquitous use, the generation quality of language models is rarely evaluated when these models are introduced. Additionally, it is unclear how existing generation tasks--while they can be used to compare systems at a high level--relate to the real world use cases for which people have been adopting them. In this work, we discuss how to adapt existing application-specific generation benchmarks to PLMs and provide an in-depth, empirical study of the limitations and capabilities of PLMs in natural language generation tasks along dimensions such as scale, architecture, input and output language. Our results show that PLMs differ in their applicability to different data regimes and their generalization to multiple languages and inform which PLMs to use for a given generation task setup. We share best practices to be taken into consideration when benchmarking generation capabilities during the development of upcoming PLMs.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

30 of 43 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
18 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  5. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  7. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  8. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  9. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  12. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  13. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  14. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  15. 2021Learning Compact Metrics for MTcited by 93
  16. 2022Evaluating Human-Language Model Interactioncited by 92
  17. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  18. 2021Automatic Construction of Evaluation Suites for Natural Language Generation Datasetscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 18; 2025: 5; 2026: 2.

29 papers.

Sort:

By area:

  1. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  2. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  3. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  4. 2024Spatial-Temporal Large Language Model for Traffic Predictioncited by 83
  5. 2023Efficient Benchmarking of Language Modelscited by 71
  6. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  7. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  8. 2024LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Studycited by 31
  9. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  10. 2024Label-Efficient Model Selection for Text Generationcited by 19
  11. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  12. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
17 more
  1. 2025TableMaster: A Recipe to Advance Table Understanding with Language Modelscited by 18
  2. 2024CEval: A Benchmark for Evaluating Counterfactual Text Generationcited by 15
  3. 2025Scaling Up Active Testing to Large Language Modelscited by 13
  4. 2025Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performancecited by 12
  5. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  6. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  7. 2024Combining Hierachical VAEs with LLMs for clinically meaningful timeline summarisation in social mediacited by 8
  8. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  9. 2024Faithful Chart Summarization with ChaTS-Picited by 5
  10. 2024Selectively Answering Visual Questionscited by 5
  11. 2024Revisiting Meta-evaluation for Grammatical Error Correctioncited by 3
  12. 2024On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directionscited by 3
  13. 2024KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmarkcited by 2
  14. 2026Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Datacited by 2
  15. 2024Text2Traj2Text: Learning-by-Synthesis Framework for Contextual Captioning of Human Movement Trajectoriescited by 1
  16. 2025LCTG Bench: LLM Controlled Text Generation Benchmarkcited by 1
  17. 2026Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speechcited by 0