mlsanityData to 14 Oct 2026

A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks

arXiv 2405.10251 ↗First posted 16 May 2024cs.CLcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent efforts have evaluated large language models (LLMs) in areas such as commonsense reasoning, mathematical reasoning, and code generation. However, to the best of our knowledge, no work has specifically investigated the performance of LLMs in natural language generation (NLG) tasks, a pivotal criterion for determining model excellence. Thus, this paper conducts a comprehensive evaluation of well-known and high-performing LLMs, namely ChatGPT, ChatGLM, T5-based models, LLaMA-based models, and Pythia-based models, in the context of NLG tasks. We select English and Chinese datasets encompassing Dialogue Generation and Text Summarization. Moreover, we propose a common evaluation setting that incorporates input templates and post-processing strategies. Our study reports both automatic results, accompanied by a detailed analysis.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 40 of its 58 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

40 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
27 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2018Hierarchical Neural Story Generationcited by 1,193
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  10. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  13. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  14. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  15. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  16. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  17. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  18. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  19. 2019MoEL: Mixture of Empathetic Listenerscited by 91
  20. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  21. 2017Event Representations for Automated Story Generation with Deep Neural Netscited by 63
  22. 2018DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encodercited by 44
  23. 2021LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generationcited by 24
  24. 2018Incorporating Structured Commonsense Knowledge in Story Completioncited by 23
  25. 2021Outline to Story: Fine-grained Controllable Story Generation from Cascaded Eventscited by 8
  26. 2023CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generationcited by 8
  27. 2022Event Transition Planning for Open-ended Text Generationcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretrainingcited by 4
  2. 2025Quality Assurance for LLM-RAG Systems: Empirical Insights from Tourism Application Testingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
  3. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
  4. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitations
  5. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  6. 2023A Comprehensive Overview of Large Language Models
  7. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
  8. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks