mlsanityData to 7 Oct 2026

Exploring Precision and Recall to assess the quality and diversity of LLMs

arXiv 2402.10693 ↗First posted 16 Feb 2024cs.CL cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

29 of 47 references.

References · 29

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
17 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  4. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  5. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  7. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  8. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  9. 2018Assessing Generative Models via Precision and Recallcited by 401
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 193
  13. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  14. 2016Neural Text Generation from Structured Data with Application to the Biography Domaincited by 20
  15. 2023TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Modelscited by 13
  16. 2023Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flowscited by 9
  17. 2022On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 9.

27 papers.

Sort:

By area:

  1. 2025Diverse Preference Optimizationcited by 48
  2. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  3. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38
  4. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  5. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  6. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  7. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  8. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  9. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9
  10. 2025Improving Diversity in Language Models: When Temperature Fails, Change the Losscited by 7
  11. 2025Measuring Diversity in Synthetic Datasetscited by 5
  12. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
15 more
  1. 2025Program Synthesis via Test-Time Transductioncited by 4
  2. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  3. 2025Investigating Numerical Translation with Large Language Modelscited by 1
  4. 2025CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalizationcited by 1
  5. 2025Evaluating List Construction and Temporal Understanding capabilities of Large Language Modelscited by 1
  6. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safetycited by 1
  7. 2026Equalized Generative Treatment: Matching f-divergences for Fairness in Generative Modelscited by 1
  8. 2025AnveshanaAI: A Multimodal Platform for Adaptive AI/ML Education through Automated Question Generation and Interactive Assessmentcited by 0
  9. 2026Measuring Dataset Diversity from a Geometric Perspectivecited by 0
  10. 2026Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generationcited by 0
  11. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  12. 2026Fidelity-Diversity Metrics for Textcited by 0
  13. 2026Evaluating Multiple LLM Generations with Validated Task Coveragecited by 0
  14. 2026Do Large Language Models Capture the Diversity in their Training Data?cited by 0
  15. 2026A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generationcited by 0