# Exploring Precision and Recall to assess the quality and diversity of LLMs

arXiv: https://arxiv.org/abs/2402.10693 · HTML: https://mlsanity.com/paper/2402.10693/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [Neural Text Generation from Structured Data with Application to the Biography Domain](https://mlsanity.com/paper/1603.07771/) (2016)
- [TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models](https://mlsanity.com/paper/2306.08013/) (2023)
- [Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows](https://mlsanity.com/paper/2305.18910/) (2023)
- [On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation](https://mlsanity.com/paper/2205.16001/) (2022)

## Built on by (most cited first, 29 of 29)

- [Diverse Preference Optimization](https://mlsanity.com/paper/2501.18101/) (2025)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [Benchmarking Linguistic Diversity of Large Language Models](https://mlsanity.com/paper/2412.10271/) (2024)
- [From Distributional to Overton Pluralism: Investigating Large Language Model Alignment](https://mlsanity.com/paper/2406.17692/) (2024)
- [CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints](https://mlsanity.com/paper/2410.04197/) (2024)
- [Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models](https://mlsanity.com/paper/2508.10599/) (2025)
- [Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs](https://mlsanity.com/paper/2510.01218/) (2025)
- [How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code](https://mlsanity.com/paper/2503.00691/) (2025)
- [Improving Diversity in Language Models: When Temperature Fails, Change the Loss](https://mlsanity.com/paper/2508.09654/) (2025)
- [Measuring Diversity in Synthetic Datasets](https://mlsanity.com/paper/2502.08512/) (2025)
- [Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.03906/) (2025)
- [Program Synthesis via Test-Time Transduction](https://mlsanity.com/paper/2509.17393/) (2025)
- [Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations](https://mlsanity.com/paper/2601.08064/) (2026)
- [Investigating Numerical Translation with Large Language Models](https://mlsanity.com/paper/2501.04927/) (2025)
- [CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization](https://mlsanity.com/paper/2503.17136/) (2025)
- [Evaluating List Construction and Temporal Understanding capabilities of Large Language Models](https://mlsanity.com/paper/2506.21783/) (2025)
- [Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety](https://mlsanity.com/paper/2509.12936/) (2025)
- [Equalized Generative Treatment: Matching f-divergences for Fairness in Generative Models](https://mlsanity.com/paper/2602.08660/) (2026)
- [Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation](https://mlsanity.com/paper/2605.28022/) (2026)
- [AnveshanaAI: A Multimodal Platform for Adaptive AI/ML Education through Automated Question Generation and Interactive Assessment](https://mlsanity.com/paper/2509.23811/) (2025)
- [Measuring Dataset Diversity from a Geometric Perspective](https://mlsanity.com/paper/2602.09340/) (2026)
- [GENIE: A Fine-Grained Measure for Novelty](https://mlsanity.com/paper/2606.12790/) (2026)
- [Fidelity-Diversity Metrics for Text](https://mlsanity.com/paper/2607.04563/) (2026)
- [Evaluating Multiple LLM Generations with Validated Task Coverage](https://mlsanity.com/paper/2608.24228/) (2026)
- [Do Large Language Models Capture the Diversity in their Training Data?](https://mlsanity.com/paper/2609.02275/) (2026)
- [A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation](https://mlsanity.com/paper/2609.26527/) (2026)
- [SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics](https://mlsanity.com/paper/2610.06513/) (2026)
- [PLDiv: Measuring Dataset Diversity and Semantic Collapse via Persistence Landscapes](https://mlsanity.com/paper/2610.14031/) (2026)

## Abstract (arXiv)

We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data.