# Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning

arXiv: https://arxiv.org/abs/2404.16807 · HTML: https://mlsanity.com/paper/2404.16807/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning](https://mlsanity.com/paper/2009.12677/) (2020)
- [Mixture Models for Diverse Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/1902.07816/) (2019)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [A Deep Generative Framework for Paraphrase Generation](https://mlsanity.com/paper/1709.05074/) (2017)
- [SemEval-2020 Task 4: Commonsense Validation and Explanation](https://mlsanity.com/paper/2007.00236/) (2020)
- [An Enhanced Knowledge Injection Model for Commonsense Generation](https://mlsanity.com/paper/2012.00366/) (2020)
- [Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts](https://mlsanity.com/paper/2203.07285/) (2022)
- [Deliberate then Generate: Enhanced Prompting Framework for Text Generation](https://mlsanity.com/paper/2305.19835/) (2023)
- [DimonGen: Diversified Generative Commonsense Reasoning for Explaining Concept Relationships](https://mlsanity.com/paper/2212.10545/) (2022)
- [Learning to Predict Concept Ordering for Common Sense Generation](https://mlsanity.com/paper/2309.06363/) (2023)

## Built on by (most cited first, 19 of 19)

- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning](https://mlsanity.com/paper/2408.02871/) (2024)
- [Pluralistic Off-policy Evaluation and Alignment](https://mlsanity.com/paper/2509.19333/) (2025)
- [Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability](https://mlsanity.com/paper/2506.15629/) (2025)
- [BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation](https://mlsanity.com/paper/2502.01697/) (2025)
- [Measuring Diversity in Synthetic Datasets](https://mlsanity.com/paper/2502.08512/) (2025)
- [Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombination](https://mlsanity.com/paper/2504.20643/) (2025)
- [Adaptive Instruction Composition for Automated LLM Red-Teaming](https://mlsanity.com/paper/2604.21159/) (2026)
- [Inducing Sustained Creativity and Diversity in Large Language Models](https://mlsanity.com/paper/2603.19519/) (2026)
- [Where You Inject Diversity Matters: A Unified Framework for Diverse Generation](https://mlsanity.com/paper/2606.10302/) (2026)
- [The Mystery of Compositional Generalization in Graph-based Generative Commonsense Reasoning](https://mlsanity.com/paper/2410.06272/) (2024)
- [Anchorless Diversification for Parallel LLM Ideation](https://mlsanity.com/paper/2605.30150/) (2026)
- [emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity](https://mlsanity.com/paper/2607.19848/) (2026)
- [Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting](https://mlsanity.com/paper/2408.06186/) (2024)
- [CaseEdit: Enhancing Localized Commonsense Reasoning via Null-Space Constrained Knowledge Editing in Small Parameter Language Models](https://mlsanity.com/paper/2505.19383/) (2025)
- [EduAgentQG: Multi-Agent Personalized Mathematics Question Generation with Explicit Diversity and Objective-Aware Evaluation](https://mlsanity.com/paper/2511.11635/) (2025)
- [ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling](https://mlsanity.com/paper/2605.22102/) (2026)
- [MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions](https://mlsanity.com/paper/2606.14715/) (2026)
- [Initialization Improves LLM-Driven Discovery](https://mlsanity.com/paper/2610.00707/) (2026)

## Abstract (arXiv)

Generative Commonsense Reasoning (GCR) requires a model to reason about a situation using commonsense knowledge, while generating coherent sentences. Although the quality of the generated sentences is crucial, the diversity of the generation is equally important because it reflects the model's ability to use a range of commonsense knowledge facts. Large Language Models (LLMs) have shown proficiency in enhancing the generation quality across various tasks through in-context learning (ICL) using given examples without the need for any fine-tuning. However, the diversity aspect in LLM outputs has not been systematically studied before. To address this, we propose a simple method that diversifies the LLM generations, while preserving their quality. Experimental results on three benchmark GCR datasets show that our method achieves an ideal balance between the quality and diversity. Moreover, the sentences generated by our proposed method can be used as training data to improve diversity in existing commonsense generators.