# Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

arXiv: https://arxiv.org/abs/2203.07285 · HTML: https://mlsanity.com/paper/2203.07285/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [A Survey of Knowledge-Enhanced Text Generation](https://mlsanity.com/paper/2010.04389/) (2020)
- [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning](https://mlsanity.com/paper/2009.12677/) (2020)
- [Mixture Models for Diverse Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/1902.07816/) (2019)
- [KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering](https://mlsanity.com/paper/2110.04330/) (2021)
- [Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs](https://mlsanity.com/paper/1911.02707/) (2019)
- [Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph](https://mlsanity.com/paper/2009.11692/) (2020)
- [A Deep Generative Framework for Paraphrase Generation](https://mlsanity.com/paper/1709.05074/) (2017)
- [Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanation](https://mlsanity.com/paper/1906.00363/) (2019)
- [SemEval-2020 Task 4: Commonsense Validation and Explanation](https://mlsanity.com/paper/2007.00236/) (2020)
- [Toward Diverse Text Generation with Inverse Reinforcement Learning](https://mlsanity.com/paper/1804.11258/) (2018)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Mixture Content Selection for Diverse Sequence Generation](https://mlsanity.com/paper/1909.01953/) (2019)
- [Enhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations](https://mlsanity.com/paper/2106.02974/) (2021)
- [Injecting Entity Types into Entity-Guided Text Generation](https://mlsanity.com/paper/2009.13401/) (2020)

## Built on by (most cited first, 19 of 19)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [Progressive-Hint Prompting Improves Reasoning in Large Language Models](https://mlsanity.com/paper/2304.09797/) (2023)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration](https://mlsanity.com/paper/2402.11441/) (2024)
- [Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning](https://mlsanity.com/paper/2404.16807/) (2024)
- [Generating Visual Spatial Description via Holistic 3D Scene Understanding](https://mlsanity.com/paper/2305.11768/) (2023)
- [Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation](https://mlsanity.com/paper/2306.16195/) (2023)
- [Retrieval Augmentation for Commonsense Reasoning: A Unified Approach](https://mlsanity.com/paper/2210.12887/) (2022)
- [Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models](https://mlsanity.com/paper/2511.19822/) (2025)
- [Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability](https://mlsanity.com/paper/2506.15629/) (2025)
- [CADGE: Context-Aware Dialogue Generation Enhanced with Graph-Structured Knowledge Aggregation](https://mlsanity.com/paper/2305.06294/) (2023)
- [R3-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL](https://mlsanity.com/paper/2311.01862/) (2023)
- [Dense-ATOMIC: Towards Densely-connected ATOMIC with High Knowledge Coverage and Massive Multi-hop Paths](https://mlsanity.com/paper/2210.07621/) (2022)
- [Mixture of Length and Pruning Experts for Knowledge Graphs Reasoning](https://mlsanity.com/paper/2507.20498/) (2025)
- [RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models](https://mlsanity.com/paper/2505.08463/) (2025)
- [Very Large Language Model as a Unified Methodology of Text Mining](https://mlsanity.com/paper/2212.09271/) (2022)

## Abstract (arXiv)

Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.