# The Mystery of Compositional Generalization in Graph-based Generative Commonsense Reasoning

arXiv: https://arxiv.org/abs/2410.06272 · HTML: https://mlsanity.com/paper/2410.06272/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 7 of its 89 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 7 of 7)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning](https://mlsanity.com/paper/2404.16807/) (2024)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [Do Fine-tuned Commonsense Language Models Really Generalize?](https://mlsanity.com/paper/2011.09159/) (2020)

## Built on by (most cited first, 1 of 1)

- [From Argumentation to Deliberation: Perspectivized Stance Vectors for Fine-grained (Dis)agreement Analysis](https://mlsanity.com/paper/2502.09644/) (2025)

## Abstract (arXiv)

While LLMs have emerged as performant architectures for reasoning tasks, their compositional generalization capabilities have been questioned. In this work, we introduce a Compositional Generalization Challenge for Graph-based Commonsense Reasoning (CGGC) that goes beyond previous evaluations that are based on sequences or tree structures - and instead involves a reasoning graph: It requires models to generate a natural sentence based on given concepts and a corresponding reasoning graph, where the presented graph involves a previously unseen combination of relation types. To master this challenge, models need to learn how to reason over relation tupels within the graph, and how to compose them when conceptualizing a verbalization. We evaluate seven well-known LLMs using in-context learning and find that performant LLMs still struggle in compositional generalization. We investigate potential causes of this gap by analyzing the structures of reasoning graphs, and find that different structures present varying levels of difficulty for compositional generalization. Arranging the order of demonstrations according to the structures' difficulty shows that organizing samples in an easy-to-hard schema enhances the compositional generalization ability of LLMs.