# Exploring the Limitations of Large Language Models in Compositional Relation Reasoning

arXiv: https://arxiv.org/abs/2403.02615 · HTML: https://mlsanity.com/paper/2403.02615/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 8 of its 49 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 8 of 8)

- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://mlsanity.com/paper/1904.01172/) (2019)
- [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://mlsanity.com/paper/2311.09835/) (2023)
- [Discrete and Soft Prompting for Multilingual Models](https://mlsanity.com/paper/2109.03630/) (2021)
- [Relation Extraction : A Survey](https://mlsanity.com/paper/1712.05191/) (2017)
- [Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://mlsanity.com/paper/2203.10316/) (2022)
- [Prompt2Model: Generating Deployable Models from Natural Language Instructions](https://mlsanity.com/paper/2308.12261/) (2023)
- [TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise](https://mlsanity.com/paper/2310.19019/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Survey of Multilingual Reasoning in Language Models](https://mlsanity.com/paper/2502.09457/) (2025)
- [SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning](https://mlsanity.com/paper/2407.12874/) (2024)
- [Timo: Towards Better Temporal Reasoning for Language Models](https://mlsanity.com/paper/2406.14192/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Exploring the Necessity of Reasoning in LLM-based Agent Scenarios](https://mlsanity.com/paper/2503.11074/) (2025)
- [Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data](https://mlsanity.com/paper/2409.12437/) (2024)
- [Cognitively Inspired Energy-Based World Models](https://mlsanity.com/paper/2406.08862/) (2024)
- [An Alternative Trajectory for Generative AI](https://mlsanity.com/paper/2603.14147/) (2026)
- [Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations](https://mlsanity.com/paper/2609.03511/) (2026)

## Abstract (arXiv)

We present a comprehensive evaluation of large language models(LLMs)' ability to reason about composition relations through a benchmark encompassing 1,500 test cases in English, designed to cover six distinct types of composition relations: Positional, Comparative, Personal, Mathematical, Identity, and Other. Acknowledging the significance of multilingual capabilities, we expanded our assessment to include translations of these cases into Chinese, Japanese, French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims at investigating the robustness and adaptability of LLMs in handling composition relation reasoning across diverse linguistic contexts.