mlsanityData to 7 Oct 2026

Exploring the Limitations of Large Language Models in Compositional Relation Reasoning

arXiv 2403.02615 ↗First posted 5 Mar 2024cs.CLcited by 9

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We present a comprehensive evaluation of large language models(LLMs)' ability to reason about composition relations through a benchmark encompassing 1,500 test cases in English, designed to cover six distinct types of composition relations: Positional, Comparative, Personal, Mathematical, Identity, and Other. Acknowledging the significance of multilingual capabilities, we expanded our assessment to include translations of these cases into Chinese, Japanese, French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims at investigating the robustness and adaptability of LLMs in handling composition relation reasoning across diverse linguistic contexts.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 49 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

8 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise29 Oct
  2. 2022Towards Reasoning in Large Language Models: A Survey20 Dec

Other references · 6

  1. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 64
  2. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  3. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  4. 2017Relation Extraction : A Surveycited by 44
  5. 2022Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extractioncited by 44
  6. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 3.

9 papers.

  1. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  2. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  3. 2024Timo: Towards Better Temporal Reasoning for Language Modelscited by 21
  4. 2026Large Language Model Reasoning Failurescited by 20
  5. 2025Exploring the Necessity of Reasoning in LLM-based Agent Scenarioscited by 10
  6. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  7. 2024Cognitively Inspired Energy-Based World Modelscited by 1
  8. 2026An Alternative Trajectory for Generative AIcited by 1
  9. 2026Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Robust Knowledge Representations in Multilingual LLMs for Equivalence and Inheritance based Consistent Reasoning
  2. 2022Language Models are Multilingual Chain-of-Thought Reasoners
  3. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  4. 2026QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi
  5. 2024Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning
  6. 2026ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions
  7. 2024Understanding and Patching Compositional Reasoning in LLMs
  8. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs