mlsanityData to 7 Oct 2026

The Concrete-Arbitrary Gap: Kinship Reasoning in LLMs Is Not Indifferent to Presentation

arXiv 2609.39913 ↗First posted 30 Sep 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We test whether large language models solve formally matched kinship problems equally well when relations are expressed in familiar vocabulary or by explicitly defined nonce predicates. Across 500 paired graphs, concrete accuracy exceeds arbitrary accuracy by 35.6 percentage points in local Qwen3.8-27B, 26.6 in Gemma 4 26B-A4B, 12.0 in Gemma 4 31B, and 5.4 in Qwen3.8-Max. All four paired gaps are statistically resolved. Reasoning budgets and prompt-language interventions can substantially reduce the difference, showing that it is modifiable rather than a fixed incapacity. The minimal conclusion is behavioral: on these tasks, the models' manifested relational competence is not indifferent to presentation. Explicit definitions provide the formal relations but do not make nonce predicates as usable as familiar vocabulary embedded in learned linguistic associations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 2 of its 16 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

2 of 16 references.

References · 2

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Kinship Data Benchmark for Multi-hop Reasoning
  2. 2022On Reality and the Limits of Language Data: Aligning LLMs with Human Norms
  3. 2026BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction
  4. 2025The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
  5. 2025Normative Reasoning in Large Language Models: A Comparative Benchmark from Logical and Modal Perspectives
  6. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  7. 2026When a Name Is Not a Name: A Benchmark Dataset and Distilled Reasoning for Culturally Entangled Bangla Homographs in Low-Resource LLMs
  8. 2026Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning