mlsanityData to 14 Oct 2026

CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density

arXiv 2509.18458 ↗First posted 22 Sep 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad, a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT's core dimensions: intrinsic difficulty (d) controls intrinsic load; distractor-to-signal ratio (ρ) regulates extraneous load; and task length (N) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances to intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogniLoad provides a reproducible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 34 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

23 of 34 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models24 Feb
  2. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?14 Feb
  3. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning3 Feb
  4. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion25 Jan
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks19 Dec
  6. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries19 Sep
  7. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  6. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  7. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  8. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  9. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  10. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  11. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  12. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
4 more
  1. 2024ARC Prize 2024: Technical Reportcited by 104
  2. 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 69
  3. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 17
  4. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMscited by 2
  2. 2026FlowMind: Execute-Summarize for Structured Workflow Generation from LLM Reasoningcited by 0
  3. 2026Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cognitive Load Limits in Large Language Models: Benchmarking Multi-Hop Reasoning
  2. 2025Cognitive Load-Aware Inference: A Neuro-Symbolic Framework for Optimizing the Token Economy of Large Language Models
  3. 2025From Chaos to Order: The Atomic Reasoner Framework for Fine-grained Reasoning in Large Language Models
  4. 2025Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
  5. 2026ReEfBench: Quantifying the Reasoning Efficiency of LLMs
  6. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  7. 2024DynaThink: Fast or Slow? A Dynamic Decision-Making Framework for Large Language Models
  8. 2023Concise and Organized Perception Facilitates Reasoning in Large Language Models