mlsanityData to 7 Oct 2026

Caliper: Probing Lexical Anchors versus Causal Structure in LLMs

arXiv 2606.04915 ↗First posted 3 Jun 2026cs.CL cs.IRcited by 2

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Large language models reach 50 to 70% accuracy on causal reasoning benchmarks such as CLadder, but it is unclear whether this reflects structural reasoning or lexical pattern matching. We introduce Caliper, a controlled perturbation that replaces semantic variable names with placeholder tokens while preserving the causal graph and probabilistic specification of each question. Across nine instruction-tuned LLMs from 3.8B to 671B and three causal reasoning benchmarks, lexical anonymization yields robust accuracy drops of +7.6, +27.0, and +11.1 pp on a local 3.8B-14B set, rising to +29.6 and +18.0 pp on CRASS and e-CARE across nine frontier models spanning the 2024-2026 generations. Of 40 engaged model-by-benchmark cells, 39 show a positive gap, and the gap collapses by 17x on CLadder's pseudoword subset. Structured scaffolding and few-shot in-context learning each narrow the gap, but mainly by lowering P0 accuracy on smaller models rather than recovering P1. Current instruction-tuned LLMs, evaluated zero-shot, show little evidence of structural causal reasoning once lexical anchors are removed.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 14 of its 24 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

14 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?26 Jun
  2. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality28 Apr

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2022Data Contamination: From Memorization to Exploitationcited by 234
  7. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  8. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  9. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  10. 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMscited by 20
  11. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attributioncited by 14
  12. 2024On the Reliability of Large Language Models for Causal Discoverycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CausalDS: Benchmarking Causal Reasoning in Data-Science Agentscited by 1
  2. 2026Vernier: Probing Representational Misalignment Behind Lexical Gaps in Causal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
  2. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models
  3. 2023CLadder: Assessing Causal Reasoning in Language Models
  4. 2024Large Language Models and Causal Inference in Collaboration: A Survey
  5. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
  6. 2026METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
  7. 2025Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
  8. 2024Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs