mlsanityData to 14 Oct 2026

Semantic Content Determines Algorithmic Performance

arXiv 2601.21618 ↗First posted 29 Jan 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Counting should not depend on what is being counted; more generally, any algorithm's behavior should be invariant to the semantic content of its arguments. We introduce WhatCounts to test this property in isolation. Unlike prior work that conflates semantic sensitivity with reasoning complexity or prompt variation, WhatCounts is atomic: count items in an unambiguous, delimited list with no duplicates, distractors, or reasoning steps for different semantic types. Frontier LLMs show over 40% accuracy variation depending solely on what is being counted - cities versus chemicals, names versus symbols. Controlled ablations rule out confounds. The gap is semantic, and it shifts unpredictably with small amounts of unrelated fine-tuning. LLMs do not implement algorithms; they approximate them, and the approximation is argument-dependent. As we show with an agentic example, this has implications beyond counting: any LLM function may carry hidden dependencies on the meaning of its inputs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

30 of 42 references.

References · 30

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  7. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  9. 2021Thinking Like Transformerscited by 184
  10. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  11. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  12. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 68
18 more
  1. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
  2. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  3. 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 38
  4. 2024Counting Ability of Large Language Models and Impact of Tokenizationcited by 32
  5. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  6. 2024Transformers in Uniform TC0cited by 25
  7. 2024When Can Transformers Count to n?cited by 24
  8. 2024Prompts Are Programs Too! Understanding How Developers Build Software Containing Promptscited by 19
  9. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  10. 2024Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilitiescited by 13
  11. 2023Counting and Algorithmic Generalization with Transformerscited by 12
  12. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  13. 2025The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Modelscited by 11
  14. 2025Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systemscited by 10
  15. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 9
  16. 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errorscited by 9
  17. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  18. 2025SortBench: Benchmarking LLMs based on their ability to sort listscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems
  2. 2025Semantic Deception: When Reasoning Models Can't Compute an Addition
  3. 2026Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal reference
  4. 2026SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation
  5. 2025A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs
  6. 2026Counting on Thinking: Tracing Evidence Integration in Language Models
  7. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
  8. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs