mlsanityData to 14 Oct 2026

Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models

arXiv 2502.12821 ↗First posted 18 Feb 2025cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Inverse tasks can uncover potential reasoning gaps as Large Language Models (LLMs) scale up. In this work, we explore the redefinition task, in which we assign alternative values to well-known physical constants and units of measure, prompting LLMs to respond accordingly. Our findings show that not only does model performance degrade with scale, but its false confidence also rises. Moreover, while factors such as prompting strategies or response formatting are influential, they do not preclude LLMs from anchoring to memorized values.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 17 of its 37 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

17 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024On Memorization of Large Language Models in Logical Reasoning30 Oct
  2. 2024RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation24 Sep
  3. 2024Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs20 May
  4. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey17 Feb
  5. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  6. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  7. 2023Can Large Language Models Infer Causation from Correlation?cited by 146
  8. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 85
  9. 2024Factuality of Large Language Models: A Surveycited by 66
  10. 2024Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilitiescited by 13
  11. 2024Can Large Language Models generalize analogy solving like children can?cited by 9
  12. 2024Can Large Language Models Learn Independent Causal Mechanisms?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Superficial Alignment Hypothesis
  2. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
  3. 2025Reasoning Capabilities and Invariability of Large Language Models
  4. 2025On the Fundamental Limits of LLMs at Scale
  5. 2024Keep Guessing? When Considering Inference Scaling, Mind the Baselines
  6. 2026Brevity Constraints Reverse Performance Hierarchies in Language Models
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluations
  8. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead