# Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models

arXiv: https://arxiv.org/abs/2502.12821 · HTML: https://mlsanity.com/paper/2502.12821/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data](https://mlsanity.com/paper/2407.14985/) (2024)
- [Factuality of Large Language Models: A Survey](https://mlsanity.com/paper/2402.02420/) (2024)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilities](https://mlsanity.com/paper/2409.07638/) (2024)
- [Can Large Language Models generalize analogy solving like children can?](https://mlsanity.com/paper/2411.02348/) (2024)
- [Can Large Language Models Learn Independent Causal Mechanisms?](https://mlsanity.com/paper/2402.02636/) (2024)
- [Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs](https://mlsanity.com/paper/2405.11880/) (2024)
- [RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation](https://mlsanity.com/paper/2409.16383/) (2024)

## Built on by (most cited first, 1 of 1)

- [Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning](https://mlsanity.com/paper/2601.18352/) (2026)

## Abstract (arXiv)

Inverse tasks can uncover potential reasoning gaps as Large Language Models (LLMs) scale up. In this work, we explore the redefinition task, in which we assign alternative values to well-known physical constants and units of measure, prompting LLMs to respond accordingly. Our findings show that not only does model performance degrade with scale, but its false confidence also rises. Moreover, while factors such as prompting strategies or response formatting are influential, they do not preclude LLMs from anchoring to memorized values.