mlsanityData to 7 Oct 2026

Inverse scaling can become U-shaped

arXiv 2211.02011 ↗First posted 3 Nov 2022cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

Scaling up language models has been empirically shown to improve performance on a wide range of downstream tasks. However, if we were to observe worse performance as a function of scale ("inverse scaling") on certain tasks, this would indicate that scaling can also encourage behaviors that are misaligned with human preferences. The Inverse Scaling Prize (McKenzie et al. 2022) identified eleven such inverse scaling tasks, evaluated on models of up to 280B parameters and up to 500 zettaFLOPs of training compute. This paper takes a closer look at these inverse scaling tasks. We evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and training compute, only four out of the eleven tasks remain inverse scaling. Six out of the eleven tasks exhibit "U-shaped scaling", where performance decreases up to a certain size, and then increases again up to the largest model evaluated (the one remaining task displays positive scaling). In addition, we find that 1-shot examples and chain-of-thought can help mitigate undesirable scaling patterns even further. U-shaped scaling suggests that the inverse scaling trend observed in McKenzie et al. (2022) may not continue to hold for larger models, which we attribute to the presence of distractor tasks that only sufficiently large models can avoid.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

13 of 21 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
1 more
  1. 2022Predictability and Surprise in Large Generative Modelscited by 209

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 24; 2024: 15; 2025: 16; 2026: 17.

76 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  6. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  7. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  8. 2023Model evaluation for extreme riskscited by 167
  9. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  10. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  11. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  12. 2023Language Model Behavior: A Comprehensive Surveycited by 92
38 more
  1. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  4. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  5. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  6. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 46
  7. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  8. 2026When Single-Agent with Skills Replace Multi-Agent Systems and When They Failcited by 34
  9. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  10. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  11. 2023Conformal Nucleus Samplingcited by 32
  12. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  13. 2023NevIR: Negation in Neural Information Retrievalcited by 28
  14. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 27
  15. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  16. 2023Scaling Laws Do Not Scalecited by 13
  17. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 13
  18. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  19. 2023Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Modelscited by 12
  20. 2024Tele-FLM Technical Reportcited by 12
  21. 2025NegVQA: Can Vision Language Models Understand Negation?cited by 12
  22. 2024Evidence of a log scaling law for political persuasion with large language modelscited by 11
  23. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  24. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  25. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  26. 2026Shaping capabilities with token-level data filteringcited by 11
  27. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  28. 2023An Investigation of LLMs' Inefficacy in Understanding Converse Relationscited by 9
  29. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 8
  30. 2025Random Scaling of Emergent Capabilitiescited by 7
  31. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
  32. 2023Probing Quantifier Comprehension in Large Language Models: Another Example of Inverse Scalingcited by 5
  33. 2025ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  34. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  35. 2022Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifierscited by 4
  36. 2023MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classificationcited by 4
  37. 2024Language Models Don't Learn the Physical Manifestation of Languagecited by 4
  38. 2024Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attackscited by 4

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Brevity Constraints Reverse Performance Hierarchies in Language Models
  2. 2025Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models
  3. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  4. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
  5. 2024Revisiting the Superficial Alignment Hypothesis
  6. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
  7. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale
  8. 2026Prescriptive Scaling Reveals the Evolution of Language Model Capabilities