# Optimisation Is Not What You Need

arXiv: https://arxiv.org/abs/2507.03045 · HTML: https://mlsanity.com/paper/2507.03045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 64 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal](https://mlsanity.com/paper/2403.01244/) (2024)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting](https://mlsanity.com/paper/1812.02464/) (2018)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting](https://mlsanity.com/paper/2404.17245/) (2024)
- [MuseumMaker: Continual Style Customization without Catastrophic Forgetting](https://mlsanity.com/paper/2404.16612/) (2024)
- [Studying Catastrophic Forgetting in Neural Ranking Models](https://mlsanity.com/paper/2101.06984/) (2021)
- [MalCL: Leveraging GAN-Based Generative Replay to Combat Catastrophic Forgetting in Malware Classification](https://mlsanity.com/paper/2501.01110/) (2025)
- [Catastrophic forgetting: still a problem for DNNs](https://mlsanity.com/paper/1905.08077/) (2019)
- [Preventing Catastrophic Forgetting through Memory Networks in Continuous Detection](https://mlsanity.com/paper/2403.14797/) (2024)
- [On robustness of generative representations against catastrophic forgetting](https://mlsanity.com/paper/2109.01844/) (2021)
- [From Manifestations to Cognitive Architectures: a Scalable Framework](https://mlsanity.com/paper/2406.09823/) (2024)
- [Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation](https://mlsanity.com/paper/2010.09403/) (2020)
- [Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System](https://mlsanity.com/paper/2405.10992/) (2024)
- [Make Domain Shift a Catastrophic Forgetting Alleviator in Class-Incremental Learning](https://mlsanity.com/paper/2501.00237/) (2024)
- [Unsupervised Cognition](https://mlsanity.com/paper/2409.18624/) (2024)
- [Beating Transformers using Synthetic Cognition](https://mlsanity.com/paper/2504.07619/) (2025)

## Abstract (arXiv)

The Artificial Intelligence field has focused on developing optimisation methods to solve multiple problems, specifically problems that we thought to be only solvable through cognition. The obtained results have been outstanding, being able to even surpass the Turing Test. However, we have found that these optimisation methods share some fundamental flaws that impede them to become a true artificial cognition. Specifically, the field have identified catastrophic forgetting as a fundamental problem to develop such cognition. This paper formally proves that this problem is inherent to optimisation methods, and as such it will always limit approaches that try to solve the Artificial General Intelligence problem as an optimisation problem. Additionally, it addresses the problem of overfitting and discuss about other smaller problems that optimisation methods pose. Finally, it empirically shows how world-modelling methods avoid suffering from either problem. As a conclusion, the field of Artificial Intelligence needs to look outside the machine learning field to find methods capable of developing an artificial cognition.