Jekyll-and-Hyde Tipping Point in an AI's Behavior
Not on the map. Why?
Abstract (arXiv)
Trust in AI is undermined by the fact that there is no science that predicts -- or that can explain to the public -- when an LLM's output (e.g. ChatGPT) is likely to tip mid-response to become wrong, misleading, irrelevant or dangerous. With deaths and trauma already being blamed on LLMs, this uncertainty is even pushing people to treat their 'pet' LLM more politely to 'dissuade' it (or its future Artificial General Intelligence offspring) from suddenly turning on them. Here we address this acute need by deriving from first principles an exact formula for when a Jekyll-and-Hyde tipping point occurs at LLMs' most basic level. Requiring only secondary school mathematics, it shows the cause to be the AI's attention spreading so thin it suddenly snaps. This exact formula provides quantitative predictions for how the tipping-point can be delayed or prevented by changing the prompt and the AI's training. Tailored generalizations will provide policymakers and the public with a firm platform for discussing any of AI's broader uses and risks, e.g. as a personal counselor, medical advisor, decision-maker for when to use force in a conflict situation. It also meets the need for clear and transparent answers to questions like ''should I be polite to my LLM?''
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 12 of its 25 references that are in the map. First posted on arXiv 29 Apr 2025.
Builds on
12 of 25 references.
References · 12
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2019Are Sixteen Heads Really Better than One?cited by 816
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
- 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
- 2019Attention in Natural Language Processingcited by 101
- 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
- 2025Capturing AI's Attention: Physics of Repetition, Hallucination, Bias and Beyondcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.