Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
Abstract (arXiv)
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment. This problem is comprised of arithmetic questions posed in natural language, with an arbitrary adversarial string inserted before the question is complete. Even in the simple setting of 1-digit addition problems, it is easy to find adversarial prompts that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and even to steer models to a particular wrong answer. We additionally provide a simple algorithm for finding successful attacks by querying those same models, which we name "prompt inversion rejection sampling" (PIRS). We finally show that models can be partially hardened against these attacks via reinforcement learning and via agentic constitutional loops. However, we were not able to make a language model fully robust against adversarial arithmetic attacks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 22 references that are in the map. First posted on arXiv 8 Nov 2023.
Builds on
16 of 22 references.
Closest ancestors · 2
Same direction, newest first
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
Other references · 14
- 2013Intriguing properties of neural networkscited by 5,114
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2022Red Teaming Language Models with Language Modelscited by 983
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
- 2022UL2: Unifying Language Learning Paradigmscited by 292
- 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
- 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
2 more
- 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
- 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists
- 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
- 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective
- 2025MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
- 2025Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks
- 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
- 2025Adversarial Reasoning at Jailbreaking Time
- 2024Adversarial Attacks on Large Language Models Using Regularized Relaxation