# Jailbreaking Large Language Models with Symbolic Mathematics

arXiv: https://arxiv.org/abs/2409.11445 · HTML: https://mlsanity.com/paper/2409.11445/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 21 of 21)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs](https://mlsanity.com/paper/2402.11753/) (2024)
- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction](https://mlsanity.com/paper/2402.18104/) (2024)
- [Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks](https://mlsanity.com/paper/2305.14965/) (2023)
- [Large Language Models Are Neurosymbolic Reasoners](https://mlsanity.com/paper/2401.09334/) (2024)

## Built on by (most cited first, 16 of 16)

- [Should LLM Safety Be More Than Refusing Harmful Instructions?](https://mlsanity.com/paper/2506.02442/) (2025)
- [Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts](https://mlsanity.com/paper/2503.17953/) (2025)
- [Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection](https://mlsanity.com/paper/2510.13893/) (2025)
- [Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysis](https://mlsanity.com/paper/2605.03441/) (2026)
- [Don't Command, Cultivate: An Exploratory Study of System-2 Alignment](https://mlsanity.com/paper/2411.17075/) (2024)
- [Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06921/) (2025)
- [Hallucinating AI Hijacking Attack: Large Language Models and Malicious Code Recommenders](https://mlsanity.com/paper/2410.06462/) (2024)
- [Semantic Compression of LLM Instructions via Symbolic Metalanguages](https://mlsanity.com/paper/2601.07354/) (2026)
- [AJF: Adaptive Jailbreak Framework Based on the Comprehension Ability of Black-Box Large Language Models](https://mlsanity.com/paper/2505.23404/) (2025)
- [Can MLLMs Detect Phishing? A Comprehensive Security Benchmark Suite Focusing on Dynamic Threats and Multimodal Evaluation in Academic Environments](https://mlsanity.com/paper/2511.15165/) (2025)
- [Self-Guided Adaptive Safety Alignment: Synthesizing and Internalizing Guidelines in Reasoning Models](https://mlsanity.com/paper/2511.21214/) (2025)
- [SRTJ: Self-Evolving Rule-Driven Training-Free LLM Jailbreaking](https://mlsanity.com/paper/2605.00974/) (2026)
- [Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks](https://mlsanity.com/paper/2607.26574/) (2026)
- [Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks](https://mlsanity.com/paper/2608.01043/) (2026)
- [EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models](https://mlsanity.com/paper/2609.26175/) (2026)
- [CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion](https://mlsanity.com/paper/2609.39902/) (2026)

## Abstract (arXiv)

Recent advancements in AI safety have led to increased efforts in training and red-teaming large language models (LLMs) to mitigate unsafe content generation. However, these safety mechanisms may not be comprehensive, leaving potential vulnerabilities unexplored. This paper introduces MathPrompt, a novel jailbreaking technique that exploits LLMs' advanced capabilities in symbolic mathematics to bypass their safety mechanisms. By encoding harmful natural language prompts into mathematical problems, we demonstrate a critical vulnerability in current AI safety measures. Our experiments across 13 state-of-the-art LLMs reveal an average attack success rate of 73.6%, highlighting the inability of existing safety training mechanisms to generalize to mathematically encoded inputs. Analysis of embedding vectors shows a substantial semantic shift between original and encoded prompts, helping explain the attack's success. This work emphasizes the importance of a holistic approach to AI safety, calling for expanded red-teaming efforts to develop robust safeguards across all potential input types and their associated risks.