# The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance

arXiv: https://arxiv.org/abs/2401.03729 · HTML: https://mlsanity.com/paper/2401.03729/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 19 of 19)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://mlsanity.com/paper/2309.07462/) (2023)
- [Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4](https://mlsanity.com/paper/2312.16171/) (2023)
- [iSarcasm: A Dataset of Intended Sarcasm](https://mlsanity.com/paper/1911.03123/) (2019)
- [Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks](https://mlsanity.com/paper/2311.09743/) (2023)
- [Making Large Language Models Better Data Creators](https://mlsanity.com/paper/2310.20111/) (2023)
- [ColBERT: Using BERT Sentence Embedding in Parallel Neural Networks for Computational Humor](https://mlsanity.com/paper/2004.12765/) (2020)
- [Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness](https://mlsanity.com/paper/2302.13793/) (2023)

## Built on by (most cited first, 30 of 100)

- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation](https://mlsanity.com/paper/2402.12649/) (2024)
- [On the Worst Prompt Performance of Large Language Models](https://mlsanity.com/paper/2406.10248/) (2024)
- [Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra](https://mlsanity.com/paper/2404.03647/) (2024)
- [Your Large Language Models Are Leaving Fingerprints](https://mlsanity.com/paper/2405.14057/) (2024)
- [AI Alignment at Your Discretion](https://mlsanity.com/paper/2502.10441/) (2025)
- [GhostWriter: Augmenting Collaborative Human-AI Writing Experiences Through Personalization and Agency](https://mlsanity.com/paper/2402.08855/) (2024)
- [Harmful Speech Detection by Language Models Exhibits Gender-Queer Dialect Bias](https://mlsanity.com/paper/2406.00020/) (2024)
- [Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization](https://mlsanity.com/paper/2502.04295/) (2025)
- [When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMs](https://mlsanity.com/paper/2510.19107/) (2025)
- [Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs](https://mlsanity.com/paper/2503.05371/) (2025)
- [Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems](https://mlsanity.com/paper/2503.06745/) (2025)
- [Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering](https://mlsanity.com/paper/2411.12395/) (2024)
- [Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective](https://mlsanity.com/paper/2504.03255/) (2025)
- [MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing](https://mlsanity.com/paper/2405.11215/) (2024)
- [COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities](https://mlsanity.com/paper/2406.12074/) (2024)
- [CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts](https://mlsanity.com/paper/2410.15453/) (2024)
- [Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring](https://mlsanity.com/paper/2502.07087/) (2025)
- [From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task](https://mlsanity.com/paper/2502.16690/) (2025)
- [Culture is Everywhere: A Call for Intentionally Cultural Evaluation](https://mlsanity.com/paper/2509.01301/) (2025)
- [An Empirical Study of In-context Learning in LLMs for Machine Translation](https://mlsanity.com/paper/2401.12097/) (2024)
- [Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2408.06663/) (2024)
- [PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation](https://mlsanity.com/paper/2410.16547/) (2024)
- [Towards More Realistic Extraction Attacks: An Adversarial Perspective](https://mlsanity.com/paper/2407.02596/) (2024)
- [Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing](https://mlsanity.com/paper/2505.21701/) (2025)
- [Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges](https://mlsanity.com/paper/2605.06161/) (2026)
- [Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits](https://mlsanity.com/paper/2605.18890/) (2026)
- [Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction](https://mlsanity.com/paper/2402.17969/) (2024)
- [Bayesian Statistical Modeling with Predictors from LLMs](https://mlsanity.com/paper/2406.09012/) (2024)
- [Learning from Contrastive Prompts: Automated Optimization and Adaptation](https://mlsanity.com/paper/2409.15199/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or ``prompting,'' practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work, we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly used jailbreaks can have cataclysmic effects on the data labeled by LLMs.