mlsanityData to 7 Oct 2026

The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance

arXiv 2401.03729 ↗First posted 8 Jan 2024cs.CL cs.AIcited by 97

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or ``prompting,'' practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work, we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly used jailbreaks can have cataclysmic effects on the data labeled by LLMs.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 19 of its 30 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

19 of 30 references.

References · 19

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  10. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  11. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  12. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
7 more
  1. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  2. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  3. 2019iSarcasm: A Dataset of Intended Sarcasmcited by 39
  4. 2023Making Large Language Models Better Data Creatorscited by 28
  5. 2023Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Taskscited by 28
  6. 2020ColBERT: Using BERT Sentence Embedding in Parallel Neural Networks for Computational Humorcited by 18
  7. 2023Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnesscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 38; 2026: 31.

97 papers.

Sort:

By area:

  1. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  2. 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
  3. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 36
  4. 2024On the Worst Prompt Performance of Large Language Modelscited by 36
  5. 2024Your Large Language Models Are Leaving Fingerprintscited by 31
  6. 2025AI Alignment at Your Discretioncited by 20
  7. 2024GhostWriter: Augmenting Collaborative Human-AI Writing Experiences Through Personalization and Agencycited by 18
  8. 2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 15
  9. 2024Harmful Speech Detection by Language Models Exhibits Gender-Queer Dialect Biascited by 14
  10. 2025Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systemscited by 14
  11. 2025When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMscited by 14
  12. 2024Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answeringcited by 13
38 more
  1. 2025Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMscited by 13
  2. 2024MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencingcited by 12
  3. 2024COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communitiescited by 12
  4. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  5. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  6. 2025Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloringcited by 10
  7. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  8. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  9. 2024An Empirical Study of In-context Learning in LLMs for Machine Translationcited by 9
  10. 2024Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Modelscited by 9
  11. 2024PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creationcited by 9
  12. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  13. 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
  14. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7
  15. 2024Learning from Contrastive Prompts: Automated Optimization and Adaptationcited by 7
  16. 2024A Comprehensive Evaluation of Large Language Models on Mental Illnessescited by 7
  17. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 7
  18. 2024Bayesian Statistical Modeling with Predictors from LLMscited by 6
  19. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  20. 2026Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Auditscited by 6
  21. 2024Engagement-Driven Content Generation with Large Language Modelscited by 5
  22. 2025Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustnesscited by 5
  23. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfittingcited by 4
  24. 2025Identifying and Mitigating the Influence of the Prior Distribution in Large Language Modelscited by 4
  25. 2024A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classificationcited by 3
  26. 2025Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspectivecited by 3
  27. 2025Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMscited by 3
  28. 2025Beyond "Hallucinations": A Framework for Stable Human-AI Reasoningcited by 3
  29. 2025MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attributioncited by 3
  30. 2026Within-Model vs Between-Prompt Variability in Large Language Models for Creative Taskscited by 3
  31. 2026Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecificationcited by 3
  32. 2024Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communitiescited by 2
  33. 2025PAFT: Prompt-Agnostic Fine-Tuningcited by 2
  34. 2025Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorizationcited by 2
  35. 2025Efficient Prompt Optimisation for Legal Text Classification with Proxy Prompt Evaluatorcited by 2
  36. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  37. 2026InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersonscited by 2
  38. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2

50 most cited of 97 All 97 (JSON)