mlsanityData to 14 Oct 2026

Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs

arXiv 2404.17120 ↗First posted 26 Apr 2024cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit excellent ability to understand human languages, but do they also understand their own language that appears gibberish to us? In this work we delve into this question, aiming to uncover the mechanisms underlying such behavior in LLMs. We employ the Greedy Coordinate Gradient optimizer to craft prompts that compel LLMs to generate coherent responses from seemingly nonsensical inputs. We call these inputs LM Babel and this work systematically studies the behavior of LLMs manipulated by these prompts. We find that the manipulation efficiency depends on the target text's length and perplexity, with the Babel prompts often located in lower loss minima compared to natural prompts. We further examine the structure of the Babel prompts and evaluate their robustness. Notably, we find that guiding the model to generate harmful texts is not more difficult than into generating benign texts, suggesting lack of alignment for out-of-distribution prompts.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

49 of 58 references.

References · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
37 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2022In-context Learning and Induction Headscited by 964
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  10. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  13. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  14. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  15. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  16. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  17. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  18. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  19. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  20. 2018Generating Natural Language Adversarial Examplescited by 424
  21. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  22. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  23. 2023Are aligned neural networks adversarially aligned?cited by 315
  24. 2016Rationalizing Neural Predictionscited by 281
  25. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  26. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  27. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  28. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  29. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  30. 2018Physical Adversarial Examples for Object Detectorscited by 163
  31. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  32. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 134
  33. 2021LowKey: Leveraging Adversarial Attacks to Protect Social Media Users from Facial Recognitioncited by 94
  34. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88
  35. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58
  36. 2022Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methodscited by 13
  37. 2021Where do Models go Wrong? Parameter-Space Saliency Maps for Explainabilitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 2.

11 papers.

  1. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  2. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 37
  3. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  4. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  5. 2024Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Modelscited by 4
  6. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  7. 2025Has My System Prompt Been Used? Large Language Model Prompt Membership Inferencecited by 3
  8. 2025Demystifying optimized prompts in language modelscited by 1
  9. 2024Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probingcited by 0
  10. 2025Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Modelscited by 0
  11. 2026BayesPrompt: human readable prompts that make sensecited by 0