mlsanityData to 14 Oct 2026

Demystifying optimized prompts in language models

arXiv 2505.02273 ↗First posted 4 May 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Modern language models (LMs) are not robust to out-of-distribution inputs. Machine generated (``optimized'') prompts can be used to modulate LM outputs and induce specific behaviors while appearing completely uninterpretable. In this work, we investigate the composition of optimized prompts, as well as the mechanisms by which LMs parse and build predictions from optimized prompts. We find that optimized prompts primarily consist of punctuation and noun tokens which are more rare in the training data. Internally, optimized prompts are clearly distinguishable from natural language counterparts based on sparse subsets of the model's activations. Across various families of instruction-tuned models, optimized prompts follow a similar path in how their representations form through the network.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

29 of 52 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2016Layer Normalizationcited by 5,669
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,353
17 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  7. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  8. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  9. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  10. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  11. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  12. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 29
  13. 2023Evaluating the Robustness of Discrete Promptscited by 18
  14. 2024Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputscited by 11
  15. 2023Can discrete information extraction prompts generalize across language models?cited by 8
  16. 2023Unnatural language processing: How do language models handle machine-generated prompts?cited by 8
  17. 2024Evil twins are not that evil: Qualitative insights into machine-generated promptscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026BayesPrompt: human readable prompts that make sensecited by 0