# Unnatural language processing: How do language models handle machine-generated prompts?

arXiv: https://arxiv.org/abs/2310.15829 · HTML: https://mlsanity.com/paper/2310.15829/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts](https://mlsanity.com/paper/2112.08348/) (2021)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [Evaluating the Robustness of Discrete Prompts](https://mlsanity.com/paper/2302.05619/) (2023)
- [Can discrete information extraction prompts generalize across language models?](https://mlsanity.com/paper/2302.09865/) (2023)

## Built on by (most cited first, 8 of 8)

- [Repetitions are not all alike: distinct mechanisms sustain repetition in language models](https://mlsanity.com/paper/2504.01100/) (2025)
- [An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing](https://mlsanity.com/paper/2403.16854/) (2024)
- [Evil twins are not that evil: Qualitative insights into machine-generated prompts](https://mlsanity.com/paper/2412.08127/) (2024)
- [Unnatural Languages Are Not Bugs but Features for LLMs](https://mlsanity.com/paper/2503.01926/) (2025)
- [Demystifying optimized prompts in language models](https://mlsanity.com/paper/2505.02273/) (2025)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Talk Freely, Execute Strictly: Schema-Gated Agentic AI for Flexible and Reproducible Scientific Workflows](https://mlsanity.com/paper/2603.06394/) (2026)
- [BayesPrompt: human readable prompts that make sense](https://mlsanity.com/paper/2608.17866/) (2026)

## Abstract (arXiv)

Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are routinely outperformed by automatically generated token sequences with no apparent meaning or syntactic structure, including sequences of vectors from a model's embedding space. We use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. We study the behavior of models of different sizes in multiple semantic tasks in response to both continuous and discrete machine-generated prompts, and compare it to the behavior in response to human-generated natural-language prompts. Even when producing a similar output, machine-generated and human prompts trigger different response patterns through the network processing pathways, including different perplexities, different attention and output entropy distributions, and different unit activation profiles. We provide preliminary insight into the nature of the units activated by different prompt types, suggesting that only natural language prompts recruit a genuinely linguistic circuit.