# Fluent dreaming for language models

arXiv: https://arxiv.org/abs/2402.01702 · HTML: https://mlsanity.com/paper/2402.01702/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Automatically Auditing Large Language Models via Discrete Optimization](https://mlsanity.com/paper/2303.04381/) (2023)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Gradient-Based Constrained Sampling from Language Models](https://mlsanity.com/paper/2205.12558/) (2022)
- [Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?](https://mlsanity.com/paper/2212.10539/) (2022)
- [Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework](https://mlsanity.com/paper/2110.15317/) (2021)
- [Interpretable Textual Neuron Representations for NLP](https://mlsanity.com/paper/1809.07291/) (2018)

## Built on by (most cited first, 5 of 5)

- [RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?](https://mlsanity.com/paper/2506.14261/) (2025)
- [FLRT: Fluent Student-Teacher Redteaming](https://mlsanity.com/paper/2407.17447/) (2024)
- [Patterns and Mechanisms of Contrastive Activation Engineering](https://mlsanity.com/paper/2505.03189/) (2025)
- [ContextBench: Modifying Contexts for Targeted Latent Activation](https://mlsanity.com/paper/2506.15735/) (2025)
- [ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization](https://mlsanity.com/paper/2602.17867/) (2026)

## Abstract (arXiv)

Feature visualization, also known as "dreaming", offers insights into vision models by optimizing the inputs to maximize a neuron's activation or other internal component. However, dreaming has not been successfully applied to language models because the input space is discrete. We extend Greedy Coordinate Gradient, a method from the language model adversarial attack literature, to design the Evolutionary Prompt Optimization (EPO) algorithm. EPO optimizes the input prompt to simultaneously maximize the Pareto frontier between a chosen internal feature and prompt fluency, enabling fluent dreaming for language models. We demonstrate dreaming with neurons, output logits and arbitrary directions in activation space. We measure the fluency of the resulting prompts and compare language model dreaming with max-activating dataset examples. Critically, fluent dreaming allows automatically exploring the behavior of model internals in reaction to mildly out-of-distribution prompts. Code for running EPO is available at https://github.com/Confirm-Solutions/dreamy. A companion page demonstrating code usage is at https://confirmlabs.org/posts/dreamy.html