# Representations Matter: Embedding Modes of Large Language Models using Dynamic Mode Decomposition

arXiv: https://arxiv.org/abs/2309.01245 · HTML: https://mlsanity.com/paper/2309.01245/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 10 references that are in the map.
- First posted on arXiv 3 Sep 2023.

## Builds on (most cited first, 5 of 5)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Visualization and Selection of Dynamic Mode Decomposition Components for Unsteady Flow](https://mlsanity.com/paper/2012.09633/) (2020)

## Built on by (most cited first, 1 of 1)

- [Interpreting Reinforcement Learning Model Behavior via Koopman with Control](https://mlsanity.com/paper/2603.19968/) (2026)

## Abstract (arXiv)

Existing large language models (LLMs) are known for generating "hallucinated" content, namely a fabricated text of plausibly looking, yet unfounded, facts. To identify when these hallucination scenarios occur, we examine the properties of the generated text in the embedding space. Specifically, we draw inspiration from the dynamic mode decomposition (DMD) tool in analyzing the pattern evolution of text embeddings across sentences. We empirically demonstrate how the spectrum of sentence embeddings over paragraphs is constantly low-rank for the generated text, unlike that of the ground-truth text. Importantly, we find that evaluation cases having LLM hallucinations correspond to ground-truth embedding patterns with a higher number of modes being poorly approximated by the few modes associated with LLM embedding patterns. In analogy to near-field electromagnetic evanescent waves, the embedding DMD eigenmodes of the generated text with hallucinations vanishes quickly across sentences as opposed to those of the ground-truth text. This suggests that the hallucinations result from both the generation techniques and the underlying representation.