# Language Models Are Implicitly Continuous

arXiv: https://arxiv.org/abs/2504.03933 · HTML: https://mlsanity.com/paper/2504.03933/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Likelihood-Based Diffusion Language Models](https://mlsanity.com/paper/2305.18619/) (2023)
- [Neural Rough Differential Equations for Long Time Series](https://mlsanity.com/paper/2009.08295/) (2020)
- [Latent Diffusion for Language Generation](https://mlsanity.com/paper/2212.09462/) (2022)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling](https://mlsanity.com/paper/2402.10635/) (2024)
- [PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model](https://mlsanity.com/paper/2306.02531/) (2023)
- [Graph-enhanced Large Language Models in Asynchronous Plan Reasoning](https://mlsanity.com/paper/2402.02805/) (2024)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)

## Built on by (most cited first, 4 of 4)

- [Text Generation Beyond Discrete Token Sampling](https://mlsanity.com/paper/2505.14827/) (2025)
- [VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion](https://mlsanity.com/paper/2509.15667/) (2025)
- [Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorization](https://mlsanity.com/paper/2609.32199/) (2026)
- [Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality](https://mlsanity.com/paper/2609.34936/) (2026)

## Abstract (arXiv)

Language is typically modelled with discrete sequences. However, the most successful approaches to language modelling, namely neural networks, are continuous and smooth function approximators. In this work, we show that Transformer-based language models implicitly learn to represent sentences as continuous-time functions defined over a continuous input space. This phenomenon occurs in most state-of-the-art Large Language Models (LLMs), including Llama2, Llama3, Phi3, Gemma, Gemma2, and Mistral, and suggests that LLMs reason about language in ways that fundamentally differ from humans. Our work formally extends Transformers to capture the nuances of time and space continuity in both input and output space. Our results challenge the traditional interpretation of how LLMs understand language, with several linguistic and engineering implications.