mlsanityData to 14 Oct 2026

Language Models Are Implicitly Continuous

arXiv 2504.03933 ↗First posted 4 Apr 2025cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Language is typically modelled with discrete sequences. However, the most successful approaches to language modelling, namely neural networks, are continuous and smooth function approximators. In this work, we show that Transformer-based language models implicitly learn to represent sentences as continuous-time functions defined over a continuous input space. This phenomenon occurs in most state-of-the-art Large Language Models (LLMs), including Llama2, Llama3, Phi3, Gemma, Gemma2, and Mistral, and suggests that LLMs reason about language in ways that fundamentally differ from humans. Our work formally extends Transformers to capture the nuances of time and space continuity in both input and output space. Our results challenge the traditional interpretation of how LLMs understand language, with several linguistic and engineering implications.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

35 of 56 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2018Neural Ordinary Differential Equationscited by 3,779
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2023Mistral 7Bcited by 3,164
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
23 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  4. 2015Generating Sentences from a Continuous Spacecited by 612
  5. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  6. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  7. 2023Language Models Represent Space and Timecited by 316
  8. 2023Function Vectors in Large Language Modelscited by 308
  9. 2023Likelihood-Based Diffusion Language Modelscited by 151
  10. 2020Neural Rough Differential Equations for Long Time Seriescited by 143
  11. 2022Latent Diffusion for Language Generationcited by 115
  12. 2022Linear Adversarial Concept Erasurecited by 101
  13. 2014Word Representations via Gaussian Embeddingcited by 96
  14. 2024ContiFormer: Continuous-Time Transformer for Irregular Time Series Modelingcited by 63
  15. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 58
  16. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  17. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  18. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 27
  19. 2016Continuous multilinguality with language vectorscited by 20
  20. 2024Diffusion Guided Language Modelingcited by 15
  21. 2020N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equationscited by 9
  22. 2023Continuous Spatiotemporal Transformerscited by 7
  23. 2023Formal Aspects of Language Modelingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Text Generation Beyond Discrete Token Samplingcited by 21
  2. 2025VOX-KRIKRI: Unifying Speech and Language through Continuous Fusioncited by 0
  3. 2026Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorizationcited by 0
  4. 2026Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionalitycited by 0