# Language Modeling with Hyperspherical Flows

arXiv: https://arxiv.org/abs/2605.11125 · HTML: https://mlsanity.com/paper/2605.11125/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 32 of its 65 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Scaling up Masked Diffusion Models on Text](https://mlsanity.com/paper/2410.18514/) (2024)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Riemannian Continuous Normalizing Flows](https://mlsanity.com/paper/2006.10605/) (2020)
- [Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking](https://mlsanity.com/paper/2505.24857/) (2025)
- [Unlocking Guidance for Discrete State-Space Diffusion and Flow Models](https://mlsanity.com/paper/2406.01572/) (2024)
- [Neural Manifold Ordinary Differential Equations](https://mlsanity.com/paper/2006.10254/) (2020)
- [Spherical Latent Spaces for Stable Variational Autoencoders](https://mlsanity.com/paper/1808.10805/) (2018)
- [Theoretical Benefit and Limitation of Diffusion Language Model](https://mlsanity.com/paper/2502.09622/) (2025)
- [Fine-Tuning Masked Diffusion for Provable Self-Correction](https://mlsanity.com/paper/2510.01384/) (2025)
- [Scaling Behavior of Discrete Diffusion Language Models](https://mlsanity.com/paper/2512.10858/) (2025)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Arrows of Time for Large Language Models](https://mlsanity.com/paper/2401.17505/) (2024)

## Built on by (most cited first, 22 of 22)

- [Self-conditioned Flow Map Language Models via Fixed-point Flows](https://mlsanity.com/paper/2607.00714/) (2026)
- [Posterior Refinement: Fast Language Generation via Any-Order Flow Maps](https://mlsanity.com/paper/2606.24773/) (2026)
- [DiffusionGemma Technical Report](https://mlsanity.com/paper/2608.00146/) (2026)
- [Thinking with Looped Flows](https://mlsanity.com/paper/2609.11801/) (2026)
- [Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors](https://mlsanity.com/paper/2606.25971/) (2026)
- [Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning](https://mlsanity.com/paper/2609.15903/) (2026)
- [Simplex Relaxation for Discrete Diffusion](https://mlsanity.com/paper/2608.10615/) (2026)
- [Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One](https://mlsanity.com/paper/2609.04531/) (2026)
- [From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models](https://mlsanity.com/paper/2607.26504/) (2026)
- [ELF-REG: Scaling Continuous Diffusion Language Models to Reasoning Tasks](https://mlsanity.com/paper/2609.29102/) (2026)
- [Simplex Diffusion Models](https://mlsanity.com/paper/2609.35553/) (2026)
- [Reasoning with Continuous Latent Diffusion](https://mlsanity.com/paper/2609.35694/) (2026)
- [Alpha Diffusion Language Models: Factorization Alone Is Not the Problem](https://mlsanity.com/paper/2609.38066/) (2026)
- [Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Maps](https://mlsanity.com/paper/2610.00497/) (2026)
- [Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.00894/) (2026)
- [Jumping up and down: Denoiser diffusion models for discrete ordinal data](https://mlsanity.com/paper/2610.02754/) (2026)
- [Representation-Space MMD for Diffusion Language Models](https://mlsanity.com/paper/2610.06648/) (2026)
- [Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling](https://mlsanity.com/paper/2610.08738/) (2026)
- [Noise Your Prompt: Noising Conditioning Tokens in Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.09145/) (2026)
- [Unrolled Flow Models for Reasoning](https://mlsanity.com/paper/2610.09759/) (2026)
- [On the Design Space of Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.13834/) (2026)
- [Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation](https://mlsanity.com/paper/2610.14162/) (2026)

## Abstract (arXiv)

Discrete Diffusion Language Models progressed rapidly as an alternative to autoregressive (AR) models, motivated by their parallel generation abilities. However, for tractability, discrete diffusion models sample from a factorized distribution, which is less expressive than AR. Recent Flow Language Models (FLMs) apply continuous flows to language, transporting noise to data with a deterministic ODE that avoids factorized sampling. FLMs operate on one-hot vectors whose dimension scales with the vocabulary size, making FLMs costly to train. Moreover, since all distinct one-hot embeddings are equidistant in ℓ2, adding Gaussian noise does not have a clear semantic interpretation (unlike images, where Gaussian noise progressively degrades structure). We introduce mathbbS-FLM, a latent FLM in the hypersphere. mathbbS-FLM generates sequences by rotating vectors in mathbbSd-1 along a velocity field learned with cross-entropy, avoiding the overhead of materializing one-hot vectors. Previous FLMs match AR in Generative Perplexity (Gen. PPL), but samples with high likelihood are not necessarily correct in verifiable domains such as math and code. mathbbS-FLM substantially improves continuous flow language models on large-vocabulary reasoning and closes the gap to masked diffusion under standard-temperature sampling (T=1), while a gap remains under optimized low-temperature (T=0.1) decoding.