# Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling

arXiv: https://arxiv.org/abs/2506.21170 · HTML: https://mlsanity.com/paper/2506.21170/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Remasking Discrete Diffusion Models with Inference-Time Scaling](https://mlsanity.com/paper/2503.00307/) (2025)
- [Simple Guidance Mechanisms for Discrete Diffusion Models](https://mlsanity.com/paper/2412.10193/) (2024)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning](https://mlsanity.com/paper/2410.14157/) (2024)
- [AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation](https://mlsanity.com/paper/2305.09515/) (2023)
- [Latent Diffusion for Language Generation](https://mlsanity.com/paper/2212.09462/) (2022)
- [EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling](https://mlsanity.com/paper/2502.09509/) (2025)
- [Improving the Diffusability of Autoencoders](https://mlsanity.com/paper/2502.14831/) (2025)

## Built on by (most cited first, 25 of 25)

- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [CANDI: Hybrid Discrete-Continuous Diffusion Models](https://mlsanity.com/paper/2510.22510/) (2025)
- [LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning](https://mlsanity.com/paper/2510.04573/) (2025)
- [Continuous Diffusion Scales Competitively with Discrete Diffusion for Language](https://mlsanity.com/paper/2605.18530/) (2026)
- [Continuous Latent Diffusion Language Model](https://mlsanity.com/paper/2605.06548/) (2026)
- [How to Train Your Latent Diffusion Language Model Jointly With the Latent Space](https://mlsanity.com/paper/2605.07933/) (2026)
- [SimFlow: Simplified and End-to-End Training of Latent Normalizing Flows](https://mlsanity.com/paper/2512.04084/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [Guided Star-Shaped Masked Diffusion](https://mlsanity.com/paper/2510.08369/) (2025)
- [Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?](https://mlsanity.com/paper/2604.02028/) (2026)
- [TextLDM: Language Modeling with Continuous Latent Diffusion](https://mlsanity.com/paper/2605.07748/) (2026)
- [AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling](https://mlsanity.com/paper/2608.02602/) (2026)
- [DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling](https://mlsanity.com/paper/2605.23605/) (2026)
- [Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster](https://mlsanity.com/paper/2605.18204/) (2026)
- [ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings](https://mlsanity.com/paper/2608.23551/) (2026)
- [CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language](https://mlsanity.com/paper/2603.20210/) (2026)
- [Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models](https://mlsanity.com/paper/2604.02340/) (2026)
- [When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2605.15557/) (2026)
- [From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models](https://mlsanity.com/paper/2607.26504/) (2026)
- [Latent-Augmented Discrete Diffusion Models](https://mlsanity.com/paper/2510.18114/) (2025)
- [Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models](https://mlsanity.com/paper/2609.15338/) (2026)
- [Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders](https://mlsanity.com/paper/2609.27248/) (2026)
- [One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion](https://mlsanity.com/paper/2609.33698/) (2026)
- [Scaling and Distilling Text Embeddings for Better Diffusibility](https://mlsanity.com/paper/2610.01016/) (2026)
- [Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation](https://mlsanity.com/paper/2610.12898/) (2026)

## Abstract (arXiv)

Autoregressive language models dominate modern text generation, yet their sequential nature introduces fundamental limitations: decoding is slow, and maintaining global coherence remains challenging. Diffusion models offer a promising alternative by enabling parallel generation and flexible control; however, their application to text generation is hindered by the high dimensionality of token-level representations. We introduce Cosmos, a novel approach to text generation that operates entirely in a compressed, smooth latent space tailored specifically for diffusion. This space is learned using an autoencoder trained simultaneously for token-level reconstruction and alignment with frozen activations from a pretrained language encoder, providing robust semantic grounding and enabling effective perturbation-based augmentations. Empirically, we demonstrate that text representations can be compressed by 8× while maintaining generation quality comparable to token-level diffusion models. Furthermore, increasing the latent sequence length allows Cosmos to surpass both diffusion-based and autoregressive baselines. We evaluate Cosmos on four diverse generative tasks including story generation, question generation, summarization, and detoxification and compare it with various generative paradigms. Cosmos achieves comparable or superior generation quality while offering more than 2× faster inference. Code is released at hrefhttps://github.com/MeshchaninovViacheslav/cosmosGitHub