mlsanityData to 7 Oct 2026

One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion

arXiv 2609.33698 ↗First posted 27 Sep 2026cs.AIcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be difficult to model with diffusion and decode reliably into tokens, which limits generation quality after compression. To address this problem, we introduce JPEG-DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Language Model), which jointly trains a compressor, a flow matching model and a decoding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens. JPEG-DLM achieves the lowest mean Gen-PPL and highest throughput among recent diffusion and flow models on LM1B and OWT. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3 times ELF's throughput. These results suggest that jointly learning compressed embeddings offers a promising path toward efficient diffusion language modeling. Code will be released soon.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 46 of its 51 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

46 of 51 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling3 Aug
  2. 2026Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models28 May
  3. 2026ELF: Embedded Language Flows11 May
  4. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Space8 May
  5. 2026Continuous Latent Diffusion Language Model7 May
  6. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling13 Apr
  7. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think3 Mar
  8. 2026Flow Map Language Models: One-step Language Modeling via Continuous Denoising18 Feb
  9. 2025Dream 7B: Diffusion Large Language Models21 Aug
  10. 2025Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling26 Jun
  11. 2025The Diffusion Duality12 Jun
  12. 2025Large Language Diffusion Models14 Feb
9 more
  1. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  3. 2024TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings29 Feb
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  5. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model5 Jun
  6. 2023Likelihood-Based Diffusion Language Models30 May
  7. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  8. 2022Latent Diffusion for Language Generation19 Dec
  9. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
13 more
  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  5. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  6. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  8. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  9. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  10. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 35
  11. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 12
  12. 2026Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metricscited by 6
  13. 2026Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-conditioned Embedding Diffusion for Text Generation
  2. 2026Scaling and Distilling Text Embeddings for Better Diffusibility
  3. 2026Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
  4. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
  5. 2026Denoising Blocks, Not Tokens: Efficient Compressed Continuous Diffusion with Branching Token Realization
  6. 2026Representation-Space MMD for Diffusion Language Models
  7. 2026x-Prediction Flow: Efficient Continuous Decoding for Masked Diffusion Language Models
  8. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching