mlsanityData to 14 Oct 2026

TESS 2: A Large-Scale Generalist Diffusion Language Model

arXiv 2502.13917 ↗First posted 19 Feb 2025cs.CLcited by 30

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We introduce TESS 2, a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models, as well as matches and sometimes exceeds strong autoregressive (AR) models. We train TESS 2 by first adapting a strong AR model via continued pretraining with the usual cross-entropy as diffusion loss, and then performing further instruction tuning. We find that adaptation training as well as the choice of the base model is crucial for training good instruction-following diffusion models. We further propose reward guidance, a novel and modular inference-time guidance procedure to align model outputs without needing to train the underlying model. Finally, we show that TESS 2 further improves with increased inference-time compute, highlighting the utility of diffusion LMs in having fine-grained controllability over the amount of compute used at inference time. Code and models are available at https://github.com/hamishivi/tess-2.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 43 of its 60 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

43 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Transfer Learning for Text Diffusion Models30 Jan
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  3. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning23 Aug
  4. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model5 Jun
  5. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  6. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  7. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
24 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  12. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  13. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  15. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  16. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  17. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  18. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  19. 2024The pitfalls of next-token predictioncited by 160
  20. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  21. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  22. 2020Limitations of Autoregressive Models and Their Alternativescited by 55
  23. 2021EdiTTS: Score-based Editing for Controllable Text-to-Speechcited by 28
  24. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 18.

30 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025A Survey on Latent Reasoningcited by 81
  3. 2025A Survey on Diffusion Language Modelscited by 78
  4. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  5. 2026ELF: Embedded Language Flowscited by 45
  6. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  7. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 20
  8. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Thinkcited by 14
  9. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Mapscited by 11
  10. 2026Revealing the Attention Floating Mechanism in Masked Diffusion Modelscited by 5
  11. 2025DMark: Order-Agnostic Watermarking for Diffusion Large Language Modelscited by 4
  12. 2025Unlocking the Potential of Diffusion Language Models through Template Infillingcited by 4
18 more
  1. 2025Simplex-to-Euclidean Bijections for Categorical Flow Matchingcited by 3
  2. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Modelscited by 3
  3. 2025Guided Transfer Learning for Discrete Diffusion Modelscited by 3
  4. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  5. 2026Entropy Aware Reward Guidance for Diffusion Language Model Alignmentcited by 3
  6. 2026Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Modelscited by 3
  7. 2025Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocationcited by 2
  8. 2026ConvergeFlow: Language Flow with Provable Convergence to Token Embeddingscited by 2
  9. 2026STDec: Spatio-Temporal Stability Guided Decoding for dLLMscited by 1
  10. 2026Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusioncited by 1
  11. 2026DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speechcited by 1
  12. 2026Reasoning with Continuous Latent Diffusioncited by 1
  13. 2026Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attentioncited by 0
  14. 2026Efficient Self-Evaluation for Diffusion Language Models via Sequence Regenerationcited by 0
  15. 2026Understanding and Accelerating the Training of Masked Diffusion Language Modelscited by 0
  16. 2026FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptationcited by 0
  17. 2026Improving Few-Step Language Flows with Untied Self-Conditioningcited by 0
  18. 2026Affix Cache for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Language Models are Super Data Learners
  2. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  3. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
  4. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
  5. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  6. 2023Likelihood-Based Diffusion Language Models
  7. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
  8. 2026Scaling Beyond Masked Diffusion Language Models