mlsanityData to 14 Oct 2026

Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation

arXiv 2605.07924 ↗First posted 8 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Discrete flow matching generates text by iteratively transforming noise tokens into coherent language, but may require hundreds of forward passes. Distillation uses the multi-step trajectory to train a student to reproduce the process in a few steps. When the student underperforms, the usual explanation is insufficient capacity. We argue the opposite: the trajectory is the bottleneck, not the student. Each training trajectory is built through a chain of blind stochastic jumps with no evaluation of sequence quality; a single bad decision at an early midpoint propagates through subsequent steps, yet the student must imitate the result. Trajectory-Shaped Discrete Flow Matching (TS-DFM) replaces these blind jumps with guided navigation: a lightweight energy compass evaluates candidate continuations at each midpoint, selecting the most coherent. All shaping is training-only; inference cost is unchanged. On 170M-parameter language modeling, the shaped student at 8 steps achieves 32% lower perplexity than the 1,024-step teacher while being 128x faster, with gains consistent across source distributions and three evaluators of increasing scale. TS-DFM achieves the best perplexity of any discrete-generation baseline we compare against, including methods trained on 6x more data or using 5x larger models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 39 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

35 of 39 references.

References · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2023Consistency Modelscited by 1,822
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  10. 2025Large Language Diffusion Modelscited by 912
  11. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
23 more
  1. 2022On Distillation of Guided Diffusion Modelscited by 725
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  3. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  5. 2023Universal Guidance for Diffusion Modelscited by 437
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  7. 2024One Step Diffusion via Shortcut Modelscited by 391
  8. 2024Discrete Flow Matchingcited by 317
  9. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 305
  10. 2021How to Train Your Energy-Based Modelscited by 274
  11. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  12. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 220
  13. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  14. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  15. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  16. 2025The Diffusion Dualitycited by 135
  17. 2020A Distributional Approach to Controlled Text Generationcited by 119
  18. 2020Residual Energy-Based Models for Text Generationcited by 95
  19. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  20. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 78
  21. 2024EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Samplingcited by 45
  22. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  23. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Modelscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DSFlow: Dual Supervision and Step-Aware Architecture for One-Step Flow Matching Speech Synthesis
  2. 2026A Kinetic Energy Perspective of Flow Matching
  3. 2026Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech
  4. 2026Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech
  5. 2026Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher
  6. 2026Few-Step Diffusion Language Models via Trajectory Self-Distillation
  7. 2026TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM
  8. 2026TACD: Distilling Efficient Text-to-Motion Models via Terminal Amplification Control