mlsanityData to 14 Oct 2026

Tensor-Train Joint Modeling for Few-Step Discrete Diffusion

arXiv 2607.03788 ↗First posted 4 Jul 2026cs.LGcited by 1

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Discrete diffusion promises orders-of-magnitude faster generation than autoregressive (AR) models for sequential discrete data, yet its full potential of few-step generation has remained out of reach due to a fundamental structural limitation. The conditional-independence assumption underlying current discrete diffusion models introduces a systematic parallelization bias that compounds with the number of tokens unmasked per step, becoming severe in the few-step regime that fast generation requires. We address this with the first framework for explicit joint distribution modeling in discrete diffusion via tensor decomposition, which represents the conditional clean distribution as a low-rank tensor with controllable expressivity. The framework supports both Canonical Polyadic (CPD) and Tensor-Train (TTD) decompositions, and we identify a structural bias of TTD toward dependencies between nearby tokens, formalized through Oseledets' theorem relating TT-rank to unfolding-matrix rank, which is well-suited to sequential data such as natural language and line notations for molecular data. To enable efficient generation, we present an iterative marginal inference procedure with specialization for predetermined position schedules. Our framework integrates into pretrained MDMs through lightweight fine-tuning, yielding substantial improvements in few-step generation at a fraction of the cost of training from scratch. Code available at https://github.com/ssamt/tensor-train.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 20 of its 30 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

20 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling23 May
  2. 2024Distillation of Discrete Diffusion through Dimensional Correlations11 Oct
  3. 2022Scalable Diffusion Models with Transformers19 Dec
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul

Other references · 16

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2025Large Language Diffusion Modelscited by 912
  3. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  6. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  8. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  9. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  10. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 131
  11. 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
  12. 2024Discrete Copula Diffusioncited by 54
4 more
  1. 2025Halton Scheduler For Masked Generative Image Transformercited by 32
  2. 2026Breaking the Factorization Barrier in Diffusion Language Modelscited by 11
  3. 2026Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspectivecited by 10
  4. 2024Faster Language Models with Better Multi-Token Prediction Using Tensor Decompositioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Alpha Diffusion Language Models: Factorization Alone Is Not the Problemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Discrete generative diffusion models without stochastic differential equations: a tensor network approach
  2. 2026Tucker Diffusion Model for High-dimensional Tensor Generation
  3. 2025Distillation of Discrete Diffusion by Exact Conditional Distribution Matching
  4. 2026Tensor Train Diffusion: Leveraging Low-Rank Structures for High-Dimensional Score-Based Sampling
  5. 2024Unified Discrete Diffusion for Categorical Data
  6. 2025Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics
  7. 2026Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster
  8. 2026Consistent Diffusion Language Models