mlsanityData to 7 Oct 2026

Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learning

arXiv 2610.12340 ↗First posted 8 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce RefineMix, a framework for training discrete diffusion models under severe data scarcity, a common constraint in scientific applications. RefineMix uses out-of-distribution data at selected diffusion times to improve generalization without biasing the sampling distribution. Although this strategy has been explored in continuous diffusion, discrete diffusion presents a distinct challenge: unlike Gaussian noise, masking preserves domain information in surviving tokens, limiting the use of related data at high noise levels. At low noise levels, however, the domains effectively disjoint supports become an advantage, allowing the model to learn from both in-domain and out-of-distribution data without biasing the sampler. We formalize these intuitions and provide a theoretical analysis for the proposed method. Experimentally, across five domain-shift settings, RefineMix matches or outperforms in-domain finetuning and data mixing. For protein sequence generation, finetuning with just 197 in-domain examples nearly doubles the fraction of generated proteins that are simultaneously novel, foldable, and in-family compared to standard finetuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

36 of 64 references.

References · 36

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  7. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  8. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  9. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
24 more
  1. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  3. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  4. 2023Generative Diffusion Models on Graphs: Methods and Applicationscited by 79
  5. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  6. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  7. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  8. 2025GenMol: A Drug Discovery Generalist with Discrete Diffusioncited by 47
  9. 2025Diffusion Language Models are Super Data Learnerscited by 45
  10. 2023GSURE-Based Diffusion Model Training with Corrupted Datacited by 40
  11. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  12. 2025Ambient Diffusion Omni: Training Good Models with Bad Datacited by 18
  13. 2024Transfer Learning for Diffusion Modelscited by 10
  14. 2025What Makes Diffusion Language Models Super Data Learners?cited by 8
  15. 2025Measurement Score-Based Diffusion Modelcited by 7
  16. 2025DiffEM: Learning from Corrupted Data with Diffusion Models via Expectation Maximizationcited by 6
  17. 2025Generative Modeling from Black-box Corruptions via Self-Consistent Stochastic Interpolantscited by 5
  18. 2026Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Roboticscited by 4
  19. 2025Score Distillation Beyond Acceleration: Generative Modeling from Corrupted Datacited by 3
  20. 2025Guided Transfer Learning for Discrete Diffusion Modelscited by 3
  21. 2025SFBD-OMNI: Bridge models for lossy measurement restoration with limited clean samplescited by 2
  22. 2026Ambient Dataloops: Generative Models for Dataset Refinementcited by 2
  23. 2026SCoRe: Clean Image Generation from Diffusion Models Trained on Noisy Imagescited by 2
  24. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 2

Built on by

Not cited yet.