mlsanityData to 7 Oct 2026

Fine-Tuning Diffusion Language Models with Context Selection and Target Weighting

arXiv 2609.38385 ↗First posted 29 Sep 2026cs.AI cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Supervised fine-tuning of discrete diffusion language models masks some response tokens and trains the model to recover their original values from the visible context. The masking pattern therefore determines both the context available to the model and the tokens it learns to predict. Uniform random masking does not explicitly account for the interaction between these choices. We introduce GoldiMask, which selects tokens to reveal as context by approximately maximizing a submodular objective. This objective uses model signals to balance the benefit of revealing tokens against their value as prediction targets. GoldiMask then weights the remaining targets according to how they benefit from the selected context and their remaining learning potential. Across three backbones and three training datasets, GoldiMask achieves the highest average accuracy in most evaluated settings, demonstrating gains on both reasoning and code generation. Component ablations show that both context selection and target weighting contribute to the gains. GoldiMask also reduces decoding iterations on GSM8K and MATH-500 under confidence-threshold parallel decoding, while maintaining comparable accuracy at higher confidence thresholds.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 37 of its 45 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

37 of 45 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models10 Jun
  2. 2026Supportive Token Revealing for Fast Diffusion Language Model Decoding2 Jun
  3. 2026Learnability-Informed Fine-Tuning of Diffusion Language Models21 May
  4. 2026Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models12 May
  5. 2026ELF: Embedded Language Flows11 May
  6. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling13 Apr
  7. 2026Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Models10 Feb
  8. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  9. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models26 Nov
  10. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Models7 Nov
  11. 2025GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Models25 Sep
  12. 2025Dream 7B: Diffusion Large Language Models21 Aug
3 more
  1. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking30 May
  2. 2025Large Language Diffusion Models14 Feb
  3. 2024Simple and Effective Masked Diffusion Language Models11 Jun

Other references · 22

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2025s1: Simple test-time scalingcited by 1,311
  7. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  8. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  9. 2017Automated Curriculum Learning for Neural Networkscited by 251
  10. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  11. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  12. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
10 more
  1. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  2. 2022Data-Efficient Structured Pruning via Submodular Optimizationcited by 22
  3. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  4. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMscited by 12
  5. 2020Risk-Aware Submodular Optimization for Multi-Robot Coordinationcited by 8
  6. 2020Deterministic Approximation for Submodular Maximization over a Matroid in Nearly Linear Timecited by 8
  7. 2023Submodular Minimax Optimization: Finding Effective Setscited by 7
  8. 2023ORBSLAM3-Enhanced Autonomous Toy Drones: Pioneering Indoor Explorationcited by 5
  9. 2024Practical 0.385-Approximation for Submodular Maximization Subject to a Cardinality Constraintcited by 3
  10. 2024Discretely Beyond 1/e: Guided Combinatorial Algorithms for Submodular Maximizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
  2. 2025Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
  3. 2026Improved Large Language Diffusion Models
  4. 2026EPIC: Efficient and Parallel Inference under CFG Constraints for Diffusion Language Models
  5. 2025Large Language Models to Diffusion Finetuning
  6. 2026Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models
  7. 2026Diffusion Language Models Are Natively Length-Aware
  8. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning