mlsanityData to 14 Oct 2026

When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs

arXiv 2606.24119 ↗First posted 23 Jun 2026cs.LG cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Discrete diffusion language model (DLM) fine-tuning inherits inexpensive diagnostics from denoising-time confidence monitors, but their PEFT-training meaning is untested. We test top-1 argmax concentration as a collapse warning. Across 816 LoRA/PEFT configurations from three DLM families, the warning fires for every configuration while logs record 0/816 actual collapses at the 200 step horizon, giving zero precision. The cause is pre-equilibrium saturation: top-1 concentration is already high before optimization and quickly becomes insensitive to final training stability. We then evaluate max LoRA gradient norm, a parameter-side signal that samples gradient routing rather than token concentration. On a pooled held-out LLaDA-family split, a train-optimized threshold identifies top-decile final-loss configurations with precision 0.68 and F1=0.79, above the all-positive top-1 baseline even at the lower split-bootstrap confidence bound. Autoregressive controls and cross-family threshold failures bound the result to short-horizon DLM-LoRA inspection rather than a universal collapse detector. Workflow: drop top-1 as a PEFT alarm, log max-gradient early in training, and calibrate thresholds per DLM family before routing runs for inspection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 20 of its 34 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

20 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Layer Collapse in Diffusion Language Models7 May
  2. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Parity30 Jan
  3. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Model29 Sep
  4. 2025Dream 7B: Diffusion Large Language Models21 Aug

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  9. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  10. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  11. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 14
  12. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 10
4 more
  1. 2025Riemannian Optimization for LoRA on the Stiefel Manifoldcited by 10
  2. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  3. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 4
  4. 2026Benchmarking Recursive-Collapse Warning Claims Under Matched False-Positive Controlcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learn from Your Mistakes: Self-Correcting Masked Diffusion Models
  2. 2026A Decomposable Probe for Few-Step Diffusion Models: Prompt, Latent, and Score Selectivity across Backbone Families and Distillation Paradigms
  3. 2026JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models
  4. 2023LoRA-Enhanced Distillation on Guided Diffusion Models
  5. 2026When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities
  6. 2026Revise, Don't Freeze: Sampler-Matched Training for Self-Correcting Masked Diffusion Language Models
  7. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  8. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models