mlsanityData to 7 Oct 2026

UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models

arXiv 2510.10481 ↗First posted 12 Oct 2025cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Diffusion LLMs have attracted growing interest, with plenty of recent work emphasizing their great potential in various downstream tasks; yet the long-context behavior of diffusion LLMs remains largely uncharted. We present a case study of post-training techniques for extending the context window of diffusion LLMs (i.e., LLaDA) without retraining from scratch. We show that a simple modification to the standard Rotary Positional Embeddings (RoPE) extension effectively accommodates the probabilistic modeling inherent in the diffusion process, enabling stable scaling to longer context ranges. We further compare masking strategies used during post-training and analyze their impact on optimization stability and long-range recall. Instantiating these insights, we introduce UltraLLaDA, a diffusion LLM with a 128K-token context window that, in our empirical evaluation on long-context tasks, significantly outperforms training-free baselines. Our experimental results highlight the special positional extension as a key lever for scaling diffusion LLMs to extended contexts and offer practical guidance for practitioners seeking 128K-scale context via efficient post-training.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 36 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

33 of 36 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2025Large Language Diffusion Modelscited by 872
  12. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
21 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  6. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  7. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  8. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  9. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  10. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  11. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  12. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  13. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  14. 2023Likelihood-Based Diffusion Language Modelscited by 147
  15. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  16. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  17. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  18. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  19. 2025A Survey on Diffusion Language Modelscited by 75
  20. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  21. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 21
  2. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 14
  3. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedulescited by 10
  4. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  5. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  6. 2026Improving Variable-Length Generation in Diffusion Language Models via Length Regularizationcited by 4
  7. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 2
  8. 2026Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Tamingcited by 2
  9. 2026TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decodingcited by 2
  10. 2026Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attentioncited by 1
  11. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scalecited by 1
  12. 2026FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generationcited by 0
1 more
  1. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Modelscited by 0