FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation
On the map: Diffusion models → Discrete and text diffusion
Abstract (arXiv)
Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 46 of its 57 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
46 of 57 references.
Closest ancestors · 17
Same direction, newest first
- 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation7 Oct
- 2025Training Optimal Large Diffusion Language Models28 Sep
- 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models8 Sep
- 2025Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs31 Aug
- 2025Dream 7B: Diffusion Large Language Models21 Aug
- 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs17 Jun
- 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking30 May
- 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
- 2025Large Language Diffusion Models14 Feb
- 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
- 2024Scaling up Masked Diffusion Models on Text24 Oct
- 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
5 more
- 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
- 2024Think While You Generate: Discrete Diffusion with Planned Denoising8 Oct
- 2024Simple and Effective Masked Diffusion Language Models11 Jun
- 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
- 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
Other references · 29
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
- 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
- 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
- 2024InternLM2 Technical Reportcited by 397
17 more
- 2022How Do Vision Transformers Work?cited by 307
- 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
- 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
- 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
- 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
- 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
- 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
- 2025Halton Scheduler For Masked Generative Image Transformercited by 32
- 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
- 2023Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operatorcited by 20
- 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
- 2022DCT-Former: Efficient Self-Attention with Discrete Cosine Transformcited by 12
- 2024Frequency-Controlled Diffusion Model for Versatile Text-Guided Image-to-Image Translationcited by 12
- 2024LongWanjuan: Towards Systematic Measurement for Long Text Qualitycited by 11
- 2023Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relationscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
- 2026Path-Dependent Denoising: A Non-Conservative Field Perspective on Order Collapse in Diffusion Language Models
- 2026Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference
- 2026Scaling Beyond Masked Diffusion Language Models
- 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
- 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
- 2026Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
- 2026Locally Coherent Parallel Decoding in Diffusion Language Models