MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models
On the map: LLM reasoning → Diffusion reasoning and test-time scaling
also in 1 more
Abstract (arXiv)
Discrete diffusion language models (dLLMs) generate text by iteratively denoising a masked sequence. However, standard dLLMs condition each denoising step solely on the current hard-masked sequence, while intermediate continuous representations are discarded after sampling and remasking. We term this bottleneck the Information Island issue: continuous information remains isolated within individual denoising steps and fails to propagate across the trajectory. This bottleneck is especially harmful for reasoning, which requires intermediate reasoning state to be preserved and updated across many denoising steps. To address this limitation, we introduce MetaState, a lightweight recurrent augmentation that equips a frozen dLLM backbone with persistent, fixed-size working memory. MetaState comprises three modules with a shared time conditioner: a cross-attention Mixer that reads backbone activations into memory slots, a GRU-style Updater that integrates information across steps, and a cross-attention Injector that writes the updated memory back into the backbone. We train these modules with a dedicated K-step unrolling pipeline to learn multi-step dynamics. MetaState adds only ~0.6% trainable parameters while keeping the backbone frozen, and consistently improves reasoning performance over frozen baselines on mathematical reasoning and code generation benchmarks, with an average gain of 4.5 percentage points across all evaluations.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 39 of its 45 references that are in the map. First posted on arXiv 2 Mar 2026.
Builds on
39 of 45 references.
Closest ancestors · 2
Same direction, newest first
- 2026Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning24 Feb
- 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
Other references · 37
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2025Large Language Diffusion Modelscited by 872
- 2024Simple and Effective Masked Diffusion Language Modelscited by 800
- 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
25 more
- 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
- 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
- 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
- 2025Dream 7B: Diffusion Large Language Modelscited by 515
- 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
- 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
- 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
- 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
- 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
- 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
- 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
- 2022Scalable Adaptive Computation for Iterative Generationcited by 190
- 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
- 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
- 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
- 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 109
- 2025A Survey on Diffusion Language Modelscited by 75
- 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
- 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
- 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 32
- 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 31
- 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 25
- 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
- 2025Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief Statescited by 9
- 2026Residual Context Diffusion Language Modelscited by 9
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning
- 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
- 2025EDIT: Early Diffusion Inference Termination for dLLMs Based on Dynamics of Training Gradients
- 2026Hierarchical Continuous Diffusion Language Models
- 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
- 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
- 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
- 2026Continuous Interaction Diffusion: A Diffusion-Native Runtime for Asynchronous Tool-Augmented Reasoning