mlsanityData to 7 Oct 2026

Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

arXiv 2606.23690 ↗First posted 9 Apr 2026cs.SE cs.AIcited by 0

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

Autoregressive (AR) language models have driven significant progress in automated software engineering, enabling powerful code generation and assistance systems. However, the next-token prediction paradigm introduces structural limitations for code reasoning, including restricted global planning, challenges in maintaining long-range dependencies, and limited grounding in program execution semantics. Noting the heavy skewness of existing literature towards AR models, we discuss emerging paradigms that could potentially overcome the logic and scaling bottlenecks of next-token prediction by unlocking next-generation architectural capabilities for code intelligence. Specifically, we discuss the potential of Diffusion Models, which generate code via holistic denoising that captures long-range syntactic constraints often missed by AR models. We also discuss Code World Models (CWMs), which simulate execution states to support reasoning, and State Space Models (SSMs), which provide linear-time efficiency for massive contexts. By connecting these developments with findings from cognitive neuroscience, we outline directions for developing "System 2" code generation agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 49 of its 68 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

49 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?21 Dec
  2. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Models30 Sep

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
35 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2023Large Language Modelscited by 774
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  6. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2023How Language Model Hallucinations Can Snowballcited by 310
  10. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  11. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  13. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  14. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  15. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  16. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  17. 2020Limitations of Autoregressive Models and Their Alternativescited by 54
  18. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  19. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  20. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  21. 2025Constrained Discrete Diffusioncited by 16
  22. 2025Exploring the Potential of Diffusion Large Language Models in Code Generationcited by 16
  23. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  24. 2025Constrained Decoding of Diffusion LLMs with Context-Free Grammarscited by 13
  25. 2024ROCODE: Integrating Backtracking Mechanism and Program Analysis in Large Language Models for Code Generationcited by 12
  26. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  27. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  28. 2024Diffusion On Syntax Trees For Program Synthesiscited by 10
  29. 2025Challenges and Paths Towards AI for Software Engineeringcited by 10
  30. 2024Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Modelscited by 6
  31. 2025Directional Diffusion-Style Code Editing Pre-trainingcited by 3
  32. 2025CodeSSM: Towards State Space Models for Code Understandingcited by 3
  33. 2025Diffusion is a code repair operator and generatorcited by 3
  34. 2025DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generationcited by 2
  35. 2026Towards Understanding What State Space Models Learn About Codecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Blackboard Intelligence Can Surpass Autoregressive on Globally Constrained Problems
  2. 2026AnCoder: Anchored Code Generation via Discrete Diffusion Models
  3. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  4. 2025Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
  5. 2026Towards Evaluation of Implicit Software World Models in Coding LLMs
  6. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
  7. 2024A new approach for encoding code and assisting code understanding
  8. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas