mlsanityData to 7 Oct 2026

Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff

arXiv 2602.16092 ↗First posted 17 Feb 2026cs.LG cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Any-order autoregressive models (AO-ARMs) offer a promising path toward efficient masked diffusion by enabling native key-value caching, but competitive performance has so far required two-stream attention, typically motivated as a means of decoupling token content from position. In this work, we argue that two-stream attention may be serving a more subtle role. We identify a structural-semantic tradeoff in any-order generation: the hidden representation at each step must simultaneously attend to semantically informative tokens for prediction and structurally recent tokens for summarization, objectives that compete for attention capacity in a single stream but can specialize across two streams. To isolate this tradeoff from position-content separation, we propose Decoupled RoPE, a modification to rotary position embeddings that provides target position information without revealing target content. Decoupled RoPE performs competitively at short sequence lengths--where semantic and structural proximity coincide--but degrades as sequence length increases and the two orderings diverge. These results suggest that the success of two-stream attention stems not merely from separating position from content, but from circumventing the deeper structural-semantic tradeoff inherent to any-order generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 10 of its 10 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

10 of 10 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Auto-Regressive Masked Diffusion Models23 Jan
  2. 2026Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation19 Jan
  3. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture24 Jun
  4. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  6. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun

Other references · 4

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  4. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
  2. 2025Unifying Autoregressive and Diffusion-Based Sequence Generation
  3. 2026Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens
  4. 2026Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
  5. 2025Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling
  6. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Token
  7. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  8. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference