mlsanityData to 7 Oct 2026

σ-GPTs: A New Approach to Autoregressive Models

arXiv 2404.09562 ↗First posted 15 Apr 2024cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Autoregressive models, such as the GPT family, use a fixed order, usually left-to-right, to generate sequences. However, this is not a necessity. In this paper, we challenge this assumption and show that by simply adding a positional encoding for the output, this order can be modulated on-the-fly per-sample which offers key advantageous properties. It allows for the sampling of and conditioning on arbitrary subsets of tokens, and it also allows sampling in one shot multiple tokens dynamically according to a rejection strategy, leading to a sub-linear number of model evaluations. We evaluate our method across various domains, including language modeling, path-solving, and aircraft vertical rate prediction, decreasing the number of steps required for generation by an order of magnitude.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

20 of 23 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2023Consistency Modelscited by 1,778
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  9. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  12. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
8 more
  1. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  2. 2023Likelihood-Based Diffusion Language Modelscited by 147
  3. 2023Explaining grokking through circuit efficiencycited by 105
  4. 2022Improved Masked Image Generation with Token-Criticcited by 63
  5. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  6. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  7. 2022Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformercited by 30
  8. 2024Arrows of Time for Large Language Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 1.

20 papers.

Sort:

By area:

  1. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  2. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  3. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  4. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  5. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 22
  6. 2025Multi-Token Prediction Needs Registerscited by 16
  7. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecturecited by 16
  8. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 14
  9. 2025Self-Speculative Masked Diffusionscited by 11
  10. 2025World Modeling with Probabilistic Structure Integrationcited by 7
  11. 20253D Scene Understanding Through Local Random Access Sequence Modelingcited by 5
  12. 2025Looking beyond the next tokencited by 5
8 more
  1. 2025NEP: Autoregressive Image Editing via Next Editing Token Predictioncited by 5
  2. 2025Scalable Single-Cell Gene Expression Generation with Latent Diffusion Modelscited by 2
  3. 2025Distilling Specialized Orders for Visual Generationcited by 1
  4. 2025Corrector Sampling in Language Modelscited by 1
  5. 2025Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learningcited by 1
  6. 2025A document is worth a structured record: Principled inductive bias design for document recognitioncited by 0
  7. 2025Spanning Tree Autoregressive Visual Generationcited by 0
  8. 2026SenBen: Sensitive Scene Graphs for Explainable Content Moderationcited by 0