mlsanityData to 13 Oct 2026

Decentralized Autoregressive Generation

arXiv 2601.03184 ↗First posted 6 Jan 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The decentralization of autoregressive generation has attracted considerable attention in recent years as a solution to scaling bottlenecks. However, despite promising empirical results, this paradigm currently lacks rigorous theoretical justification. In this work, we formally establish the theoretical equivalence between decentralized and centralized training. To achieve this, we adapt the Discrete Flow Matching framework for autoregressive generation, leveraging its inherent properties to demonstrate that global models naturally decompose into independent experts. Finally, we conduct extensive experiments across diverse multimodal benchmarks, empirically validating that decentralized training maintains competitive parity with standard centralized architectures.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

32 of 38 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Scaling Laws for Neural Language Modelscited by 6,494
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  4. 2022Flow Matching for Generative Modelingcited by 5,542
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,118
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  10. 2019Towards VQA Models That Can Readcited by 1,705
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,586
20 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  3. 2016Modeling Context in Referring Expressionscited by 1,057
  4. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,012
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 898
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  7. 2016A Diagram Is Worth A Dozen Imagescited by 796
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 706
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 514
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  11. 2024Discrete Flow Matchingcited by 309
  12. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 299
  13. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  14. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 87
  15. 2024DeFoG: Discrete Flow Matching for Graph Generationcited by 78
  16. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 70
  17. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  18. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  19. 2025Decentralized Diffusion Modelscited by 12
  20. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models
  2. 2026Heterogeneous Decentralized Diffusion Models
  3. 2025Multi-scale Autoregressive Models are Laplacian, Discrete, and Latent Diffusion Models in Disguise
  4. 2025Gradients: When Markets Meet Fine-tuning -- A Distributed Approach to Model Optimisation
  5. 2026Memory-Conditioned Flow-Matching for Stable Autoregressive PDE Rollouts
  6. 2026Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation
  7. 2025Paris: A Decentralized Trained Open-Weight Diffusion Model
  8. 2025Nested AutoRegressive Models