mlsanityData to 14 Oct 2026

FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

arXiv 2610.16301 ↗First posted 12 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position signals bound to the training length. Existing attention-free alternatives all optimize the single function of generation. We propose FON (Fusion-Organ Network), which differentiates one weight stack into a drafting organ (multi-tap learnable delays + content-adaptive recursion; causal, O(S) training, O(1) decode state, no sequence-level quadratic attention) and a revision organ that re-processes the written prefix through multi-scale ripple kernels injected via weak gates, never generating. At 30M parameters, iso-parameter and iso-token budget, FON reaches perplexity 131.03 on WikiText-103, beating Transformer (197.73) and Mamba2 (152.53); the advantage persists under all depth-aligned baselines and at the 70M tier (-20.8%). At 8x length extrapolation FON degrades by ~8-9% without positional-encoding engineering, between ALiBi and RoPE. The revision channel provides a structural pathway for in-domain infilling that causal architectures lack in principle (+19% over on-demand coupling), at a perplexity cost indistinguishable within measurement precision. We report all costs and boundaries: impaired long-range retrieval, synthetic-task disadvantages, weaker out-of-domain code infilling than a same-recipe Transformer, and a 6.5-12.3x wall-clock gap.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

50 of 65 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  9. 2022In-context Learning and Induction Headscited by 964
  10. 2025Large Language Diffusion Modelscited by 912
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  13. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  14. 2014Neural Turing Machinescited by 644
  15. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  16. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  17. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  18. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  19. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  20. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  21. 2014Memory Networkscited by 363
  22. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  23. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  24. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  25. 2022Generating Sequences by Learning to Self-Correctcited by 277
  26. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  27. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  28. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  29. 2023Looped Transformers as Programmable Computerscited by 237
  30. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  31. 2020Enabling Language Models to Fill in the Blankscited by 122
  32. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  33. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 25
  34. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoningcited by 9
  35. 2025Making, not Taking, the Best of Ncited by 6
  36. 2025Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modelingcited by 2
  37. 2026The Bicameral Model: Bidirectional Hidden-State Coupling Between Parallel Language Modelscited by 2
  38. 2026ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modelingcited by 2

Built on by

Not cited yet.