mlsanityData to 14 Oct 2026

Kairos: Toward a Byte-Level Foundation Model for Every Modality

arXiv 2610.12495 ↗First posted 26 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Small language models trained on a single local machine leave little room for architectural trial and error. Kairos combines bidirectional gated-DeltaNet linear attention and sliding-window attention over shared projections (LiZAttention2), block-wise Attention-Residual aggregation, a sparse Mixture-of-Experts (MoE) feed-forward layer and a multi-scale byte-level codec, trained with a uniform-noise denoising objective over one byte space shared by text, image, audio, video, lidar and control streams; this report asks whether the combination optimizes at all before asking whether it is useful. We give code-derived algorithms for each component, including a cross-session recurrent-state memory gate, then run convergence sanity checks: overfittinga single 440-byte sentence at 1--15M parameters, text only, with three seeds for codec and attention choices and single-run sweeps for seven further factors. The two attention variants tested at 3,000 steps (LiZAttention2 and window attention alone) reach a minimum loss near 10-3. After 600 steps, the mean minimum loss ranges from 0.018 (patch codec) to 0.354 (depthwise convolutional codec) with an identical ranking on three seeds. An apparent advantage of shared over separate attention projections closes with more steps or more diverse data and reverses when the projections are frozen: it is a convergence-speed effect, not a capacity effect. These checks suggest that the implementation optimizes; they say nothing about generalization, multimodal training or comparison with existing models, which a planned pretraining run is meant to address. The Discussion sizes a 200M-total/25M-active target analytically; none of its throughput figures is measured. Code is available at https://github.com/fabienfrfr/Kairos . Python package at https://pypi.org/project/kairos-fm/ .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 39 of its 43 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

39 of 43 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023Mistral 7Bcited by 3,164
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
27 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2025Large Language Diffusion Modelscited by 912
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  10. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  12. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  13. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  14. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  15. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  16. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  17. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  18. 2024Titans: Learning to Memorize at Test Timecited by 328
  19. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  20. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  21. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  22. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  23. 2026Attention Residualscited by 65
  24. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  25. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 31
  26. 2024PixelBytes: Catching Unified Representation for Multimodal Generationcited by 2
  27. 2025TPTT: Transforming Pretrained Transformers into Titanscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion
  2. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  3. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
  4. 2026Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
  5. 2026Sliding-window beats linear attention
  6. 2024Cross-layer Attention Sharing for Pre-trained Large Language Models
  7. 2025DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
  8. 2026DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding