mlsanityData to 14 Oct 2026

The Wiola Architecture for Efficient Small Language Models

arXiv 2607.01394 ↗First posted 1 Jul 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon. Wiola introduces five independently novel components: (i) Spiral Rotary Positional Encoding (SRPE), which embeds token positions on a three-dimensional helical manifold combining absolute, relative, and hierarchical positional signals; (ii) Gated Cross-Layer Attention (GCLA), providing each decoder layer with soft cross-attention access to compressed summaries of two preceding layers for inter-layer coherence; (iii) Adaptive Token Merging (ATM), which dynamically merges se mantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss; (iv) Dual Stream Feed-Forward (DSFF), replacing the conventional MLP with two parallel streams fused by a learned per-dimension gate; and (v) WiolaRMSNorm, a modified normalisation introducing a per-dimension learned offset vector that prevents representation collapse. We provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters) and is fully compatible with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

23 of 29 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2016Layer Normalizationcited by 5,669
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2020GLU Variants Improve Transformercited by 1,804
11 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  4. 2022Token Merging: Your ViT But Fastercited by 1,021
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  6. 2020On Layer Normalization in the Transformer Architecturecited by 922
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  11. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Return of the Encoder: Maximizing Parameter Efficiency for SLMs
  2. 2024Hymba: A Hybrid-head Architecture for Small Language Models
  3. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation
  4. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
  5. 2024A Survey of Small Language Models
  6. 2025SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment
  7. 2025Large Language Model Partitioning for Low-Latency Inference at the Edge
  8. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models