mlsanityData to 7 Oct 2026

Hymba: A Hybrid-head Architecture for Small Language Models

arXiv 2411.13676 ↗First posted 20 Nov 2024cs.CL cs.AI cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) for enhanced efficiency. Attention heads provide high-resolution recall, while SSM heads enable efficient context summarization. Additionally, we introduce learnable meta tokens that are prepended to prompts, storing critical information and alleviating the "forced-to-attend" burden associated with attention mechanisms. This model is further optimized by incorporating cross-layer key-value (KV) sharing and partial sliding window attention, resulting in a compact cache size. During development, we conducted a controlled study comparing various architectures under identical settings and observed significant advantages of our proposed architecture. Notably, Hymba achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model surpasses all sub-2B public models in performance and even outperforms Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size reduction, and 3.49x throughput.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 64 of its 81 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

64 of 81 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Mistral 7Bcited by 3,065
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  13. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  14. 2023Vision Transformers Need Registerscited by 844
  15. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  16. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  17. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  18. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  19. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  20. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  21. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  22. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  23. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  24. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  25. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  26. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  27. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  28. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  29. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  30. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  31. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  32. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  33. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  34. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  35. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  36. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  37. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  38. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 50; 2026: 43.

95 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  4. 2025Small Language Models are the Future of Agentic AIcited by 254
  5. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  6. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  7. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  8. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  9. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  10. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  11. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  12. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
38 more
  1. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  2. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  3. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  4. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  5. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
  6. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  7. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  8. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  9. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  10. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  11. 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 12
  12. 2026δ-mem: Efficient Online Memory for Large Language Modelscited by 12
  13. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  14. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  15. 2025ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformercited by 9
  16. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  17. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  18. 2025Resona: Improving Context Copying in Linear Recurrence Models with Retrievalcited by 5
  19. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  20. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  21. 2025Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attentioncited by 5
  22. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  23. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  24. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  25. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  26. 2026Kwai Summary Attention Technical Reportcited by 4
  27. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 3
  28. 2025RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modelingcited by 3
  29. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modelingcited by 3
  30. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  31. 2026Frayed RoPE and Long Inputs: A Geometric Perspectivecited by 3
  32. 2026Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Designcited by 3
  33. 2025Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)cited by 2
  34. 2025WuNeng: Hybrid State with Attentioncited by 2
  35. 2025Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolationcited by 2
  36. 2025Differential Mambacited by 2
  37. 2025Delayed Attention Training Improves Length Generalization in Transformer--RNN Hybridscited by 2
  38. 2025Untangling Component Imbalance in Hybrid Linear Attention Conversion Methodscited by 2

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
  2. 2026Rethinking the Role of Efficient Attention in Hybrid Architectures
  3. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
  4. 2026Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
  5. 2026Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
  6. 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
  7. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
  8. 2024Taipan: Efficient and Expressive State Space Language Models with Selective Attention