# Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

arXiv: https://arxiv.org/abs/2507.22448 · HTML: https://mlsanity.com/paper/2507.22448/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2025.

## Built on by (most cited first, 30 of 57)

- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence](https://mlsanity.com/paper/2511.07384/) (2025)
- [Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search](https://mlsanity.com/paper/2508.15884/) (2025)
- [Olmo Hybrid: From Theory to Practice and Back](https://mlsanity.com/paper/2604.03444/) (2026)
- [The Rogue Scalpel: Activation Steering Compromises LLM Safety](https://mlsanity.com/paper/2509.22067/) (2025)
- [Weight Decay may matter more than muP for Learning Rate Transfer in Practice](https://mlsanity.com/paper/2510.19093/) (2025)
- [Distilling to Hybrid Attention Models via KL-Guided Layer Selection](https://mlsanity.com/paper/2512.20569/) (2025)
- [Hybrid Architectures for Language Models: Systematic Analysis and Design Insights](https://mlsanity.com/paper/2510.04800/) (2025)
- [Arcee Trinity Large Technical Report](https://mlsanity.com/paper/2602.17004/) (2026)
- [Optimal Scaling Needs Optimal Norm](https://mlsanity.com/paper/2510.03871/) (2025)
- [DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone](https://mlsanity.com/paper/2511.15927/) (2025)
- [SpikingBrain: Spiking Brain-inspired Large Models](https://mlsanity.com/paper/2509.05276/) (2025)
- [MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling](https://mlsanity.com/paper/2602.11761/) (2026)
- [Efficient Hyperparameter Tuning via Trajectory Invariance Principle](https://mlsanity.com/paper/2509.25049/) (2025)
- [KORMo: Korean Open Reasoning Model for Everyone](https://mlsanity.com/paper/2510.09426/) (2025)
- [TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding](https://mlsanity.com/paper/2511.16595/) (2025)
- [Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications](https://mlsanity.com/paper/2603.22473/) (2026)
- [Memory for Large Language Models](https://mlsanity.com/paper/2607.25380/) (2026)
- [When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models](https://mlsanity.com/paper/2602.13215/) (2026)
- [OntoLearner: A Modular Python Library for Ontology Learning with Large Language Models](https://mlsanity.com/paper/2607.01977/) (2026)
- [Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design](https://mlsanity.com/paper/2609.02986/) (2026)
- [Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs](https://mlsanity.com/paper/2509.25380/) (2025)
- [Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers](https://mlsanity.com/paper/2601.04890/) (2026)
- [UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop](https://mlsanity.com/paper/2601.21000/) (2026)
- [AllMem: A Memory-centric Recipe for Efficient Long-context Modeling](https://mlsanity.com/paper/2602.13680/) (2026)
- [S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models](https://mlsanity.com/paper/2604.01168/) (2026)
- [SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference](https://mlsanity.com/paper/2604.22575/) (2026)
- [XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity](https://mlsanity.com/paper/2605.05662/) (2026)
- [Morphing into Hybrid Attention Models](https://mlsanity.com/paper/2606.30562/) (2026)
- [Buy versus Build an LLM: A Decision Framework for Governments](https://mlsanity.com/paper/2602.13033/) (2026)

## Abstract (arXiv)

In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across diverse use cases. Unlike earlier Falcon models built solely on Transformer or Mamba architectures, Falcon-H1 adopts a parallel hybrid approach that combines Transformer-based attention with State Space Models (SSMs), known for superior long-context memory and computational efficiency. We systematically revisited model design, data strategy, and training dynamics, challenging conventional practices in the field. Falcon-H1 is released in multiple configurations, including base and instruction-tuned variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters. Quantized instruction-tuned models are also available, totaling over 30 checkpoints on Hugging Face Hub. Falcon-H1 models demonstrate state-of-the-art performance and exceptional parameter and training efficiency. The flagship Falcon-H1-34B matches or outperforms models up to 70B scale, such as Qwen3-32B, Qwen2.5-72B, and Llama3.3-70B, while using fewer parameters and less data. Smaller models show similar trends: the Falcon-H1-1.5B-Deep rivals current leading 7B-10B models, and Falcon-H1-0.5B performs comparably to typical 7B models from 2024. These models excel across reasoning, mathematics, multilingual tasks, instruction following, and scientific knowledge. With support for up to 256K context tokens and 18 languages, Falcon-H1 is suitable for a wide range of applications. All models are released under a permissive open-source license, underscoring our commitment to accessible and impactful AI research.