# Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

arXiv: https://arxiv.org/abs/2504.03624 · HTML: https://mlsanity.com/paper/2504.03624/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 62 of its 91 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 62)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)

## Built on by (most cited first, 30 of 100)

- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [Mamba-3: Improved Sequence Modeling using State Space Principles](https://mlsanity.com/paper/2603.15569/) (2026)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [Pretraining Large Language Models with NVFP4](https://mlsanity.com/paper/2509.25149/) (2025)
- [NVIDIA Nemotron 3: Efficient and Open Intelligence](https://mlsanity.com/paper/2512.20856/) (2025)
- [End-to-End Test-Time Training for Long Context](https://mlsanity.com/paper/2512.23675/) (2025)
- [Training a Scientific Reasoning Model for Chemistry](https://mlsanity.com/paper/2506.17238/) (2025)
- [Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search](https://mlsanity.com/paper/2508.15884/) (2025)
- [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2604.12374/) (2026)
- [EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes](https://mlsanity.com/paper/2507.11407/) (2025)
- [Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning](https://mlsanity.com/paper/2504.13941/) (2025)
- [IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse](https://mlsanity.com/paper/2603.12201/) (2026)
- [Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data](https://mlsanity.com/paper/2510.03264/) (2025)
- [Olmo Hybrid: From Theory to Practice and Back](https://mlsanity.com/paper/2604.03444/) (2026)
- [EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT](https://mlsanity.com/paper/2510.23569/) (2025)
- [Comba: Improving Bilinear RNNs with Closed-loop Control](https://mlsanity.com/paper/2506.02475/) (2025)
- [Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models](https://mlsanity.com/paper/2511.18890/) (2025)
- [Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling](https://mlsanity.com/paper/2601.02346/) (2026)
- [Short window attention enables long-term memorization](https://mlsanity.com/paper/2509.24552/) (2025)
- [Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow](https://mlsanity.com/paper/2601.14243/) (2026)
- [InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models](https://mlsanity.com/paper/2512.08829/) (2025)
- [HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing](https://mlsanity.com/paper/2602.03560/) (2026)
- [Detecting Data Contamination in LLMs via In-Context Learning](https://mlsanity.com/paper/2510.27055/) (2025)
- [Lost in the Noise: How Reasoning Models Fail with Contextual Distractors](https://mlsanity.com/paper/2601.07226/) (2026)
- [Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline](https://mlsanity.com/paper/2603.05484/) (2026)
- [Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning](https://mlsanity.com/paper/2504.11409/) (2025)
- [Sampling from Your Language Model One Byte at a Time](https://mlsanity.com/paper/2506.14123/) (2025)

## Abstract (arXiv)

As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid Mamba-Transformer models designed to reduce inference cost for a given accuracy level. To achieve this goal, we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token. We show that Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized state-of-the-art open-sourced Transformer models (e.g., Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3× faster at inference. To further increase inference speed and reduce the memory required at inference time, we created Nemotron-H-47B-Base from the 56B model using a new compression via pruning and distillation technique called MiniPuzzle. Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20% faster to infer. In addition, we introduce an FP8-based training recipe and show that it can achieve on par results with BF16-based training. This recipe is used to train the 56B model. We are releasing Nemotron-H base model checkpoints with support in Hugging Face and NeMo.