mlsanityData to 14 Oct 2026

Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

arXiv 2504.03624 ↗First posted 4 Apr 2025cs.CL cs.AI cs.LGcited by 100

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid Mamba-Transformer models designed to reduce inference cost for a given accuracy level. To achieve this goal, we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token. We show that Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized state-of-the-art open-sourced Transformer models (e.g., Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3× faster at inference. To further increase inference speed and reduce the memory required at inference time, we created Nemotron-H-47B-Base from the 56B model using a new compression via pruning and distillation technique called MiniPuzzle. Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20% faster to infer. In addition, we introduce an FP8-based training recipe and show that it can achieve on par results with BF16-based training. This recipe is used to train the 56B model. We are releasing Nemotron-H base model checkpoints with support in Hugging Face and NeMo.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 62 of its 91 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

62 of 91 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2024Qwen2.5 Technical Reportcited by 4,942
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  13. 2019Root Mean Square Layer Normalizationcited by 1,480
  14. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  15. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  16. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  17. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  18. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  19. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  20. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  21. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  22. 2024DeepSeek-V3 Technical Reportcited by 811
  23. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  24. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  25. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  26. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  27. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  28. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  29. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  30. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  31. 2022FP8 Formats for Deep Learningcited by 196
  32. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  33. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  34. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 164
  35. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  36. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  37. 2024Nemotron-4 340B Technical Reportcited by 144
  38. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 39; 2026: 61.

100 papers.

Sort:

By area:

  1. 2025Small Language Models are the Future of Agentic AIcited by 270
  2. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  3. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  4. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  5. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  6. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  7. 2025Pretraining Large Language Models with NVFP4cited by 76
  8. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 65
  9. 2025End-to-End Test-Time Training for Long Contextcited by 62
  10. 2025Training a Scientific Reasoning Model for Chemistrycited by 56
  11. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  12. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
38 more
  1. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 32
  2. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoningcited by 31
  3. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  4. 2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 27
  5. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  6. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  7. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  8. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  9. 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 13
  10. 2025Short window attention enables long-term memorizationcited by 11
  11. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  12. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 10
  13. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 10
  14. 2025Detecting Data Contamination in LLMs via In-Context Learningcited by 8
  15. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  16. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  17. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  18. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  19. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Servingcited by 7
  20. 2025AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasonerscited by 6
  21. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 6
  22. 2025Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMscited by 6
  23. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  24. 2026On State Reduction in Linear Attentioncited by 6
  25. 2026FLARE: Diffusion for Hybrid Language Modelcited by 6
  26. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  27. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deploymentcited by 5
  28. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  29. 2026Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Designcited by 5
  30. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 4
  31. 2026Expressivity-Efficiency Tradeoffs for Hybrid Sequence Modelscited by 4
  32. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 4
  33. 2026The Capability Convergence Hypothesis: Capability from Access Structure, Not Scalecited by 4
  34. 2025Scaling Reasoning without Attentioncited by 3
  35. 2025TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embeddingcited by 3
  36. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 3
  37. 2025HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learningcited by 3
  38. 2026VMonarch: Efficient Video Diffusion Transformers with Structured Attentioncited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models
  3. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models
  4. 2024Zamba: A Compact 7B SSM Hybrid Model
  5. 2022Efficiently Scaling Transformer Inference
  6. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners
  7. 2025Llama-Nemotron: Efficient Reasoning Models
  8. 2024Exploring the Limitations of Mamba in COPY and CoT Reasoning