mlsanityData to 14 Oct 2026

Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search

arXiv 2508.15884 ↗First posted 21 Aug 2025cs.CL cs.AI cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

We present Jet-Nemotron, a new family of hybrid-architecture language models, which matches or exceeds the accuracy of leading full-attention models while significantly improving generation throughput. Jet-Nemotron is developed using Post Neural Architecture Search (PostNAS), a novel neural architecture exploration pipeline that enables efficient model design. Unlike prior approaches, PostNAS begins with a pre-trained full-attention model and freezes its MLP weights, allowing efficient exploration of attention block designs. The pipeline includes four key components: (1) learning optimal full-attention layer placement and elimination, (2) linear attention block selection, (3) designing new attention blocks, and (4) performing hardware-aware hyperparameter search. Our Jet-Nemotron-2B model achieves comparable or superior accuracy to Qwen3, Qwen2.5, Gemma3, and Llama3.2 across a comprehensive suite of benchmarks while delivering up to 53.6x generation throughput speedup and 6.1x prefilling speedup. It also achieves higher accuracy on MMLU and MMLU-Pro than recent advanced MoE full-attention models, such as DeepSeek-V3-Small and Moonlight, despite their larger scale with 15B total and 2.2B activated parameters.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 91 of its 106 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

91 of 106 references.

References · 91

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
38 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2025Gemma 3 Technical Reportcited by 1,824
  13. 2020Reformer: The Efficient Transformercited by 1,668
  14. 2020Rethinking Attention with Performerscited by 1,601
  15. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  16. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  17. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  18. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  19. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  20. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  21. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  22. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  23. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  24. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  25. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  26. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  27. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  28. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  29. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  30. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  31. 2025Muon is Scalable for LLM Trainingcited by 428
  32. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  33. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  34. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  35. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  36. 2024Titans: Learning to Memorize at Test Timecited by 328
  37. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  38. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 29.

41 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  2. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  3. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 20
  4. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 18
  5. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 16
  6. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  7. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  8. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  9. 2025DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacecited by 8
  10. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  11. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  12. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5
29 more
  1. 2026MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modelingcited by 5
  2. 2026Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Designcited by 5
  3. 2026Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attentioncited by 5
  4. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 4
  5. 2025What Limits Agentic Systems Efficiency?cited by 3
  6. 2026Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Constructioncited by 3
  7. 2026When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Modelscited by 3
  8. 2025Apriel-H1: Towards Efficient Enterprise Reasoning Modelscited by 2
  9. 2026AllMem: A Memory-centric Recipe for Efficient Long-context Modelingcited by 2
  10. 2026SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inferencecited by 2
  11. 2026Morphing into Hybrid Attention Modelscited by 2
  12. 2026MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deploymentcited by 1
  13. 2026When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Modelscited by 1
  14. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Modelscited by 0
  15. 2026LayerBoost: Layer-Aware Attention Reduction for Efficient LLMscited by 0
  16. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scalingcited by 0
  17. 2026HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillationcited by 0
  18. 2026LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Modelscited by 0
  19. 2026DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPUcited by 0
  20. 2026Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencecited by 0
  21. 2026JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Searchcited by 0
  22. 2026Dynamic Short Convolutions Improve Transformerscited by 0
  23. 2026Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix Itcited by 0
  24. 2026Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoningcited by 0
  25. 2026Agentic evolution of physically constrained foundation modelscited by 0
  26. 2026Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPEcited by 0
  27. 2026Multilinguality in Hybrid Attention LLMscited by 0
  28. 2026The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendscited by 0
  29. 2026GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding
  2. 2026Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context
  3. 2024Compressing Large Language Models with Automated Sub-Network Search
  4. 2024LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models
  5. 2025Elastic Architecture Search for Efficient Language Models
  6. 2025FFN Fusion: Rethinking Sequential Computation in Large Language Models
  7. 2026Rethinking the Role of Efficient Attention in Hybrid Architectures
  8. 2022AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models