# Nemotron-4 15B Technical Report

arXiv: https://arxiv.org/abs/2402.16819 · HTML: https://mlsanity.com/paper/2402.16819/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)

## Built on by (most cited first, 30 of 36)

- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [An Empirical Study of Mamba-based Language Models](https://mlsanity.com/paper/2406.07887/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://mlsanity.com/paper/2504.03624/) (2025)
- [MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models](https://mlsanity.com/paper/2409.17481/) (2024)
- [Upcycling Large Language Models into Mixture of Experts](https://mlsanity.com/paper/2410.07524/) (2024)
- [Flextron: Many-in-One Flexible Large Language Model](https://mlsanity.com/paper/2406.10260/) (2024)
- [SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain](https://mlsanity.com/paper/2407.19584/) (2024)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Mid-Training of Large Language Models: A Survey](https://mlsanity.com/paper/2510.06826/) (2025)
- [ESPACE: Dimensionality Reduction of Activations for Model Compression](https://mlsanity.com/paper/2410.05437/) (2024)
- [MIND: Math Informed syNthetic Dialogues for Pretraining LLMs](https://mlsanity.com/paper/2410.12881/) (2024)
- [Zyda-2: a 5 Trillion Token High-Quality Dataset](https://mlsanity.com/paper/2411.06068/) (2024)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [Data, Data Everywhere: A Guide for Pretraining Dataset Construction](https://mlsanity.com/paper/2407.06380/) (2024)
- [Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding](https://mlsanity.com/paper/2406.12331/) (2024)
- [Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning](https://mlsanity.com/paper/2412.08587/) (2024)
- [LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inference](https://mlsanity.com/paper/2502.05376/) (2025)
- [PARAM-1 BharatGen 2.9B Model](https://mlsanity.com/paper/2507.13390/) (2025)
- [Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws](https://mlsanity.com/paper/2602.14208/) (2026)
- [7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement](https://mlsanity.com/paper/2412.06845/) (2024)
- [Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models](https://mlsanity.com/paper/2507.04023/) (2025)
- [Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods](https://mlsanity.com/paper/2406.13936/) (2024)
- [Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism](https://mlsanity.com/paper/2412.21124/) (2024)
- [PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing](https://mlsanity.com/paper/2503.12167/) (2025)
- [Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language](https://mlsanity.com/paper/2603.11881/) (2026)
- [Membership Inference Attacks on Tokenizers of Large Language Models](https://mlsanity.com/paper/2510.05699/) (2025)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)

## Abstract (arXiv)

We introduce Nemotron-4 15B, a 15-billion-parameter large multilingual language model trained on 8 trillion text tokens. Nemotron-4 15B demonstrates strong performance when assessed on English, multilingual, and coding tasks: it outperforms all existing similarly-sized open models on 4 out of 7 downstream evaluation areas and achieves competitive performance to the leading open models in the remaining ones. Specifically, Nemotron-4 15B exhibits the best multilingual capabilities of all similarly-sized models, even outperforming models over four times larger and those explicitly specialized for multilingual tasks.