# Liger Kernel: Efficient Triton Kernels for LLM Training

arXiv: https://arxiv.org/abs/2410.10989 · HTML: https://mlsanity.com/paper/2410.10989/

## Key facts

- Cited by 130 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [ZeRO++: Extremely Efficient Collective Communication for Giant Model Training](https://mlsanity.com/paper/2306.10209/) (2023)

## Built on by (most cited first, 30 of 130)

- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [Helios: Real Real-Time Long Video Generation Model](https://mlsanity.com/paper/2603.04379/) (2026)
- [Intern-S1: A Scientific Multimodal Foundation Model](https://mlsanity.com/paper/2508.15763/) (2025)
- [BitNet b1.58 2B4T Technical Report](https://mlsanity.com/paper/2504.12285/) (2025)
- [Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://mlsanity.com/paper/2502.17407/) (2025)
- [FlashThink: An Early Exit Method For Efficient Reasoning](https://mlsanity.com/paper/2505.13949/) (2025)
- [Latent Thought Models with Variational Bayes Inference-Time Computation](https://mlsanity.com/paper/2502.01567/) (2025)
- [RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination](https://mlsanity.com/paper/2505.21925/) (2025)
- [SEA-LION: Southeast Asian Languages in One Network](https://mlsanity.com/paper/2504.05747/) (2025)
- [LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification](https://mlsanity.com/paper/2502.17421/) (2025)
- [SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models](https://mlsanity.com/paper/2502.09604/) (2025)
- [VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo](https://mlsanity.com/paper/2508.02317/) (2025)
- [Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation](https://mlsanity.com/paper/2605.16826/) (2026)
- [Docopilot: Improving Multimodal Models for Document-Level Understanding](https://mlsanity.com/paper/2507.14675/) (2025)
- [LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training](https://mlsanity.com/paper/2510.14969/) (2025)
- [Cross-lingual Transfer of Reward Models in Multilingual Alignment](https://mlsanity.com/paper/2410.18027/) (2024)
- [TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling](https://mlsanity.com/paper/2504.07053/) (2025)
- [QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs](https://mlsanity.com/paper/2510.11696/) (2025)
- [Thinking in Streaming Video](https://mlsanity.com/paper/2603.12938/) (2026)
- [TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action](https://mlsanity.com/paper/2505.01583/) (2025)
- [Krikri: Advancing Open Large Language Models for Greek](https://mlsanity.com/paper/2505.13772/) (2025)
- [NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation](https://mlsanity.com/paper/2601.02204/) (2026)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [GoRA: Gradient-driven Adaptive Low Rank Adaptation](https://mlsanity.com/paper/2502.12171/) (2025)
- [AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages](https://mlsanity.com/paper/2601.06395/) (2026)
- [Rethinking the Generation of High-Quality CoT Data from the Perspective of LLM-Adaptive Question Difficulty Grading](https://mlsanity.com/paper/2504.11919/) (2025)
- [SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines](https://mlsanity.com/paper/2509.21320/) (2025)
- [Towards Automated Kernel Generation in the Era of LLMs](https://mlsanity.com/paper/2601.15727/) (2026)

## Abstract (arXiv)

Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.