mlsanityData to 14 Oct 2026

Liger Kernel: Efficient Triton Kernels for LLM Training

arXiv 2410.10989 ↗First posted 14 Oct 2024cs.LG cs.AI cs.CLcited by 130

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

18 of 28 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2016Layer Normalizationcited by 5,669
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2023Mistral 7Bcited by 3,164
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
6 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  6. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 68; 2026: 55.

130 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  3. 2024OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviewscited by 65
  4. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  5. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  6. 2025BitNet b1.58 2B4T Technical Reportcited by 38
  7. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 37
  8. 2025FlashThink: An Early Exit Method For Efficient Reasoningcited by 28
  9. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  10. 2025RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illuminationcited by 23
  11. 2025SEA-LION: Southeast Asian Languages in One Networkcited by 22
  12. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
38 more
  1. 2025SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Modelscited by 20
  2. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  3. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 20
  4. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  5. 2025LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Trainingcited by 18
  6. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  7. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  8. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  9. 2026Thinking in Streaming Videocited by 16
  10. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 14
  11. 2025Krikri: Advancing Open Large Language Models for Greekcited by 14
  12. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 14
  13. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 13
  14. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 12
  15. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  16. 2025Rethinking the Generation of High-Quality CoT Data from the Perspective of LLM-Adaptive Question Difficulty Gradingcited by 11
  17. 2025SciReasoner: Laying the Scientific Reasoning Ground Across Disciplinescited by 11
  18. 2026Towards Automated Kernel Generation in the Era of LLMscited by 11
  19. 2026Arcee Trinity Large Technical Reportcited by 11
  20. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Expertscited by 10
  21. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
  22. 2024Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselinescited by 9
  23. 2025Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging -- An Open Recipecited by 9
  24. 2025TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendationcited by 9
  25. 2025On The Fragility of Benchmark Contamination Detection in Reasoning Modelscited by 9
  26. 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 9
  27. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 9
  28. 2026EuroLLM-22B: Technical Reportcited by 9
  29. 2024YuLan-Mini: An Open Data-efficient Language Modelcited by 7
  30. 2025Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoningcited by 7
  31. 2025MD-LLM-1: A Large Language Model for Molecular Dynamicscited by 7
  32. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
  33. 2025VARCO-VISION-2.0 Technical Reportcited by 6
  34. 2025Training Large Language Models To Reason In Parallel With Global Forking Tokenscited by 6
  35. 2025SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Spacecited by 6
  36. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  37. 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 6
  38. 2025Pensez: Less Data, Better Reasoning -- Rethinking French LLMcited by 5

50 most cited of 130 All 130 (JSON)