mlsanityData to 13 Oct 2026

Mirage: A Multi-Level Superoptimizer for Tensor Programs

arXiv 2405.05751 ↗First posted 9 May 2024cs.LG cs.AI cs.PLcited by 41

Not on the map. Why?

Abstract (arXiv)

We introduce Mirage, the first multi-level superoptimizer for tensor programs. A key idea in Mirage is μGraphs, a uniform representation of tensor programs at the kernel, thread block, and thread levels of the GPU compute hierarchy. μGraphs enable Mirage to discover novel optimizations that combine algebraic transformations, schedule transformations, and generation of new custom kernels. To navigate the large search space, Mirage introduces a pruning technique based on abstraction that significantly reduces the search space and provides a certain optimality guarantee. To ensure that the optimized μGraph is equivalent to the input program, Mirage introduces a probabilistic equivalence verification procedure with strong theoretical guarantees. Our evaluation shows that Mirage outperforms existing approaches by up to 3.3× even for DNNs that are widely used and heavily optimized. Mirage is publicly available at https://github.com/mirage-project/mirage.

Key facts. Cited by 41 papers in the map (data through 13 Oct 2026). Builds on 10 of its 51 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

10 of 51 references.

References · 10

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  3. 2019Root Mean Square Layer Normalizationcited by 1,449
  4. 2014cuDNN: Efficient Primitives for Deep Learningcited by 243
  5. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
  6. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  7. 2018Learning to Optimize Tensor Programscited by 139
  8. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 79
  9. 2022TensorIR: An Abstraction for Automatic Tensorized Program Optimizationcited by 38
  10. 2021Equality Saturation for Tensor Graph Superoptimizationcited by 27

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 20; 2026: 21.

41 papers.

Sort:

By area:

  1. 2025AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimizationcited by 22
  2. 2025The FM Agentcited by 21
  3. 2025MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programscited by 15
  4. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  5. 2025SuperCoder: Assembly Program Superoptimization with Large Language Modelscited by 6
  6. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  7. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 5
  8. 2026CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programscited by 5
  9. 2025Neptune: Advanced ML Operator Fusion for Locality and Parallelism on GPUscited by 4
  10. 2025PyGraph: Robust Compiler Support for CUDA Graphs in PyTorchcited by 3
  11. 2025FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inferencecited by 3
  12. 2025LoRAFusion: Efficient LoRA Fine-Tuning for LLMscited by 3
29 more
  1. 2025GPU Kernel Optimization Beyond Full Builds: An LLM Framework with Minimal Executable Programscited by 3
  2. 2026A Two-Stage GPU Kernel Tuner Combining Semantic Refactoring and Search-Based Optimizationcited by 3
  3. 2026AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimizationcited by 3
  4. 2025Blockbuster, Part 1: Block-level AI Operator Fusioncited by 1
  5. 2025TTrace: Lightweight Error Checking and Diagnosis for Distributed Trainingcited by 1
  6. 2025VeriLocc: End-to-End Cross-Architecture Register Allocation via LLMcited by 1
  7. 2025MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUscited by 1
  8. 2025Evolution of Kernels: Automated RISC-V Kernel Optimization with Large Language Modelscited by 1
  9. 2026Nautilus: An Auto-Scheduling Tensor Compiler for Efficient Tiled GPU Kernelscited by 1
  10. 2026Prism: Symbolic Superoptimization of Tensor Programscited by 1
  11. 2026AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelismcited by 1
  12. 2026Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inferencecited by 1
  13. 2025Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelizationcited by 0
  14. 2025Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inferencecited by 0
  15. 2025Verifying Computational Graphs in Production-Grade Distributed Machine Learning Frameworkscited by 0
  16. 2025Flashlight: PyTorch Compiler Extensions to Accelerate Attention Variantscited by 0
  17. 2026A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Networkcited by 0
  18. 2026VTC: DNN Compilation with Virtual Tensors for Data Movement Eliminationcited by 0
  19. 2026Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariantscited by 0
  20. 2026SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Servingcited by 0
  21. 2026Axon: A Synthesizing Superoptimizer for Tensor Programscited by 0
  22. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  23. 2026CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systemscited by 0
  24. 2026HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimizationcited by 0
  25. 2026Tensor Seeks Layout: Formalizing Layout Selection for ML Compilerscited by 0
  26. 2026BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inferencecited by 0
  27. 2026KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel Optimizationcited by 0
  28. 2026AI as a Compiler: Compiling Triton kernels without the Triton compilercited by 0
  29. 2026Cleave: Scaling Tensor Program Optimization via Decoupled Algebraic Search and Operator Schedulingcited by 0