mlsanityData to 7 Oct 2026

Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces

arXiv 2305.14516 ↗First posted 23 May 2023cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Benchmarking and co-design are essential for driving optimizations and innovation around ML models, ML software, and next-generation hardware. Full workload benchmarks, e.g. MLPerf, play an essential role in enabling fair comparison across different software and hardware stacks especially once systems are fully designed and deployed. However, the pace of AI innovation demands a more agile methodology to benchmark creation and usage by simulators and emulators for future system co-design. We propose Chakra, an open graph schema for standardizing workload specification capturing key operations and dependencies, also known as Execution Trace (ET). In addition, we propose a complementary set of tools/capabilities to enable collection, generation, and adoption of Chakra ETs by a wide range of simulators, emulators, and benchmarks. For instance, we use generative AI models to learn latent statistical properties across thousands of Chakra ETs and use these models to synthesize Chakra ETs. These synthetic ETs can obfuscate key proprietary information and also target future what-if scenarios. As an example, we demonstrate an end-to-end proof-of-concept that converts PyTorch ETs to Chakra ETs and uses this to drive an open-source training system simulator (ASTRA-sim). Our end-goal is to build a vibrant industry-wide ecosystem of agile benchmarks and tools to drive future AI system co-design.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 9 of its 17 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

9 of 17 references.

References · 9

  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  3. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  4. 2022AutoShard: Automated Embedding Table Sharding for Recommender Systemscited by 17
  5. 2022DreamShard: Generalizable Embedding Table Placement for Recommender Systemscited by 14
  6. 2021Automap: Towards Ergonomic Automated Parallelism for ML Modelscited by 12
  7. 2022Building a Performance Model for Deep Learning Recommendation Model Training on GPUscited by 10
  8. 2022Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarkscited by 5
  9. 2021DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distributioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  2. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  3. 2024LayerDAG: A Layerwise Autoregressive Diffusion Model for Directed Acyclic Graph Generationcited by 8
  4. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  5. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  6. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  7. 2025Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learningcited by 3
  8. 2024Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platformscited by 1
  9. 2025Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphscited by 1
  10. 2025Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimationcited by 0
  11. 2026MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Tracescited by 0
  12. 2026A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLMcited by 0
1 more
  1. 2026ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modelingcited by 0