# ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques

arXiv: https://arxiv.org/abs/2507.11506 · HTML: https://mlsanity.com/paper/2507.11506/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 74 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism](https://mlsanity.com/paper/2211.13878/) (2022)
- [Dissecting the Graphcore IPU Architecture via Microbenchmarking](https://mlsanity.com/paper/1912.03413/) (2019)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)
- [TensorIR: An Abstraction for Automatic Tensorized Program Optimization](https://mlsanity.com/paper/2207.04296/) (2022)
- [Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models](https://mlsanity.com/paper/2110.04478/) (2021)
- [AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness](https://mlsanity.com/paper/2210.07297/) (2022)
- [G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations](https://mlsanity.com/paper/2310.09443/) (2023)
- [Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing](https://mlsanity.com/paper/2404.16914/) (2024)
- [LLM-Aided Compilation for Tensor Accelerators](https://mlsanity.com/paper/2408.03408/) (2024)
- [Optimal Kernel Orchestration for Tensor Programs with Korch](https://mlsanity.com/paper/2406.09465/) (2024)
- [Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10](https://mlsanity.com/paper/2408.04808/) (2024)
- [SoMa: Identifying, Exploring, and Understanding the DRAM Communication Scheduling Space for DNN Accelerators](https://mlsanity.com/paper/2501.12634/) (2025)
- [Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms](https://mlsanity.com/paper/2408.04104/) (2024)
- [Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units](https://mlsanity.com/paper/2506.11446/) (2025)

## Abstract (arXiv)

To meet the increasing demand of deep learning (DL) models, AI chips are employing both off-chip memory (e.g., HBM) and high-bandwidth low-latency interconnect for direct inter-core data exchange. However, it is not easy to explore the efficiency of these inter-core connected AI (ICCA) chips, due to a fundamental tussle among compute (per-core execution), communication (inter-core data exchange), and I/O (off-chip data access). In this paper, we develop Elk, a DL compiler framework to maximize the efficiency of ICCA chips by jointly trading off all the three performance factors discussed above. Elk structures these performance factors into configurable parameters and forms a global trade-off space in the DL compiler. To systematically explore this space and maximize overall efficiency, Elk employs a new inductive operator scheduling policy and a cost-aware on-chip memory allocation algorithm. It generates globally optimized execution plans that best overlap off-chip data loading and on-chip execution. To examine the efficiency of Elk, we build a full-fledged emulator based on a real ICCA chip IPU-POD4, and an ICCA chip simulator for sensitivity analysis with different interconnect network topologies. Elk achieves 94% of the ideal roofline performance of ICCA chips on average, showing the benefits of supporting large DL models on ICCA chips. We also show Elk's capability of enabling architecture design space exploration for new ICCA chip development.