# Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

arXiv: https://arxiv.org/abs/2409.15654 · HTML: https://mlsanity.com/paper/2409.15654/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2024.

## Built on by (most cited first, 18 of 18)

- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memory](https://mlsanity.com/paper/2606.25115/) (2026)
- [Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference](https://mlsanity.com/paper/2510.05497/) (2025)
- [Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing](https://mlsanity.com/paper/2510.16040/) (2025)
- [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026)
- [SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution](https://mlsanity.com/paper/2508.18983/) (2025)
- [FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference](https://mlsanity.com/paper/2607.10186/) (2026)
- [Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.20507/) (2025)
- [KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing](https://mlsanity.com/paper/2512.03608/) (2025)
- [TOM: A Ternary Read-only Memory Accelerator for LLM-powered Edge Intelligence](https://mlsanity.com/paper/2602.20662/) (2026)
- [Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures](https://mlsanity.com/paper/2603.07006/) (2026)
- [LLM Inference in a Flash!](https://mlsanity.com/paper/2609.16161/) (2026)
- [Mozart: A Chiplet Ecosystem-Accelerator Codesign Framework for Composable Bespoke Application Specific Integrated Circuits](https://mlsanity.com/paper/2510.08873/) (2025)
- [PICNIC: Silicon Photonic Interconnected Chiplets with Computational Network and In-memory Computing for LLM Inference Acceleration](https://mlsanity.com/paper/2511.04036/) (2025)
- [NASiC: 3D NAND-based CAM-Selected Multibit CIM Architecture for Efficient On-Device Mixture-of-Experts LLM Inference](https://mlsanity.com/paper/2605.23294/) (2026)
- [Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling](https://mlsanity.com/paper/2607.25498/) (2026)
- [Hardware Design and Security in the Era of Chiplets and LLMs](https://mlsanity.com/paper/2608.05063/) (2026)
- [AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing](https://mlsanity.com/paper/2610.00465/) (2026)

## Abstract (arXiv)

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses the significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash's on-die processing capabilities. Overall, Cambricon-LLM enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, and 7B LLMs at a speed of 36.34 token/s, which is over 22X to 45X faster than existing flash-offloading technologies, showing the potentiality of deploying powerful LLMs in edge devices.