# Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT

arXiv: https://arxiv.org/abs/2606.26861 · HTML: https://mlsanity.com/paper/2606.26861/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [Fluctuation-based Adaptive Structured Pruning for Large Language Models](https://mlsanity.com/paper/2312.11983/) (2023)
- [Efficient LLMs with AMP: Attention Heads and MLP Pruning](https://mlsanity.com/paper/2504.21174/) (2025)
- [VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning](https://mlsanity.com/paper/2406.05276/) (2024)
- [Progressive Knowledge-Guided Large Language Model Framework for Bearing Fault Diagnosis](https://mlsanity.com/paper/2606.16684/) (2026)

## Built on by (most cited first, 1 of 1)

- [Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models](https://mlsanity.com/paper/2610.15781/) (2026)

## Abstract (arXiv)

Deploying large language models (LLMs) on Industrial Internet of Things (IIoT) edge devices demands extreme compression, yet existing structured pruning methods collapse at high compression ratios due to one-shot importance estimation, and their cross-architecture behavior remains unpredictable. This article presents a cascaded multi-granularity pruning framework that removes layers, attention heads, and feed-forward channels in coarse-to-fine order, with lightweight low-rank recovery between stages to re-estimate component importance. An information-theoretic analysis motivates this ordering, and the Structural Independence Assumption (SIA) is formalized as a checkable condition predicting whether per-component pruning criteria are reliable for a given architecture: Multi-Head Attention (MHA)+GELU designs satisfy the SIA, whereas Grouped Query Attention (GQA)+SwiGLU designs violate it. On bearing fault diagnosis spanning 88M to 6.25B-parameter models, the framework extends achievable compression to 13.8 times on MHA+GELU architectures with 83.82% accuracy (+3.70 percentage points (pp) over the strongest baseline), while exposing a ~74pp accuracy collapse on GQA+SwiGLU architectures that violate the SIA. Deployed on an industrial slewing bearing fault diagnosis platform with NVIDIA DGX Spark, compressed models reduce inference latency by up to 67.2% and peak memory by 62.5%, demonstrating viability for IIoT edge inference.