# Heuristic Rank Selection with Progressively Searching Tensor Ring Network

arXiv: https://arxiv.org/abs/2009.10580 · HTML: https://mlsanity.com/paper/2009.10580/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 28 of 28)

- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks](https://mlsanity.com/paper/2301.09422/) (2023)
- [Reusing Pretrained Models by Multi-linear Operators for Efficient Training](https://mlsanity.com/paper/2310.10699/) (2023)
- [Permutation Search of Tensor Network Structures via Local Sampling](https://mlsanity.com/paper/2206.06597/) (2022)
- [Alternating Local Enumeration (TnALE): Solving Tensor Network Structure Search with Fewer Evaluations](https://mlsanity.com/paper/2304.12875/) (2023)
- [ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks](https://mlsanity.com/paper/2401.10341/) (2024)
- [BViT: Broad Attention based Vision Transformer](https://mlsanity.com/paper/2202.06268/) (2022)
- [Preparing Lessons for Progressive Training on Language Models](https://mlsanity.com/paper/2401.09192/) (2024)
- [TedNet: A Pytorch Toolkit for Tensor Decomposition Networks](https://mlsanity.com/paper/2104.05018/) (2021)
- [SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective](https://mlsanity.com/paper/2305.14912/) (2023)
- [A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks](https://mlsanity.com/paper/2205.15307/) (2022)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs)](https://mlsanity.com/paper/2402.02456/) (2024)
- [IDInit: A Universal and Stable Initialization Method for Neural Network Training](https://mlsanity.com/paper/2503.04626/) (2025)
- [ABCP: Automatic Block-wise and Channel-wise Network Pruning via Joint Search](https://mlsanity.com/paper/2110.03858/) (2021)
- [CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial Robustness](https://mlsanity.com/paper/2212.01957/) (2022)
- [Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search](https://mlsanity.com/paper/2111.07722/) (2021)
- [Multiscale Tensor Summation Factorization as a New Neural Network Layer (MTS Layer) for Multidimensional Data Processing](https://mlsanity.com/paper/2504.13975/) (2025)
- [A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression](https://mlsanity.com/paper/2309.13077/) (2023)
- [An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compression](https://mlsanity.com/paper/2408.01534/) (2024)
- [Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning](https://mlsanity.com/paper/2401.08014/) (2024)
- [Reduced storage direct tensor ring decomposition for convolutional neural networks compression](https://mlsanity.com/paper/2405.10802/) (2024)
- [Convolutional Neural Network Compression Based on Low-Rank Decomposition](https://mlsanity.com/paper/2408.16289/) (2024)
- [Unified Framework for Pre-trained Neural Network Compression via Decomposition and Optimized Rank Selection](https://mlsanity.com/paper/2409.03555/) (2024)
- [FGFP: A Fractional Gaussian Filter and Pruning for Deep Neural Networks Compression](https://mlsanity.com/paper/2507.22527/) (2025)
- [Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Models](https://mlsanity.com/paper/2508.10435/) (2025)
- [Tensor Decomposition Structure Search Framework from an Interaction Perspective](https://mlsanity.com/paper/2603.02720/) (2026)

## Abstract (arXiv)

Recently, Tensor Ring Networks (TRNs) have been applied in deep networks, achieving remarkable successes in compression ratio and accuracy. Although highly related to the performance of TRNs, rank selection is seldom studied in previous works and usually set to equal in experiments. Meanwhile, there is not any heuristic method to choose the rank, and an enumerating way to find appropriate rank is extremely time-consuming. Interestingly, we discover that part of the rank elements is sensitive and usually aggregate in a narrow region, namely an interest region. Therefore, based on the above phenomenon, we propose a novel progressive genetic algorithm named Progressively Searching Tensor Ring Network Search (PSTRN), which has the ability to find optimal rank precisely and efficiently. Through the evolutionary phase and progressive phase, PSTRN can converge to the interest region quickly and harvest good performance. Experimental results show that PSTRN can significantly reduce the complexity of seeking rank, compared with the enumerating method. Furthermore, our method is validated on public benchmarks like MNIST, CIFAR10/100, UCF11 and HMDB51, achieving the state-of-the-art performance.