# Bayesian Tensorized Neural Networks with Automatic Rank Selection

arXiv: https://arxiv.org/abs/1905.10478 · HTML: https://mlsanity.com/paper/1905.10478/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 26 of 26)

- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Tensor networks for interpretable and efficient quantum-inspired machine learning](https://mlsanity.com/paper/2311.11258/) (2023)
- [CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization](https://mlsanity.com/paper/2405.14377/) (2024)
- [Permutation Search of Tensor Network Structures via Local Sampling](https://mlsanity.com/paper/2206.06597/) (2022)
- [Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling](https://mlsanity.com/paper/2205.14283/) (2022)
- [Alternating Local Enumeration (TnALE): Solving Tensor Network Structure Search with Fewer Evaluations](https://mlsanity.com/paper/2304.12875/) (2023)
- [Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networks](https://mlsanity.com/paper/2308.09858/) (2023)
- [LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2402.11417/) (2024)
- [TT-PINN: A Tensor-Compressed Neural PDE Solver for Edge Computing](https://mlsanity.com/paper/2207.01751/) (2022)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [Block-Term Tensor Decomposition Model Selection and Computation: The Bayesian Way](https://mlsanity.com/paper/2101.02931/) (2021)
- [Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks](https://mlsanity.com/paper/2505.20132/) (2025)
- [3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration](https://mlsanity.com/paper/2105.06250/) (2021)
- [Tensor Yard: One-Shot Algorithm of Hardware-Friendly Tensor-Train Decomposition for Convolutional Neural Networks](https://mlsanity.com/paper/2108.04029/) (2021)
- [Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization](https://mlsanity.com/paper/2501.06663/) (2025)
- [Low-Rank+Sparse Tensor Compression for Neural Networks](https://mlsanity.com/paper/2111.01697/) (2021)
- [Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding](https://mlsanity.com/paper/2306.01076/) (2023)
- [Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks](https://mlsanity.com/paper/2502.12384/) (2025)
- [Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models](https://mlsanity.com/paper/2505.14871/) (2025)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [Tensorized Hypergraph Neural Networks](https://mlsanity.com/paper/2306.02560/) (2023)
- [Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability](https://mlsanity.com/paper/2608.30505/) (2026)
- [Tensor Shape Search for Optimum Data Compression](https://mlsanity.com/paper/2205.10651/) (2022)
- [Comprehensive Design Space Exploration for Tensorized Neural Network Hardware Accelerators](https://mlsanity.com/paper/2511.17971/) (2025)
- [Gaussian Universality and Its Breakdown in Tensor-Network Machine Learning](https://mlsanity.com/paper/2610.06080/) (2026)

## Abstract (arXiv)

Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to achieve this goal, this paper proposes a Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. We provide experimental validation on a fully connected neural network, a CNN and a residual neural network where our work produces 7.4× to 137× more compact neural networks directly from the training.