# Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression

arXiv: https://arxiv.org/abs/2610.10085 · HTML: https://mlsanity.com/paper/2610.10085/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 6 of 6)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Tensor Ring Decomposition](https://mlsanity.com/paper/1606.05535/) (2016)
- [CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks](https://mlsanity.com/paper/2401.14109/) (2024)
- [Scaling Laws for Energy Efficiency of Local LLMs](https://mlsanity.com/paper/2512.16531/) (2025)
- [Accuracy and Consumption analysis from a compressed model by CompactifAI from Multiverse Computing](https://mlsanity.com/paper/2507.08836/) (2025)
- [A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models](https://mlsanity.com/paper/2602.01613/) (2026)

## Abstract (arXiv)

Post-training tensor-network compression replaces Transformer linear layers with Tensor Train (TT) or Tree Tensor Network (TTN) operators, but standard decompositions minimize weight-space Frobenius error rather than functional error under the layer's activation distribution. We propose Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that preconditions each weight matrix with a diagonal activation-derived scale before an unchanged TT/TTN solver and deploys the result with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression: under single-operator replacement, AWT closes 12-35% of the WikiText perplexity gap to the dense baseline across the three model families and 2-6 times compression settings; while under multi-operator Llama suffix replacement it closes 27-60% across attention-group and all-seven-matrix settings. The gains also transfer to downstream HellaSwag and ARC-Challenge evaluations. We further show that diagonal preconditioning is a robustness-modularity tradeoff rather than a diagonal-covariance assumption: a dense full-covariance oracle wins its own weighted objective in 80/81 cases, yet diagonal AWT gives better held-out functional fidelity in 53/81 cases. Together, these results position AWT as a principled, modular preconditioner for improving functional fidelity in fixed TT/TTN compression pipelines without modifying the decomposition solver.