# QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation

arXiv: https://arxiv.org/abs/2406.00132 · HTML: https://mlsanity.com/paper/2406.00132/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 63 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)

## Built on by (most cited first, 23 of 23)

- [Quantum-PEFT: Ultra parameter-efficient fine-tuning](https://mlsanity.com/paper/2503.05431/) (2025)
- [L2M: Mutual Information Scaling Law for Long-Context Language Modeling](https://mlsanity.com/paper/2503.04725/) (2025)
- [Quantum-Enhanced LLM Efficient Fine Tuning](https://mlsanity.com/paper/2503.12790/) (2025)
- [PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning](https://mlsanity.com/paper/2409.16722/) (2024)
- [QuIC: Quantum-Inspired Compound Adapters for Parameter Efficient Fine-Tuning](https://mlsanity.com/paper/2502.06916/) (2025)
- [MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2506.09105/) (2025)
- [DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models](https://mlsanity.com/paper/2412.20891/) (2024)
- [Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models](https://mlsanity.com/paper/2501.08727/) (2025)
- [Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes](https://mlsanity.com/paper/2507.00327/) (2025)
- [TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2509.03234/) (2025)
- [You Only Measure Once: On Designing Single-Shot Quantum Machine Learning Models](https://mlsanity.com/paper/2509.20090/) (2025)
- [S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain](https://mlsanity.com/paper/2605.08589/) (2026)
- [Quantum Implicit Neural Compression](https://mlsanity.com/paper/2412.19828/) (2024)
- [Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networks](https://mlsanity.com/paper/2507.18112/) (2025)
- [PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs](https://mlsanity.com/paper/2509.20105/) (2025)
- [Tensor Computing Interface: An Application-Oriented, Lightweight Interface for Portable High-Performance Tensor Network Applications](https://mlsanity.com/paper/2512.23917/) (2025)
- [On the Approximation Complexity of Matrix Product Operator (MPO) Born Machines](https://mlsanity.com/paper/2605.11471/) (2026)
- [FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning](https://mlsanity.com/paper/2605.22869/) (2026)
- [EinSort: Sorting is All We Need for Tensorizing LLM](https://mlsanity.com/paper/2606.08565/) (2026)
- [MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning](https://mlsanity.com/paper/2607.12418/) (2026)
- [Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention](https://mlsanity.com/paper/2608.17288/) (2026)
- [QuPID: Quantum Parameter-Efficient Input-Dependent Retrieval Adaptation for Medical RAG](https://mlsanity.com/paper/2609.33351/) (2026)
- [QuanVI: Score-based Variational Inference via Quantum Maximally Mixed States](https://mlsanity.com/paper/2609.39164/) (2026)

## Abstract (arXiv)

We propose Quantum-informed Tensor Adaptation (QuanTA), a novel, easy-to-implement, fine-tuning method with no inference overhead for large-scale pre-trained language models. By leveraging quantum-inspired methods derived from quantum circuit structures, QuanTA enables efficient high-rank fine-tuning, surpassing the limitations of Low-Rank Adaptation (LoRA)--low-rank approximation may fail for complicated downstream tasks. Our approach is theoretically supported by the universality theorem and the rank representation theorem to achieve efficient high-rank adaptations. Experiments demonstrate that QuanTA significantly enhances commonsense reasoning, arithmetic reasoning, and scalability compared to traditional methods. Furthermore, QuanTA shows superior performance with fewer trainable parameters compared to other approaches and can be designed to integrate with existing fine-tuning algorithms for further improvement, providing a scalable and efficient solution for fine-tuning large language models and advancing state-of-the-art in natural language processing.