mlsanityData to 14 Oct 2026

QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation

arXiv 2406.00132 ↗First posted 31 May 2024cs.LG quant-phcited by 23

Not on the map. Why?

Abstract (arXiv)

We propose Quantum-informed Tensor Adaptation (QuanTA), a novel, easy-to-implement, fine-tuning method with no inference overhead for large-scale pre-trained language models. By leveraging quantum-inspired methods derived from quantum circuit structures, QuanTA enables efficient high-rank fine-tuning, surpassing the limitations of Low-Rank Adaptation (LoRA)--low-rank approximation may fail for complicated downstream tasks. Our approach is theoretically supported by the universality theorem and the rank representation theorem to achieve efficient high-rank adaptations. Experiments demonstrate that QuanTA significantly enhances commonsense reasoning, arithmetic reasoning, and scalability compared to traditional methods. Furthermore, QuanTA shows superior performance with fewer trainable parameters compared to other approaches and can be designed to integrate with existing fine-tuning algorithms for further improvement, providing a scalable and efficient solution for fine-tuning large language models and advancing state-of-the-art in natural language processing.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

42 of 63 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
30 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  12. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  13. 2019Hamiltonian Neural Networkscited by 712
  14. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  15. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  16. 2020Lagrangian Neural Networkscited by 398
  17. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  18. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  19. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  20. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  21. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  22. 2018Differentiable Learning of Quantum Circuit Born Machinecited by 93
  23. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  24. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  25. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  26. 2020Gauge equivariant neural networks for quantum lattice gauge theoriescited by 28
  27. 2023GenPhys: From Physical Processes to Generative Modelscited by 19
  28. 2021Gauge Invariant and Anyonic Symmetric Transformer and RNN Quantum States for Quantum Lattice Modelscited by 18
  29. 2022Simulating 2+1D Lattice Quantum Electrodynamics at Finite Density with Neural Flow Wavefunctionscited by 10
  30. 2023ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 11
  2. 2025L2M: Mutual Information Scaling Law for Long-Context Language Modelingcited by 9
  3. 2025Quantum-Enhanced LLM Efficient Fine Tuningcited by 6
  4. 2024PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuningcited by 5
  5. 2025QuIC: Quantum-Inspired Compound Adapters for Parameter Efficient Fine-Tuningcited by 5
  6. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
  7. 2024DoTA: Weight-Decomposed Tensor Adaptation for Large Language Modelscited by 3
  8. 2025Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Modelscited by 3
  9. 2025Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimescited by 3
  10. 2025TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Modelscited by 2
  11. 2025You Only Measure Once: On Designing Single-Shot Quantum Machine Learning Modelscited by 2
  12. 2026S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domaincited by 1
11 more
  1. 2024Quantum Implicit Neural Compressioncited by 0
  2. 2025Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networkscited by 0
  3. 2025PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMscited by 0
  4. 2025Tensor Computing Interface: An Application-Oriented, Lightweight Interface for Portable High-Performance Tensor Network Applicationscited by 0
  5. 2026On the Approximation Complexity of Matrix Product Operator (MPO) Born Machinescited by 0
  6. 2026FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioningcited by 0
  7. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
  8. 2026MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuningcited by 0
  9. 2026Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attentioncited by 0
  10. 2026QuPID: Quantum Parameter-Efficient Input-Dependent Retrieval Adaptation for Medical RAGcited by 0
  11. 2026QuanVI: Score-based Variational Inference via Quantum Maximally Mixed Statescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fine-Tuning Large Language Models for Quantum Reasoning
  2. 2025Agent-Q: Fine-Tuning Large Language Models for Quantum Circuit Generation and Optimization
  3. 2024Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance
  4. 2025QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models
  5. 2024Low-Rank Quantization-Aware Training for LLMs
  6. 2025Superpositional Gradient Descent: Harnessing Quantum Principles for Model Training
  7. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
  8. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation