mlsanityData to 7 Oct 2026

TRAWL: Tensor Reduced and Approximated Weights for Large Language Models

arXiv 2406.17261 ↗First posted 25 Jun 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent research has shown that pruning large-scale language models for inference is an effective approach to improving model efficiency, significantly reducing model weights with minimal impact on performance. Interestingly, pruning can sometimes even enhance accuracy by removing noise that accumulates during training, particularly through matrix decompositions. However, recent work has primarily focused on single matrix decompositions or lower precision techniques, which may fail to fully capture structural patterns. To address these limitations, we introduce TRAWL (Tensor Reduced and Approximated Weights for Large Language Models), a technique that applies tensor decomposition across multiple weight matrices to effectively denoise LLMs by capturing global structural patterns. Our experiments show that TRAWL improves model performance by up to 16% over baseline models on benchmark datasets, without requiring additional data, training, or fine-tuning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

23 of 37 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
11 more
  1. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  2. 2016Tensor Decomposition for Signal Processing and Machine Learningcited by 223
  3. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  4. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  5. 2016Tensor Ring Decompositioncited by 125
  6. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  7. 2016TensorLy: Tensor Learning in Pythoncited by 104
  8. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
  9. 2017Tensor Contraction Layers for Parsimonious Deep Netscited by 16
  10. 2017SPARTan: Scalable PARAFAC2 for Large & Sparse Datacited by 13
  11. 2022FRAPPE: underlineFast underlineRank underlineApproximation with underlineExplainable Features for Tensorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMscited by 11
  2. 2026Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoderscited by 5
  3. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 3
  4. 2026Low-Rank Tensor Approximation of Weights in Large Language Models via Cosine Lanczos Bidiagonalizationcited by 0
  5. 2026Structured Multidimensional Representation Learning for Large Language Modelscited by 0