mlsanityData to 14 Oct 2026

A Tensorized Transformer for Language Modeling

arXiv 1906.09777 ↗First posted 24 Jun 2019cs.CL cs.LGcited by 86

Not on the map. Why?

Abstract (arXiv)

Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 19; 2023: 17; 2024: 18; 2025: 9; 2026: 7.

86 papers.

Sort:

By area:

  1. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  2. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  3. 2023Scalable Transformer for PDE Surrogate Modelingcited by 128
  4. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  5. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  6. 2021A Survey on Green Deep Learningcited by 52
  7. 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
  8. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  9. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  10. 2023Transformers in Speech Processing: A Surveycited by 39
  11. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  12. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 32
38 more
  1. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  2. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 31
  3. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 30
  4. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  5. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  6. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  7. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  8. 2023Parameter-efficient Tuning of Large-scale Multimodal Foundation Modelcited by 23
  9. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  10. 2024What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)?cited by 22
  11. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  12. 2022Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Taskscited by 21
  13. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  14. 2021Causal Distillation for Language Modelscited by 19
  15. 2021KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantizationcited by 15
  16. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  17. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  18. 2021You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradientcited by 14
  19. 2023Tensor Decomposition for Model Reduction in Neural Networks: A Reviewcited by 14
  20. 2024Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMscited by 14
  21. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  22. 2024Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformerscited by 11
  23. 2022Privacy-preserving machine learning with tensor networkscited by 9
  24. 2022HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compressioncited by 9
  25. 2024Guaranteed Nonconvex Factorization Approach for Tensor Train Recoverycited by 8
  26. 2021TENT: Tensorized Encoder Transformer for Temperature Forecastingcited by 7
  27. 2022tntorch: Tensor Network Learning with PyTorchcited by 7
  28. 2021Adversarial Training with Contrastive Learning in NLPcited by 6
  29. 2021SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensionscited by 6
  30. 2025CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learningcited by 6
  31. 2023rm SP3: Enhancing Structured Pruning via PCA Projectioncited by 5
  32. 2024Efficiently Distilling LLMs for Edge Applicationscited by 5
  33. 2021Regularizing Transformers With Deep Probabilistic Layerscited by 4
  34. 2024Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglementcited by 4
  35. 2024Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Modelscited by 4
  36. 2025Tensorization is a powerful but underexplored tool for compression and interpretability of neural networkscited by 4
  37. 2025A Scalable Factorization Approach for High-Order Structured Tensor Recoverycited by 4
  38. 2022Position: Tensor Networks are a Valuable Asset for Green AIcited by 3

50 most cited of 86 All 86 (JSON)