mlsanityData to 14 Oct 2026

TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models

arXiv 2101.11714 ↗First posted 25 Jan 2021cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

The memory capacity of embedding tables in deep learning recommendation models (DLRMs) is increasing dramatically from tens of GBs to TBs across the industry. Given the fast growth in DLRMs, novel solutions are urgently needed, in order to enable fast and efficient DLRM innovations. At the same time, this must be done without having to exponentially increase infrastructure capacity demands. In this paper, we demonstrate the promising potential of Tensor Train decomposition for DLRMs (TT-Rec), an important yet under-investigated context. We design and implement optimized kernels (TT-EmbeddingBag) to evaluate the proposed TT-Rec design. TT-EmbeddingBag is 3 times faster than the SOTA TT implementation. The performance of TT-Rec is further optimized with the batched matrix multiplication and caching strategies for embedding vector lookup operations. In addition, we present mathematically and empirically the effect of weight initialization distribution on DLRM accuracy and propose to initialize the tensor cores of TT-Rec following the sampled Gaussian distribution. We evaluate TT-Rec across three important design space dimensions -- memory capacity, accuracy, and timing performance -- by training MLPerf-DLRM with Criteo's Kaggle and Terabyte data sets. TT-Rec achieves 117 times and 112 times model size compression, for Kaggle and Terabyte, respectively. This impressive model size reduction can come with no accuracy nor training time overhead as compared to the uncompressed baseline.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 25 Jan 2021.

Builds on

25 of 50 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  5. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  7. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  8. 2015Tensorizing Neural Networkscited by 250
  9. 2019MLPerf Inference Benchmarkcited by 199
  10. 2019MLPerf Training Benchmarkcited by 115
  11. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  12. 2019The Architectural Implications of Facebook's DNN-based Personalized Recommendationcited by 83
13 more
  1. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
  2. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  3. 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
  4. 2018Wide Compression: Tensor Ring Netscited by 50
  5. 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
  6. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  7. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 38
  8. 2020Understanding Capacity-Driven Scale-Out Neural Recommendation Inferencecited by 29
  9. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  10. 2019Bayesian Tensorized Neural Networks with Automatic Rank Selectioncited by 26
  11. 2019Post-Training 4-bit Quantization on Embedding Tablescited by 22
  12. 2020Developing a Recommendation Benchmark for MLPerf Training and Inferencecited by 12
  13. 2020Training with Multi-Layer Embeddings for Model Reductioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 12; 2024: 15; 2025: 5; 2026: 4.

58 papers.

Sort:

By area:

  1. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  2. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 86
  3. 2022A Comprehensive Survey on Trustworthy Recommender Systemscited by 32
  4. 2022RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendationcited by 28
  5. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
  6. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  7. 2022Conditional Imitation Learning for Multi-Agent Gamescited by 13
  8. 2022Learning Compressed Embeddings for On-Device Inferencecited by 13
  9. 2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
  10. 2023Experimental Analysis of Large-scale Learnable Vector Storage Compressioncited by 12
  11. 2022Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwardscited by 11
  12. 2022Adaptive Low-Precision Training for Embeddings in Click-Through Rate Predictioncited by 11
38 more
  1. 2022BagPipe: Accelerating Deep Recommendation Model Trainingcited by 10
  2. 2022RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructurecited by 9
  3. 2022HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compressioncited by 9
  4. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9
  5. 2023Generative Learning of Continuous Data by Tensor Networkscited by 9
  6. 2021RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performancecited by 8
  7. 2023Efficient GPT Model Pre-training using Tensor Train Matrix Representationcited by 8
  8. 2023CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Modelscited by 7
  9. 2022Nimble GNN Embedding with Tensor-Train Decompositioncited by 6
  10. 2022Learning to Collide: Recommendation System Model Compression with Learned Hash Functionscited by 5
  11. 2022MorphTE: Injecting Morphology in Tensorized Embeddingscited by 5
  12. 2024Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compressioncited by 5
  13. 2021Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000× Compression and 3.1× Faster Inferencecited by 4
  14. 2023FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Modelscited by 4
  15. 2024A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systemscited by 4
  16. 2024GraphScale: A Framework to Enable Machine Learning over Billion-node Graphscited by 4
  17. 2023Mem-Rec: Memory Efficient Recommendation System using Alternative Representationcited by 3
  18. 2024DoTA: Weight-Decomposed Tensor Adaptation for Large Language Modelscited by 3
  19. 2022Efficient model compression with Random Operation Access Specific Tile (ROAST) hashingcited by 2
  20. 2022The trade-offs of model size in large recommendation models : A 10000 × compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)cited by 2
  21. 2022Clustering the Sketch: A Novel Approach to Embedding Table Compressioncited by 2
  22. 2024DQRM: Deep Quantized Recommendation Modelscited by 2
  23. 2024On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspectivecited by 2
  24. 2025Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Trainingcited by 2
  25. 2025Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-trainingcited by 2
  26. 2022Data Leakage via Access Patterns of Sparse Features in Deep Learning-based Recommendation Systemscited by 1
  27. 2023From the Pursuit of Universal AGI Architecture to Systematic Approach to Heterogenous AGI: Addressing Alignment, Energy, & AGI Grand Challengescited by 1
  28. 2024LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Modelscited by 1
  29. 2024PIFS-Rec: Process-In-Fabric-Switch for Large-Scale Recommendation System Inferencescited by 1
  30. 2024Disaggregating Embedding Recommendation Systems with FlexEMRcited by 1
  31. 2025Rec-AD: An Efficient Computation Framework for FDIA Detection Based on Tensor Train Decomposition and Deep Learning Recommendation Modelcited by 1
  32. 2021ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Codingcited by 0
  33. 2023Review of compressed embedding layers and their applications for recommender systemscited by 0
  34. 2023TensorCodec: Compact Lossy Compression of Tensors without Strong Data Assumptionscited by 0
  35. 2023Enhancing Cross-Category Learning in Recommendation Systems with Multi-Layer Embedding Trainingcited by 0
  36. 2024ProactivePIM: Accelerating Weight-Sharing Embedding Layer with PIM for Scalable Recommendation Systemcited by 0
  37. 2024Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUscited by 0
  38. 2024Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddingscited by 0

50 most cited of 58 All 58 (JSON)