mlsanityData to 7 Oct 2026

VcLLM: Video Codecs are Secretly Tensor Codecs

arXiv 2407.00467 ↗First posted 29 Jun 2024cs.LG cs.DC eess.IVcited by 0

Not on the map. Why?

Abstract (arXiv)

As the parameter size of large language models (LLMs) continues to expand, the need for a large memory footprint and high communication bandwidth have become significant bottlenecks for the training and inference of LLMs. To mitigate these bottlenecks, various tensor compression techniques have been proposed to reduce the data size, thereby alleviating memory requirements and communication pressure. Our research found that video codecs, despite being originally designed for compressing videos, show excellent efficiency when compressing various types of tensors. We demonstrate that video codecs can be versatile and general-purpose tensor codecs while achieving the state-of-the-art compression efficiency in various tasks. We further make use of the hardware video encoding and decoding module available on GPUs to create a framework capable of both inference and training with video codecs repurposed as tensor codecs. This greatly reduces the requirement for memory capacity and communication bandwidth, enabling training and inference of large models on consumer-grade GPUs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 29 Jun 2024.

Builds on

52 of 76 references.

References · 52

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  11. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  12. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  13. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  14. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  15. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  16. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  17. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  18. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  19. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  20. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  21. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  22. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  23. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  24. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  25. 2017Revisiting Distributed Synchronous SGDcited by 142
  26. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  27. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  28. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  29. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  30. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  31. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  32. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  33. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 59
  34. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  35. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  36. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  37. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  38. 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMscited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.