Evaluating Zero-Shot Long-Context LLM Compression
Not on the map. Why?
Abstract (arXiv)
This study evaluates the effectiveness of zero-shot compression techniques on large language models (LLMs) under long-context. We identify the tendency for computational errors to increase under long-context when employing certain compression methods. We propose a hypothesis to explain the varied behavior of different LLM compression techniques and explore remedies to mitigate the performance decline observed in some techniques under long-context. This is a course report for COS 598D Machine Learning and Systems by Prof. Kai Li at Princeton University. Due to limited computational resources, our experiments were conducted only on LLaMA-2-7B-32K.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 10 Jun 2024.
Builds on
34 of 43 references.
References · 34
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2023Mistral 7Bcited by 3,164
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
22 more
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
- 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
- 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
- 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
- 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
- 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
- 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
- 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
- 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
- 2024Evaluating Quantized Large Language Modelscited by 77
- 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
- 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
- 2023EPIM: Efficient Processing-In-Memory Accelerators based on Epitomecited by 3
Built on by
Not cited yet.