mlsanityData to 14 Oct 2026

Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective

arXiv 2406.03768 ↗First posted 6 Jun 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Pre-trained large language models (LLMs) based on Transformer have demonstrated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can predict the label for an unseen input without any parameter updates. In this paper, we show an exciting phenomenon that SVD-based weight pruning can enhance ICL performance, and more surprising, pruning weights in deep layers often results in more stable performance improvements than in shallow layers. However, the underlying mechanism of those findings still remains an open question. To reveal those findings, we conduct an in-depth theoretical analysis by presenting the implicit gradient descent (GD) trajectories of ICL and giving the mutual information based generalization bounds of ICL via full implicit GD trajectories. This helps us reasonably explain the surprising experimental findings. Besides, based on all our experimental and theoretical insights, we intuitively propose a simple, model-compression and derivative-free algorithm for downstream tasks in enhancing ICL inference. Experiments on benchmark datasets and open source LLMs display the method effectivenessfootnoteThe code is available at urlhttps://github.com/chen123CtrlS/EnhancingICLSVDPruning..

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 40 of its 61 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

40 of 61 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  8. 2020Rethinking Attention with Performerscited by 1,601
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
28 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  8. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  10. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  11. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  12. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  13. 2024Many-Shot In-Context Learningcited by 206
  14. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  15. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  16. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  17. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  18. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  19. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  20. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  21. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  22. 2019Learning Filter Basis for Convolutional Neural Network Compressioncited by 57
  23. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  24. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 46
  25. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 43
  26. 2023In-Context Learning with Many Demonstration Examplescited by 39
  27. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  28. 2024Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layerscited by 8

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  2. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 10
  3. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  4. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9