mlsanityData to 7 Oct 2026

The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction

arXiv 2312.13558 ↗First posted 21 Dec 2023cs.LG cs.AI cs.CLcited by 120

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Transformer-based Large Language Models (LLMs) have become a fixture in modern machine learning. Correspondingly, significant resources are allocated towards research that aims to further advance this technology, typically resulting in models of increasing size that are trained on increasing amounts of data. This work, however, demonstrates the surprising result that it is often possible to significantly improve the performance of LLMs by selectively removing higher-order components of their weight matrices. This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed, and requires no additional parameters or data. We show extensive experiments demonstrating the generality of this finding across language models and datasets, and provide in-depth analyses offering insights into both when LASER is effective and the mechanism by which it operates.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 44 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

27 of 44 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
15 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  6. 2013Do Deep Nets Really Need to be Deep?cited by 466
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  9. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  10. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  11. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  12. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  13. 2022Characterizing Intrinsic Compositionality in Transformers with Tree Projectionscited by 40
  14. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  15. 2021Of Non-Linearity and Commutativity in BERTcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 58; 2025: 28; 2026: 34.

120 papers.

Sort:

By area:

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  3. 2024How do Large Language Models Handle Multilingualism?cited by 148
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  5. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  6. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  7. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  8. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  9. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  10. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  11. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
  12. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
38 more
  1. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  2. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
  3. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  4. 2024Weight decay induces low-rank attention layerscited by 27
  5. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  6. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuningcited by 24
  7. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  8. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  9. 2025The Universal Weight Subspace Hypothesiscited by 22
  10. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  11. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  12. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  13. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  14. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  15. 2024Towards a theory of model distillationcited by 14
  16. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
  17. 2024SLIP: Securing LLMs IP Using Weights Decompositioncited by 13
  18. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  19. 2025Guiding Reasoning in Small Language Models with LLM Assistancecited by 13
  20. 2024Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layerscited by 12
  21. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  22. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  23. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMscited by 11
  24. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  25. 2024Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Fieldcited by 10
  26. 2025Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learningcited by 10
  27. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
  28. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 9
  29. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Modelscited by 9
  30. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 9
  31. 2024EVCL: Elastic Variational Continual Learning with Weight Consolidationcited by 9
  32. 2024Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Modelscited by 9
  33. 2025EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inferencecited by 9
  34. 2025Calibration Across Layers: Understanding Calibration Evolution in LLMscited by 9
  35. 2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
  36. 2025Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibilitycited by 8
  37. 2024Federated Dynamical Low-Rank Training with Global Loss Convergence Guaranteescited by 7
  38. 2024Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysiscited by 6

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection
  2. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  3. 2024Surgical Feature-Space Decomposition of LLMs: Why, When and How?
  4. 2024A Survey on Efficient Inference for Large Language Models
  5. 2024Compressing Large Language Models with Automated Sub-Network Search
  6. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference
  7. 2024Search for Efficient Large Language Models
  8. 2024Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models