mlsanityData to 14 Oct 2026

RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

arXiv 2410.10075 ↗First posted 14 Oct 2024cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 58 of its 71 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

58 of 71 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  13. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  15. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  16. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  17. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  18. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  19. 2015Teaching Machines to Read and Comprehendcited by 1,049
  20. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  21. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  22. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  23. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  24. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  25. 2024A Survey on Large Language Models for Code Generationcited by 658
  26. 2016Solving General Arithmetic Word Problemscited by 360
  27. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  28. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  29. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  30. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  31. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  32. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  33. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  34. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  35. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  36. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  37. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  38. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  2. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2
  3. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learningcited by 2
  4. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 2
  5. 2026Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuningcited by 1
  6. 2025FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Modelscited by 0
  7. 2026LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learningcited by 0
  8. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0