mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning

arXiv 2410.08598 ↗First posted 11 Oct 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are gaining significant popularity in recent years for specialized tasks using prompts due to their low computational cost. Standard methods like prefix tuning utilize special, modifiable tokens that lack semantic meaning and require extensive training for best performance, often falling short. In this context, we propose a novel method called Semantic Knowledge Tuning (SK-Tuning) for prompt and prefix tuning that employs meaningful words instead of random tokens. This method involves using a fixed LLM to understand and process the semantic content of the prompt through zero-shot capabilities. Following this, it integrates the processed prompt with the input text to improve the model's performance on particular tasks. Our experimental results show that SK-Tuning exhibits faster training times, fewer parameters, and superior performance on tasks such as text classification and understanding compared to other tuning methods. This approach offers a promising method for optimizing the efficiency and effectiveness of LLMs in processing language tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 60 of its 84 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

60 of 84 references.

References · 60

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  10. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  11. 2021GPT Understands, Toocited by 715
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  14. 2019Meta-Learning with Implicit Gradientscited by 523
  15. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  16. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 253
  17. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  18. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  19. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  20. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  21. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  22. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  23. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  24. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  25. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  26. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  27. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  28. 2015Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guaranteescited by 68
  29. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  30. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  31. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  32. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  33. 2022Dynamic Prefix-Tuning for Generative Template-based Event Extractioncited by 35
  34. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  35. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  36. 2016Non-Convex Projected Gradient Descent for Generalized Low-Rank Tensor Regressioncited by 23
  37. 2018A Neural Multi-Task Learning Framework to Jointly Model Medical Named Entity Recognition and Normalizationcited by 20
  38. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 4.

10 papers.

  1. 2024Fine-tuning foundational models to code diagnoses from veterinary health recordscited by 3
  2. 2025Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuningcited by 3
  3. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learningcited by 2
  4. 2025Prior Distribution and Model Confidencecited by 1
  5. 2025Optimizing GPT for Video Understanding: Zero-Shot Performance and Prompt Engineeringcited by 0
  6. 2025Systematic Optimization of Open Source Large Language Models for Mathematical Reasoningcited by 0
  7. 2025A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMscited by 0
  8. 2026LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learningcited by 0
  9. 2026Towards Lightweight Reliability: Using Soft Prompts for Hallucination Mitigation in Large Language Modelscited by 0
  10. 2026CRAMER: Control via Request-Aware Masking for Editing Recommenderscited by 0