Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
Not on the map. Why?
Abstract (arXiv)
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 26 Sep 2025.
Builds on
29 of 44 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2023A Survey of Large Language Modelscited by 2,576
17 more
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
- 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
- 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
- 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
- 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
- 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
- 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
- 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
- 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.