mlsanityData to 14 Oct 2026

Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation

arXiv 2510.23123 ↗First posted 27 Oct 2025cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as BSigmaX A, where A and B are low-rank matrices (as in standard LoRA), and SigmaX is a diagonal matrix generated from each input token X. Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

39 of 49 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  12. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
27 more
  1. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  3. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  4. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  5. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  6. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  7. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  8. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  9. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  10. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  11. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 56
  12. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  13. 2025FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Modelscited by 29
  14. 2024Unleashing the Power of Continual Learning on Non-Centralized Devices: A Surveycited by 23
  15. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  16. 2024Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentationcited by 17
  17. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  18. 2024Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decompositioncited by 15
  19. 2022Adaptive Low-Precision Training for Embeddings in Click-Through Rate Predictioncited by 11
  20. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  21. 2024Comprehending Knowledge Graphs with Large Language Models for Recommender Systemscited by 9
  22. 2025Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentationcited by 7
  23. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
  24. 2024Masked Random Noise for Communication Efficient Federated Learningcited by 4
  25. 2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
  26. 2025Less Is More: Elevating RAG via Performance-Driven Context Compressioncited by 2
  27. 2025Quantifying Distributional Invariance in Causal Subgraph for IRM-Free Graph Generalizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functionscited by 1
  2. 2026FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Modelscited by 0
  3. 2026ARCA: Adapter-Residual Credit Assignment When Token Signals Degeneratecited by 0
  4. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  5. 2026Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlasescited by 0
  6. 2026Adaptive Utilization of Low-Rank Adaptation via Conditioned Gatingcited by 0