mlsanityData to 7 Oct 2026

Low-Rank Interconnected Adaptation across Layers

arXiv 2407.09946 ↗First posted 13 Jul 2024cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates Delta W = AB for pretrained weights W through low-rank adapters A and B. While LoRA ensures hardware efficiency, its low-rank weight updates limit adaptation performance. In this paper, we propose low-rank interconnected adaptation across layers (Lily), a novel PEFT method that introduces an interconnected framework with locally shared A and globally shared B experts. This structure eliminates redundant per-layer AB pairs, enabling higher-rank Delta W with equal or fewer parameters. To enhance expressiveness, we use data-dependent routers to determine A-B interconnections, preventing B experts from converging to the same behavior and improving representational power across domains. Experiments across modalities, architectures, and model sizes demonstrate Lily's superior performance and efficiency. GitHub: https://github.com/yibozhong/lily

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

48 of 60 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
36 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  13. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  14. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  15. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  16. 2024LoRA Learns Less and Forgets Lesscited by 313
  17. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  18. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  19. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  20. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  21. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  22. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  23. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  24. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  25. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  26. 2024Higher Layers Need More LoRA Expertscited by 101
  27. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  28. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  29. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 67
  30. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  31. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  32. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  33. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  34. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  35. 2015Dynamic Capacity Networkscited by 16
  36. 2024Rethinking Low-Rank Adaptation in Vision: Exploring Head-Level Responsiveness across Diverse Taskscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  2. 2025Steering When Necessary: Flexible Steering Large Language Models with Backtrackingcited by 7
  3. 2025GeoEdit: Geometric Knowledge Editing for Large Language Modelscited by 6
  4. 2026CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learningcited by 5
  5. 2025AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learningcited by 4
  6. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contextscited by 4
  7. 2025Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Modelscited by 3
  8. 2025Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrievalcited by 2
  9. 2025MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 1
  10. 2026Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atomscited by 0