mlsanityData to 7 Oct 2026

Latent Space Factorization in LoRA

arXiv 2510.19640 ↗First posted 22 Oct 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning. However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages a VAE to learn two distinct latent spaces. Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information. Extensive experiments on text, audio, and image tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA. Moreover, spurious correlation evaluations confirm that FVAE-LoRA better isolates task-relevant signals, leading to improved robustness under distribution shifts. Our code is publicly available at: https://github.com/idiap/FVAE-LoRA

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

44 of 62 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Segment Anythingcited by 8,158
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
32 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  13. 2021GPT Understands, Toocited by 715
  14. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  15. 2018Disentangling by Factorisingcited by 614
  16. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  17. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  18. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  19. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  20. 2018Understanding disentangling in β-VAEcited by 332
  21. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  22. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  23. 2020Environment Inference for Invariant Learningcited by 306
  24. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  25. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 240
  26. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  27. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  28. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  29. 2020Robustness to Spurious Correlations via Human Annotationscited by 62
  30. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  31. 2023Spurious Correlations and Where to Find Themcited by 6
  32. 2023Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Featurescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 1