# Activation Boundary Matching: Task-Informed Initialization for Low-Rank Adaptation

arXiv: https://arxiv.org/abs/2511.19145 · HTML: https://mlsanity.com/paper/2511.19145/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [LoRA-GA: Low-Rank Adaptation with Gradient Approximation](https://mlsanity.com/paper/2407.05000/) (2024)
- [On the number of response regions of deep feed forward networks with piece-wise linear activations](https://mlsanity.com/paper/1312.6098/) (2013)
- [LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently](https://mlsanity.com/paper/2502.01235/) (2025)
- [Parameter Efficient Fine-tuning via Explained Variance Adaptation](https://mlsanity.com/paper/2410.07170/) (2024)
- [Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics](https://mlsanity.com/paper/2505.23194/) (2025)
- [ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints](https://mlsanity.com/paper/2507.08044/) (2025)
- [HRP: High-Rank Preheating for Superior LoRA Initialization](https://mlsanity.com/paper/2502.07739/) (2025)

## Built on by (most cited first, 1 of 1)

- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)

## Abstract (arXiv)

Low-Rank Adaptation (LoRA) is highly sensitive to initialization, yet existing schemes construct the initial subspace from statistics at the pretrained point, capturing pre-adaptation geometry rather than how the adapter must move during learning. We examine the early adaptation trajectory and uncover a temporal asymmetry: task-induced activation boundaries---the signs of layer-wise pre-activations---recover markedly faster than activation values or effective low-rank updates and become reusable well before the probe adapter converges. Motivated by this, we propose Activation Boundary Matching for LoRA (ABM-LoRA), which briefly trains a standard adapter as an early task probe, then uses its pre-activation signs as layer-wise targets for a fresh adapter under a margin-based hinge objective. Both are discarded before otherwise unchanged downstream fine-tuning. We show that this boundary-supervision lifting can expose LoRA directions attenuated or locally unobservable through the downstream Jacobian, and that what transfers is the boundary side---not the exact activation value or the output-level signal. Because a partial boundary trace suffices, ABM-LoRA recovers most of the benefit of a fully trained reference at a fraction of the overhead, requiring only probe forward passes. It improves over standard LoRA on T5-base/GLUE, ConvNeXt-T and Swin-T fine-grained classification, and instruction tuning with Qwen2.5-1.5B and LLaMA2-7B, and matches or surpasses SVD- and gradient-based initializers without their preprocessing.