One Rank at a Time: Cascading Error Dynamics in Sequential Learning
Not on the map. Why?
Abstract (arXiv)
Sequential learning -- where complex tasks are broken down into simpler, hierarchical components -- has emerged as a paradigm in AI. This paper views sequential learning through the lens of low-rank linear regression, focusing specifically on how errors propagate when learning rank-1 subspaces sequentially. We present an analysis framework that decomposes the learning process into a series of rank-1 estimation problems, where each subsequent estimation depends on the accuracy of previous steps. Our contribution is a characterization of the error propagation in this sequential process, establishing bounds on how errors -- e.g., due to limited computational budgets and finite precision -- affect the overall model accuracy. We prove that these errors compound in predictable ways, with implications for both algorithmic design and stability guarantees.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 46 of its 89 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
46 of 89 references.
References · 46
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
- 2016Learning to learn by gradient descent by gradient descentcited by 620
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
- 2019Orthogonal Gradient Descent for Continual Learningcited by 369
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2018Understanding Batch Normalizationcited by 159
- 2022Understanding The Robustness in Vision Transformerscited by 144
- 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 129
34 more
- 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
- 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
- 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
- 2014Guaranteed Matrix Completion via Non-convex Factorizationcited by 82
- 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 82
- 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
- 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
- 2015Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guaranteescited by 70
- 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
- 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
- 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
- 2015Dropping Convexity for Faster Semi-definite Optimizationcited by 40
- 2020EigenGame: PCA as a Nash Equilibriumcited by 39
- 2016Non-square matrix sensing without spurious local minima via the Burer-Monteiro approachcited by 38
- 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
- 2015A Convergent Gradient Descent Algorithm for Rank Minimization and Semidefinite Programming from Random Linear Measurementscited by 34
- 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
- 2013Provable Inductive Matrix Completioncited by 32
- 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
- 2023Continual Learning with Low Rank Adaptationcited by 29
- 2017Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recoverycited by 25
- 2017Accelerated Stochastic Power Iterationcited by 22
- 2018Streaming PCA and Subspace Tracking: The Missing Data Casecited by 19
- 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
- 2017Learning to Compose Skillscited by 15
- 2021EigenGame Unloaded: When playing games is better than optimizingcited by 13
- 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
- 2018Robustness via Deep Low-Rank Representationscited by 11
- 2017IHT dies hard: Provable accelerated Iterative Hard Thresholdingcited by 10
- 2021Fast quantum state reconstruction via accelerated non-convex programmingcited by 5
- 2023Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamicscited by 3
- 2017Provable quantum state tomography via non-convex methodscited by 2
- 2024Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysiscited by 2
- 2024Sequential Learning in the Dense Associative Memorycited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.