# Parameter-Efficient Temporal Domain Generalization

arXiv: https://arxiv.org/abs/2602.11965 · HTML: https://mlsanity.com/paper/2602.11965/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Builds on (most cited first, 30 of 40)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [Learning to Learn Single Domain Generalization](https://mlsanity.com/paper/2003.13216/) (2020)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning](https://mlsanity.com/paper/2110.07577/) (2021)
- [Learning to Learn with Variational Information Bottleneck for Domain Generalization](https://mlsanity.com/paper/2007.07645/) (2020)
- [Revisiting Catastrophic Forgetting in Large Language Model Tuning](https://mlsanity.com/paper/2406.04836/) (2024)
- [Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://mlsanity.com/paper/2111.07408/) (2021)
- [OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling](https://mlsanity.com/paper/2309.12659/) (2023)
- [AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2301.12132/) (2023)
- [Mixture-of-Subspaces in Low-Rank Adaptation](https://mlsanity.com/paper/2406.11909/) (2024)
- [Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks](https://mlsanity.com/paper/2205.10664/) (2022)
- [Time is Encoded in the Weights of Finetuned Language Models](https://mlsanity.com/paper/2312.13401/) (2023)
- [Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time](https://mlsanity.com/paper/2108.06721/) (2021)
- [Sequential Learning for Domain Generalization](https://mlsanity.com/paper/2004.01377/) (2020)

## Abstract (arXiv)

Temporal domain generalization (TDG) learns from a sequence of observed domains to predict models for unseen future domains. Existing parameter-dynamics methods become prohibitively expensive at modern backbone scales because their temporal encoders and generators operate on the entire adapted parameter state. Replacing full-model updates with LoRA does not remove this failure mode: the dynamics model must still encode and reconstruct every adapter parameter, making the auxiliary model itself prohibitively large for modern backbones. We propose Low-rank Time-varying Core (LoTiC), a parameter-efficient TDG method that decomposes temporal updates into shared low-rank bases and a compact core. The bases define a stable update subspace, while only the core evolves. LoTiC therefore learns temporal dynamics in an r*r space instead of tracking the full adapter. We justify this factorization through the geometry of parameter increments and support continuous, recurrent, and time-conditioned core models. Experiments span six controlled and real-world benchmarks, three data modalities, classification and generation, and backbones from 66M to 8B parameters. LoTiC ranks first in 14 of 15 settings, including near-perfect future-domain accuracy with an 8B backbone on Rotating 2-Moons. It adds only 0.13--0.51% trainable parameters beyond LoRA and trains 1.8--7.1× faster than parameter-dynamics baselines. These results establish that temporal parameter dynamics can remain both learnable and efficient as pretrained backbones scale.