# Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time

arXiv: https://arxiv.org/abs/2108.06721 · HTML: https://mlsanity.com/paper/2108.06721/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 15 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [Understanding Self-Training for Gradual Domain Adaptation](https://mlsanity.com/paper/2002.11361/) (2020)
- [Efficient Domain Generalization via Common-Specific Low-Rank Decomposition](https://mlsanity.com/paper/2003.12815/) (2020)
- [Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL](https://mlsanity.com/paper/1812.07671/) (2018)
- [Continual Backprop: Stochastic Gradient Descent with Persistent Randomness](https://mlsanity.com/paper/2108.06325/) (2021)
- [Continuously Indexed Domain Adaptation](https://mlsanity.com/paper/2007.01807/) (2020)
- [AdaGraph: Unifying Predictive and Continuous Domain Adaptation through Graphs](https://mlsanity.com/paper/1903.07062/) (2019)
- [Continuous Domain Adaptation with Variational Domain-Agnostic Feature Replay](https://mlsanity.com/paper/2003.04382/) (2020)
- [Follow the Perturbed Leader: Optimism and Fast Parallel Algorithms for Smooth Minimax Games](https://mlsanity.com/paper/2006.07541/) (2020)

## Built on by (most cited first, 28 of 28)

- [Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks](https://mlsanity.com/paper/2205.10664/) (2022)
- [Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data](https://mlsanity.com/paper/2411.10634/) (2024)
- [MADG: Margin-based Adversarial Learning for Domain Generalization](https://mlsanity.com/paper/2311.08503/) (2023)
- [Continuous Temporal Domain Generalization](https://mlsanity.com/paper/2405.16075/) (2024)
- [Learning Robust Spectral Dynamics for Temporal Domain Generalization](https://mlsanity.com/paper/2505.12585/) (2025)
- [DEJA VU: Continual Model Generalization For Unseen Domains](https://mlsanity.com/paper/2301.10418/) (2023)
- [Generalizing across Temporal Domains with Koopman Operators](https://mlsanity.com/paper/2402.07834/) (2024)
- [Foresee What You Will Learn: Data Augmentation for Domain Generalization in Non-stationary Environment](https://mlsanity.com/paper/2301.07845/) (2023)
- [FedDAR: Federated Domain-Aware Representation Learning](https://mlsanity.com/paper/2209.04007/) (2022)
- [Continuous Treatment Effect Estimation Using Gradient Interpolation and Kernel Smoothing](https://mlsanity.com/paper/2401.15447/) (2024)
- [Temporal Test-Time Adaptation with State-Space Models](https://mlsanity.com/paper/2407.12492/) (2024)
- [CODA: Temporal Domain Generalization via Concept Drift Simulator](https://mlsanity.com/paper/2310.01508/) (2023)
- [Enhancing Evolving Domain Generalization through Dynamic Latent Representations](https://mlsanity.com/paper/2401.08464/) (2024)
- [Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains](https://mlsanity.com/paper/2506.17718/) (2025)
- [Deep Spatial Domain Generalization](https://mlsanity.com/paper/2210.00729/) (2022)
- [Online time series prediction using feature adjustment](https://mlsanity.com/paper/2509.03810/) (2025)
- [Continuous Domain Generalization](https://mlsanity.com/paper/2505.13519/) (2025)
- [Optimal Empirical Risk Minimization under Temporal Distribution Shifts](https://mlsanity.com/paper/2507.13287/) (2025)
- [TEFL: Prediction-Residual-Guided Rolling Forecasting for Multi-Horizon Time Series](https://mlsanity.com/paper/2602.22520/) (2026)
- [Prompting-based Temporal Domain Generalization](https://mlsanity.com/paper/2310.02473/) (2023)
- [Learning Latent Spaces for Domain Generalization in Time Series Forecasting](https://mlsanity.com/paper/2412.11171/) (2024)
- [Generalizing to New Dynamical Systems via Frequency Domain Adaptation](https://mlsanity.com/paper/2507.00025/) (2025)
- [Temporal Generalization: A Reality Check](https://mlsanity.com/paper/2509.23487/) (2025)
- [Parameter-Efficient Temporal Domain Generalization](https://mlsanity.com/paper/2602.11965/) (2026)
- [Autonomous Drift Learning in Data Streams: A Unified Perspective](https://mlsanity.com/paper/2605.01295/) (2026)
- [NOMADD: Numerical Optimization of Models Adapting to Data Drift](https://mlsanity.com/paper/2608.02845/) (2026)
- [FreKoo++: Learning Continuous Spectral Dynamics for Temporal Domain Generalization](https://mlsanity.com/paper/2608.22224/) (2026)
- [Adaptive Spectral-Koopman Dynamics Modeling for Temporal Domain Generalization](https://mlsanity.com/paper/2610.02822/) (2026)

## Abstract (arXiv)

In several real world applications, machine learning models are deployed to make predictions on data whose distribution changes gradually along time, leading to a drift between the train and test distributions. Such models are often re-trained on new data periodically, and they hence need to generalize to data not too far into the future. In this context, there is much prior work on enhancing temporal generalization, e.g. continuous transportation of past data, kernel smoothed time-sensitive parameters and more recently, adversarial learning of time-invariant features. However, these methods share several limitations, e.g, poor scalability, training instability, and dependence on unlabeled data from the future. Responding to the above limitations, we propose a simple method that starts with a model with time-sensitive parameters but regularizes its temporal complexity using a Gradient Interpolation (GI) loss. GI allows the decision boundary to change along time and can still prevent overfitting to the limited training time snapshots by allowing task-specific control over changes along time. We compare our method to existing baselines on multiple real-world datasets, which show that GI outperforms more complicated generative and adversarial approaches on the one hand, and simpler gradient regularization methods on the other.