# HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts

arXiv: https://arxiv.org/abs/2305.18421 · HTML: https://mlsanity.com/paper/2305.18421/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 75 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 30 of 37)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Adversarial Domain Adaptation with Domain Mixup](https://mlsanity.com/paper/1912.01805/) (2019)
- [Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data](https://mlsanity.com/paper/1909.00889/) (2019)
- [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022)
- [How to Index Item IDs for Recommendation Foundation Models](https://mlsanity.com/paper/2305.06569/) (2023)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [Data-Driven Robust Optimization](https://mlsanity.com/paper/1401.0212/) (2013)
- [A Comparative Analysis of XGBoost](https://mlsanity.com/paper/1911.01914/) (2019)
- [Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2204.00570/) (2022)
- [A Century of Portraits: A Visual Historical Record of American High School Yearbooks](https://mlsanity.com/paper/1511.02575/) (2015)
- [The Power of Contrast for Feature Learning: A Theoretical Analysis](https://mlsanity.com/paper/2110.02473/) (2021)
- [Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learning](https://mlsanity.com/paper/2102.05824/) (2021)

## Built on by (most cited first, 8 of 8)

- [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023)
- [StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows](https://mlsanity.com/paper/2403.11322/) (2024)
- [Cantor: Inspiring Multimodal Chain-of-Thought of MLLM](https://mlsanity.com/paper/2404.16033/) (2024)
- [IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models](https://mlsanity.com/paper/2310.10873/) (2023)
- [EcoAct: Economic Agent Determines When to Register What Action](https://mlsanity.com/paper/2411.01643/) (2024)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level](https://mlsanity.com/paper/2311.13892/) (2023)
- [A nonparametric two-sample test using a parametric integral probability metric](https://mlsanity.com/paper/2606.16941/) (2026)

## Abstract (arXiv)

In this work, we propose a hyperparameter optimization method named HyperTime to find hyperparameters robust to potential temporal distribution shifts in the unseen test data. Our work is motivated by an important observation that it is, in many cases, possible to achieve temporally robust predictive performance via hyperparameter optimization. Based on this observation, we leverage the `worst-case-oriented' philosophy from the robust optimization literature to help find such robust hyperparameter configurations. HyperTime imposes a lexicographic priority order on average validation loss and worst-case validation loss over chronological validation sets. We perform a theoretical analysis on the upper bound of the expected test loss, which reveals the unique advantages of our approach. We also demonstrate the strong empirical performance of the proposed method on multiple machine learning tasks with temporal distribution shifts.