# Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?

arXiv: https://arxiv.org/abs/2606.08578 · HTML: https://mlsanity.com/paper/2606.08578/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 7 Jun 2026.

## Builds on (most cited first, 28 of 28)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [Sundial: A Family of Highly Capable Time Series Foundation Models](https://mlsanity.com/paper/2502.00816/) (2025)
- [Time-Series Anomaly Detection Service at Microsoft](https://mlsanity.com/paper/1906.03821/) (2019)
- [Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress](https://mlsanity.com/paper/2009.13807/) (2020)
- [Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models](https://mlsanity.com/paper/1909.11299/) (2019)
- [Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series](https://mlsanity.com/paper/2401.03955/) (2024)
- [The Goldilocks zone: Towards better understanding of neural network loss landscapes](https://mlsanity.com/paper/1807.02581/) (2018)
- [What can linear interpolation of neural network loss landscapes tell us?](https://mlsanity.com/paper/2106.16004/) (2021)
- [Continual Learning with Weight Interpolation](https://mlsanity.com/paper/2404.04002/) (2024)
- [Multi-period Learning for Financial Time Series Forecasting](https://mlsanity.com/paper/2511.08622/) (2025)
- [SEMixer: Semantics Enhanced MLP-Mixer for Multiscale Mixing and Long-term Time Series Forecasting](https://mlsanity.com/paper/2602.16220/) (2026)
- [A Lightweight Sparse Interaction Network for Time Series Forecasting](https://mlsanity.com/paper/2602.01585/) (2026)
- [Diff-MN: Diffusion Parameterized MoE-NCDE for Continuous Time Series Generation with Irregular Observations](https://mlsanity.com/paper/2601.13534/) (2026)
- [Amortized Predictability-aware Training Framework for Time Series Forecasting and Classification](https://mlsanity.com/paper/2602.16224/) (2026)

## Built on by (most cited first, 1 of 1)

- [Self-Adaptive Scale Handling for Forecasting Time Series with Scale Heterogeneity](https://mlsanity.com/paper/2606.20010/) (2026)

## Abstract (arXiv)

Recently, large time series models (LTSMs) have gained increasing attention due to their similarities to large language models, including flexible context length, scalability, and task generality, outperforming advanced task-specific models. However, prior studies indicate that pre-trained LTSMs may exhibit a poorly conditioned non-convex loss landscape, leading to limited trainability. As a result, direct fine-tuning tends to cause overfitting and suboptimal performance, sometimes even worse than training from scratch, substantially diminishing the benefits of pre-training. To overcome this limitation, we propose Smoothed Full Fine-tuning (SFF), a novel fine-tuning technology. Specifically, we construct an auxiliary LTSM via random initialization to obtain a smoother loss landscape, and then linearly interpolate its weights with those of the pre-trained model to smooth the original landscape. This process improves trainability while preserving pre-trained knowledge, thereby enabling more effective downstream fine-tuning. From an optimization perspective, SFF perturbs sharp minima without significantly harming flat regions, facilitating escape from poor local basins toward smoother and more generalizable solutions. Extensive experiments on benchmark datasets demonstrate consistent improvements across eight representative LTSMs, including Timer, TimesFM, MOMENT, UniTS, MOIRAI, Chronos, TTMs, and Sundial, on diverse downstream tasks. The code is available at the link: https://github.com/Meteor-Stars/SFF.