# NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining

arXiv: https://arxiv.org/abs/2310.07402 · HTML: https://mlsanity.com/paper/2310.07402/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Temporal Pattern Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/1809.04206/) (2018)

## Built on by (most cited first, 20 of 20)

- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Mantis: Lightweight Foundation Model for Time Series Classification](https://mlsanity.com/paper/2502.15637/) (2025)
- [CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data](https://mlsanity.com/paper/2508.02879/) (2025)
- [MantisV2: Closing the Zero-Shot Gap in Time Series Classification with Synthetic Data and Test-Time Strategies](https://mlsanity.com/paper/2602.17868/) (2026)
- [Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification](https://mlsanity.com/paper/2510.26777/) (2025)
- [Unlocking Pretrained Vision Transformers for Time Series Classification](https://mlsanity.com/paper/2506.08641/) (2025)
- [TelecomTS: A Multi-Modal Observability Dataset for Time Series and Language Analysis](https://mlsanity.com/paper/2510.06063/) (2025)
- [LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning](https://mlsanity.com/paper/2607.00958/) (2026)
- [User-friendly Foundation Model Adapters for Multivariate Time Series Classification](https://mlsanity.com/paper/2409.12264/) (2024)
- [Leveraging Generic Time Series Foundation Models for EEG Classification](https://mlsanity.com/paper/2510.27522/) (2025)
- [A Unified Shape-Aware Foundation Model for Time Series Classification](https://mlsanity.com/paper/2601.06429/) (2026)
- [Rethinking Zero-Shot Time Series Classification: From Task-specific Classifiers to In-Context Inference](https://mlsanity.com/paper/2602.00620/) (2026)
- [Post-Training in Time Series Foundation Models: A Unifying Framework](https://mlsanity.com/paper/2607.20002/) (2026)
- [FIC-TSC: Learning Time Series Classification with Fisher Information Constraint](https://mlsanity.com/paper/2505.06114/) (2025)
- [UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification](https://mlsanity.com/paper/2603.01348/) (2026)
- [Tabby: An Open Pretraining Recipe for Time Series Foundation Models](https://mlsanity.com/paper/2609.13956/) (2026)
- [Adaptive Law-Based Transformation (ALT): A Lightweight Feature Representation for Time Series Classification](https://mlsanity.com/paper/2501.09217/) (2025)
- [TimEE: End-to-end Time Series Classification via In-Context Learning](https://mlsanity.com/paper/2607.07500/) (2026)
- [ChorusTIC: Training-Free Multivariate Time Series Classification via Chorus In-Context Learning](https://mlsanity.com/paper/2608.24033/) (2026)
- [FreqCondNorm: Towards Cross-domain Predictive Maintenance through a Frequency-Conditioned Transformer Foundation Model](https://mlsanity.com/paper/2609.20535/) (2026)

## Abstract (arXiv)

Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical amplitudes in a high-dimensional space, we propose a numerically multi-scaled embedding module enumerating all possible numerical scales for the scalars. The model undergoes pretraining with a simple contrastive objective on a large-scale dataset over a million sequences collected by merging existing public data. We study its transfer performance on a number of univariate and multivariate classification tasks, few shot learning, unsupervised clustering and anomaly detection benchmarks. Our method exhibits remarkable improvement against previous pretraining approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods. Code is available at: urlhttps://github.com/chenguolin/NuTime.