# Scalable Numerical Embeddings for Multivariate Time Series: Enhancing Healthcare Data Representation Learning

arXiv: https://arxiv.org/abs/2405.16557 · HTML: https://mlsanity.com/paper/2405.16557/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 26 May 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [SAITS: Self-Attention-based Imputation for Time Series](https://mlsanity.com/paper/2202.08516/) (2022)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Graph-Guided Network for Irregularly Sampled Multivariate Time Series](https://mlsanity.com/paper/2110.05357/) (2021)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](https://mlsanity.com/paper/2107.14293/) (2021)
- [Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer](https://mlsanity.com/paper/1906.04716/) (2019)
- [AirFormer: Predicting Nationwide Air Quality in China with Transformers](https://mlsanity.com/paper/2211.15979/) (2022)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling](https://mlsanity.com/paper/2210.12156/) (2022)
- [Transformed Distribution Matching for Missing Value Imputation](https://mlsanity.com/paper/2302.10363/) (2023)
- [CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation](https://mlsanity.com/paper/1905.09904/) (2019)
- [MIWAE: Deep Generative Modelling and Imputation of Incomplete Data](https://mlsanity.com/paper/1812.02633/) (2018)

## Built on by (most cited first, 1 of 1)

- [Echo Flow Networks](https://mlsanity.com/paper/2509.24122/) (2025)

## Abstract (arXiv)

Multivariate time series (MTS) data, when sampled irregularly and asynchronously, often present extensive missing values. Conventional methodologies for MTS analysis tend to rely on temporal embeddings based on timestamps that necessitate subsequent imputations, yet these imputed values frequently deviate substantially from their actual counterparts, thereby compromising prediction accuracy. Furthermore, these methods typically fail to provide robust initial embeddings for values infrequently observed or even absent within the training set, posing significant challenges to model generalizability. In response to these challenges, we propose SCAlable Numerical Embedding (SCANE), a novel framework that treats each feature value as an independent token, effectively bypassing the need for imputation. SCANE regularizes the traits of distinct feature embeddings and enhances representational learning through a scalable embedding mechanism. Coupling SCANE with the Transformer Encoder architecture, we develop the Scalable nUMerical eMbeddIng Transformer (SUMMIT), which is engineered to deliver precise predictive outputs for MTS characterized by prevalent missing entries. Our experimental validation, conducted across three disparate electronic health record (EHR) datasets marked by elevated missing value frequencies, confirms the superior performance of SUMMIT over contemporary state-of-the-art approaches addressing similar challenges. These results substantiate the efficacy of SCANE and SUMMIT, underscoring their potential applicability across a broad spectrum of MTS data analytical tasks.