# MATA-Former & SIICU: Semantic Aware Temporal Alignment for High-Fidelity ICU Risk Prediction

arXiv: https://arxiv.org/abs/2604.01727 · HTML: https://mlsanity.com/paper/2604.01727/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Learning to Diagnose with LSTM Recurrent Neural Networks](https://mlsanity.com/paper/1511.03677/) (2015)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](https://mlsanity.com/paper/2107.14293/) (2021)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [A Survey on Principles, Models and Methods for Learning from Irregularly Sampled Time Series](https://mlsanity.com/paper/2012.00168/) (2020)
- [Large Language Models are Powerful Electronic Health Record Encoders](https://mlsanity.com/paper/2502.17403/) (2025)
- [Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs](https://mlsanity.com/paper/2112.05195/) (2021)
- [Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series](https://mlsanity.com/paper/2307.10923/) (2023)
- [Multimodal Medical Code Tokenizer](https://mlsanity.com/paper/2502.04397/) (2025)
- [Temporal Label Smoothing for Early Event Prediction](https://mlsanity.com/paper/2208.13764/) (2022)
- [Medical Concept Embedding with Time-Aware Attention](https://mlsanity.com/paper/1806.02873/) (2018)
- [Time-Aware Attention for Enhanced Electronic Health Records Modeling](https://mlsanity.com/paper/2507.14847/) (2025)
- [Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification](https://mlsanity.com/paper/2508.12418/) (2025)

## Abstract (arXiv)

Forecasting evolving clinical risks relies on intrinsic pathological dependencies rather than mere chronological proximity, yet current methods struggle with coarse binary supervision and physical timestamps. To align predictive modeling with clinical logic, we propose the Medical-semantics Aware Time-ALiBi Transformer (MATA-Former), utilizing event semantics to dynamically parameterize attention weights to prioritize causal validity over time lags. Furthermore, we introduce Plateau-Gaussian Soft Labeling (PSL), reformulating binary classification into continuous multi-horizon regression for full-trajectory risk modeling. Evaluated on SIICU -- a newly constructed dataset featuring over 506k events with rigorous expert-verified, fine-grained annotations -- and the MIMIC-IV dataset, our framework demonstrates superior efficacy and robust generalization in capturing risks from text-intensive, irregular clinical time series.