# STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data

arXiv: https://arxiv.org/abs/2504.10097 · HTML: https://mlsanity.com/paper/2504.10097/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://mlsanity.com/paper/2205.14415/) (2022)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [Set Functions for Time Series](https://mlsanity.com/paper/1909.12064/) (2019)
- [AdaRNN: Adaptive Learning and Forecasting of Time Series](https://mlsanity.com/paper/2108.04443/) (2021)
- [Graph-Guided Network for Irregularly Sampled Multivariate Time Series](https://mlsanity.com/paper/2110.05357/) (2021)
- [Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting](https://mlsanity.com/paper/2402.05956/) (2024)
- [Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion](https://mlsanity.com/paper/2202.04770/) (2022)
- [Interpolation-Prediction Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/1909.07782/) (2019)
- [Modeling Missing Data in Clinical Time Series with RNNs](https://mlsanity.com/paper/1606.04130/) (2016)
- [Time Series as Images: Vision Transformer for Irregularly Sampled Time Series](https://mlsanity.com/paper/2303.12799/) (2023)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting](https://mlsanity.com/paper/2305.12095/) (2023)
- [DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation](https://mlsanity.com/paper/2201.04038/) (2022)
- [MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2312.02530/) (2023)

## Built on by (most cited first, 1 of 1)

- [AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://mlsanity.com/paper/2604.05550/) (2026)

## Abstract (arXiv)

Understanding user intent is essential for situational and context-aware decision-making. Motivated by a real-world scenario, this work addresses intent predictions of smart device users in the vicinity of vehicles by modeling sequential spatiotemporal data. However, in real-world scenarios, environmental factors and sensor limitations can result in non-stationary and irregularly sampled data, posing significant challenges. To address these issues, we propose STaRFormer, a Transformer-based approach that can serve as a universal framework for sequential modeling. STaRFormer utilizes a new dynamic attention-based regional masking scheme combined with a novel semi-supervised contrastive learning paradigm to enhance task-specific latent representations. Comprehensive experiments on 56 datasets varying in types (including non-stationary and irregularly sampled), tasks, domains, sequence lengths, training samples, and applications demonstrate the efficacy of STaRFormer, achieving notable improvements over state-of-the-art approaches.