# Self-attention with Functional Time Representation Learning

arXiv: https://arxiv.org/abs/1911.12864 · HTML: https://mlsanity.com/paper/1911.12864/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 30 of 52)

- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks](https://mlsanity.com/paper/2101.05974/) (2021)
- [Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer](https://mlsanity.com/paper/2108.06625/) (2021)
- [ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling](https://mlsanity.com/paper/2402.10635/) (2024)
- [SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization](https://mlsanity.com/paper/2104.09125/) (2021)
- [Provably expressive temporal graph networks](https://mlsanity.com/paper/2209.15059/) (2022)
- [Neighborhood-aware Scalable Temporal Network Representation Learning](https://mlsanity.com/paper/2209.01084/) (2022)
- [Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders](https://mlsanity.com/paper/2305.18612/) (2023)
- [NRTSI: Non-Recurrent Time Series Imputation](https://mlsanity.com/paper/2102.03340/) (2021)
- [MIRA: Medical Time Series Foundation Model for Real-World Health Data](https://mlsanity.com/paper/2506.07584/) (2025)
- [Inductive Representation Learning in Temporal Networks via Mining Neighborhood and Community Influences](https://mlsanity.com/paper/2110.00267/) (2021)
- [TIGER: Temporal Interaction Graph Embedding with Restarts](https://mlsanity.com/paper/2302.06057/) (2023)
- [Uniform Sequence Better: Time Interval Aware Data Augmentation for Sequential Recommendation](https://mlsanity.com/paper/2212.08262/) (2022)
- [Dynamic Graph Representation Learning with Neural Networks: A Survey](https://mlsanity.com/paper/2304.05729/) (2023)
- [VigDet: Knowledge Informed Neural Temporal Point Process for Coordination Detection on Social Media](https://mlsanity.com/paper/2110.15454/) (2021)
- [Encoder-Decoder Architecture for Supervised Dynamic Graph Learning: A Survey](https://mlsanity.com/paper/2203.10480/) (2022)
- [Neural Predicting Higher-order Patterns in Temporal Networks](https://mlsanity.com/paper/2106.06039/) (2021)
- [Attention Mixtures for Time-Aware Sequential Recommendation](https://mlsanity.com/paper/2304.08158/) (2023)
- [Neural Markov Jump Processes](https://mlsanity.com/paper/2305.19744/) (2023)
- [Learning Self-Modulating Attention in Continuous Time Space with Applications to Sequential Recommendation](https://mlsanity.com/paper/2204.06517/) (2022)
- [Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding](https://mlsanity.com/paper/2407.09370/) (2024)
- [Learning Time-aware Graph Structures for Spatially Correlated Time Series Forecasting](https://mlsanity.com/paper/2312.16403/) (2023)
- [Time-aware Graph Structure Learning via Sequence Prediction on Temporal Graphs](https://mlsanity.com/paper/2306.07699/) (2023)
- [CAT-Walk: Inductive Hypergraph Learning via Set Walks](https://mlsanity.com/paper/2306.11147/) (2023)
- [Neural Gaussian Scale-Space Fields](https://mlsanity.com/paper/2405.20980/) (2024)
- [ConTIG: Continuous Representation Learning on Temporal Interaction Graphs](https://mlsanity.com/paper/2110.06088/) (2021)
- [Time-Parameterized Convolutional Neural Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2308.03210/) (2023)
- [Exploiting Session Information in BERT-based Session-aware Sequential Recommendation](https://mlsanity.com/paper/2204.10851/) (2022)
- [A Survey on Temporal Graph Representation Learning and Generative Modeling](https://mlsanity.com/paper/2208.12126/) (2022)
- [UniRec: A Dual Enhancement of Uniformity and Frequency in Sequential Recommendations](https://mlsanity.com/paper/2406.18470/) (2024)

## Abstract (arXiv)

Sequential modelling with self-attention has achieved cutting edge performances in natural language processing. With advantages in model flexibility, computation complexity and interpretability, self-attention is gradually becoming a key component in event sequence models. However, like most other sequence models, self-attention does not account for the time span between events and thus captures sequential signals rather than temporal patterns. Without relying on recurrent network structures, self-attention recognizes event orderings via positional encoding. To bridge the gap between modelling time-independent and time-dependent event sequence, we introduce a functional feature map that embeds time span into high-dimensional spaces. By constructing the associated translation-invariant time kernel function, we reveal the functional forms of the feature map under classic functional function analysis results, namely Bochner's Theorem and Mercer's Theorem. We propose several models to learn the functional time representation and the interactions with event representation. These methods are evaluated on real-world datasets under various continuous-time event sequence prediction tasks. The experiments reveal that the proposed methods compare favorably to baseline models while also capturing useful time-event interactions.