# Temporal Attention Augmented Transformer Hawkes Process

arXiv: https://arxiv.org/abs/2112.14472 · HTML: https://mlsanity.com/paper/2112.14472/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 29 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [Modeling The Intensity Function Of Point Process Via Recurrent Neural Networks](https://mlsanity.com/paper/1705.08982/) (2017)
- [SEISMIC: A Self-Exciting Point Process Model for Predicting Tweet Popularity](https://mlsanity.com/paper/1506.02594/) (2015)
- [Learning Granger Causality for Hawkes Processes](https://mlsanity.com/paper/1602.04511/) (2016)
- [Language Models with Transformers](https://mlsanity.com/paper/1904.09408/) (2019)
- [TiDeH: Time-Dependent Hawkes Process for Predicting Retweet Dynamics](https://mlsanity.com/paper/1603.09449/) (2016)

## Built on by (most cited first, 5 of 5)

- [Mamba Hawkes Process](https://mlsanity.com/paper/2407.05302/) (2024)
- [Exploiting Session Information in BERT-based Session-aware Sequential Recommendation](https://mlsanity.com/paper/2204.10851/) (2022)
- [RoTHP: Rotary Position Embedding-based Transformer Hawkes Process](https://mlsanity.com/paper/2405.06985/) (2024)
- [Cox-Hawkes: doubly stochastic spatiotemporal Poisson processes](https://mlsanity.com/paper/2210.11844/) (2022)
- [Uncovering Causal Relation Shifts in Event Sequences under Out-of-Domain Interventions](https://mlsanity.com/paper/2507.10809/) (2025)

## Abstract (arXiv)

In recent years, mining the knowledge from asynchronous sequences by Hawkes process is a subject worthy of continued attention, and Hawkes processes based on the neural network have gradually become the most hotly researched fields, especially based on the recurrence neural network (RNN). However, these models still contain some inherent shortcomings of RNN, such as vanishing and exploding gradient and long-term dependency problems. Meanwhile, Transformer based on self-attention has achieved great success in sequential modeling like text processing and speech recognition. Although the Transformer Hawkes process (THP) has gained huge performance improvement, THPs do not effectively utilize the temporal information in the asynchronous events, for these asynchronous sequences, the event occurrence instants are as important as the types of events, while conventional THPs simply convert temporal information into position encoding and add them as the input of transformer. With this in mind, we come up with a new kind of Transformer-based Hawkes process model, Temporal Attention Augmented Transformer Hawkes Process (TAA-THP), we modify the traditional dot-product attention structure, and introduce the temporal encoding into attention structure. We conduct numerous experiments on a wide range of synthetic and real-life datasets to validate the performance of our proposed TAA-THP model, significantly improvement compared with existing baseline models on the different measurements is achieved, including log-likelihood on the test dataset, and prediction accuracies of event types and occurrence times. In addition, through the ablation studies, we vividly demonstrate the merit of introducing additional temporal attention by comparing the performance of the model with and without temporal attention.