# RoTHP: Rotary Position Embedding-based Transformer Hawkes Process

arXiv: https://arxiv.org/abs/2405.06985 · HTML: https://mlsanity.com/paper/2405.06985/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 11 May 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [Neural Temporal Point Processes: A Review](https://mlsanity.com/paper/2104.03528/) (2021)
- [Modeling The Intensity Function Of Point Process Via Recurrent Neural Networks](https://mlsanity.com/paper/1705.08982/) (2017)
- [SEISMIC: A Self-Exciting Point Process Model for Predicting Tweet Popularity](https://mlsanity.com/paper/1506.02594/) (2015)
- [SNAP: A General Purpose Network Analysis and Graph Mining Library](https://mlsanity.com/paper/1606.07550/) (2016)
- [Wasserstein Learning of Deep Generative Point Process Models](https://mlsanity.com/paper/1705.08051/) (2017)
- [Lecture Notes: Temporal Point Processes and the Conditional Intensity Function](https://mlsanity.com/paper/1806.00221/) (2018)
- [Learning Temporal Point Processes via Reinforcement Learning](https://mlsanity.com/paper/1811.05016/) (2018)
- [Shaping Social Activity by Incentivizing Users](https://mlsanity.com/paper/1408.0406/) (2014)
- [Temporal Attention Augmented Transformer Hawkes Process](https://mlsanity.com/paper/2112.14472/) (2021)
- [Universal Transformer Hawkes Process with Adaptive Recursive Iteration](https://mlsanity.com/paper/2112.14479/) (2021)

## Built on by (most cited first, 3 of 3)

- [TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Models](https://mlsanity.com/paper/2410.02062/) (2024)
- [Mamba Hawkes Process](https://mlsanity.com/paper/2407.05302/) (2024)
- [Enhancing Temporal Awareness in LLMs for Temporal Point Processes](https://mlsanity.com/paper/2601.00845/) (2025)

## Abstract (arXiv)

Temporal Point Processes (TPPs), especially Hawkes Process are commonly used for modeling asynchronous event sequences data such as financial transactions and user behaviors in social networks. Due to the strong fitting ability of neural networks, various neural Temporal Point Processes are proposed, among which the Neural Hawkes Processes based on self-attention such as Transformer Hawkes Process (THP) achieve distinct performance improvement. Although the THP has gained increasing studies, it still suffers from the {sequence prediction issue}, i.e., training on history sequences and inferencing about the future, which is a prevalent paradigm in realistic sequence analysis tasks. What's more, conventional THP and its variants simply adopt initial sinusoid embedding in transformers, which shows performance sensitivity to temporal change or noise in sequence data analysis by our empirical study. To deal with the problems, we propose a new Rotary Position Embedding-based THP (RoTHP) architecture in this paper. Notably, we show the translation invariance property and {sequence prediction flexibility} of our RoTHP induced by the {relative time embeddings} when coupled with Hawkes process theoretically. Furthermore, we demonstrate empirically that our RoTHP can be better generalized in sequence data scenarios with timestamp translations and in sequence prediction tasks.