mlsanityData to 7 Oct 2026

Transformer Embeddings of Irregularly Spaced Events and Their Participants

arXiv 2201.00044 ↗First posted 31 Dec 2021cs.LG cs.AI cs.LOcited by 90

Not on the map. Why?

Abstract (arXiv)

The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and the possible event has a time-varying embedding that is derived from its symbolic provenance. In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

23 of 51 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2016Layer Normalizationcited by 5,579
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  9. 2020Transformer Hawkes Processcited by 230
  10. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  11. 2019Intensity-Free Learning of Temporal Point Processescited by 138
  12. 2019Fully Neural Network based Model for General Temporal Point Processescited by 119
11 more
  1. 2017Modeling The Intensity Function Of Point Process Via Recurrent Neural Networkscited by 93
  2. 2016Learning Granger Causality for Hawkes Processescited by 68
  3. 2020Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviourscited by 46
  4. 2019Imputing Missing Events in Continuous-Time Event Streamscited by 43
  5. 2020Neural Temporal Point Processes For Modelling Electronic Health Recordscited by 39
  6. 2020CAUSE: Learning Granger Causality from Event Sequences using Attribution Methodscited by 36
  7. 2020Noise-Contrastive Estimation for Multivariate Point Processescited by 20
  8. 2020User-Dependent Neural Sequence Models for Continuous-Time Event Datacited by 17
  9. 2016Learning Network of Multivariate Hawkes Processes: A Time Series Approachcited by 15
  10. 2020Neural Datalog Through Time: Informed Temporal Modeling via Logical Specificationcited by 10
  11. 2017Joint Modeling of Event Sequence and Time Series with Attentional Twin Recurrent Neural Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 21; 2024: 25; 2025: 21; 2026: 20.

90 papers.

Sort:

By area:

  1. 2022Transformers in Time Series: A Surveycited by 489
  2. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  3. 2023EasyTPP: Towards Open Benchmarking Temporal Point Processescited by 63
  4. 2022HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequencescited by 57
  5. 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 50
  6. 2023Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoningcited by 44
  7. 2022Exploring Generative Neural Temporal Point Processcited by 30
  8. 2023Prompt-augmented Temporal Point Process for Streaming Event Sequencecited by 24
  9. 2023Meta Temporal Point Processescited by 16
  10. 2024Decomposable Transformer Point Processescited by 15
  11. 2023Real-time Network Intrusion Detection via Decision Transformerscited by 12
  12. 2024HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?cited by 11
38 more
  1. 2023A Systematic Review for Transformer-based Long-term Series Forecastingcited by 10
  2. 2023Interacting Diffusion Processes for Event Sequence Forecastingcited by 9
  3. 2024Latent Logic Tree Extraction for Event Sequence Explanation from LLMscited by 9
  4. 2024Deep Continuous-Time State-Space Models for Marked Event Sequencescited by 9
  5. 2024TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Modelscited by 8
  6. 2024EventFlow: Forecasting Temporal Point Processes with Flow Matchingcited by 8
  7. 2023Non-Autoregressive Diffusion-based Temporal Point Processes for Continuous-Time Long-Term Event Predictioncited by 7
  8. 2024Distribution-Free Conformal Joint Prediction Regions for Neural Marked Temporal Point Processescited by 7
  9. 2025DanmakuTPPBench: A Multi-modal Benchmark for Temporal Point Process Modeling and Understandingcited by 7
  10. 2025In-Context Learning of Temporal Point Processes with Foundation Inference Modelscited by 7
  11. 2025Edit-Based Flow Matching for Temporal Point Processescited by 7
  12. 2023Intensity-free Convolutional Temporal Point Process: Incorporating Local and Global Event Contextscited by 6
  13. 2024Decoupled Marked Temporal Point Process using Neural Ordinary Differential Equationscited by 6
  14. 2024Learning Transactions Representations for Information Management in Banks: Mastering Local, Global, and External Knowledgecited by 5
  15. 2024Detecting the Future: All-at-Once Event Sequence Forecasting with Horizon Matchingcited by 5
  16. 2025Byte-token Enhanced Language Models for Temporal Point Processes Analysiscited by 5
  17. 2025LAST SToP For Modeling Asynchronous Time Seriescited by 4
  18. 2024TransformerLSR: Attentive Joint Model of Longitudinal Data, Survival, and Recurrent Events with Concurrent Latent Structurecited by 3
  19. 2025HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokenscited by 3
  20. 2023Label Attention Network for Temporal Sets Prediction: You Were Looking at a Wrong Self-Attentioncited by 2
  21. 2023Proactive Resource Request for Disaster Response: A Deep Learning-based Optimization Modelcited by 2
  22. 2023Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identificationcited by 2
  23. 2024Self-Supervised Contrastive Pre-Training for Multivariate Point Processescited by 2
  24. 2024Nonparametric estimation of Hawkes processes with RKHSscited by 2
  25. 2025Fine-grained Spatio-temporal Event Prediction with Self-adaptive Anchor Graphcited by 2
  26. 2025Advances in Temporal Point Processes: Bayesian, Neural, and LLM Approachescited by 2
  27. 2025Balancing Interpretability and Flexibility in Modeling Diagnostic Trajectories with an Embedded Neural Hawkes Process Modelcited by 2
  28. 2025TPP-SD: Accelerating Transformer Point Process Sampling with Speculative Decodingcited by 2
  29. 2023Tapestry of Time and Actions: Modeling Human Activity Sequences using Temporal Point Process Flowscited by 1
  30. 2023YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUscited by 1
  31. 2024Pretext Training Algorithms for Event Sequence Datacited by 1
  32. 2024A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformerscited by 1
  33. 2024A Bayesian Mixture Model of Temporal Point Processes with Determinantal Point Process Priorcited by 1
  34. 2025ADiff4TPP: Asynchronous Diffusion Models for Temporal Point Processescited by 1
  35. 2025Still Competitive: Revisiting Recurrent Models for Irregular Time Series Predictioncited by 1
  36. 2023Continuous-time convolutions model of event sequencescited by 0
  37. 2023Enhancing Asynchronous Time Series Forecasting with Contrastive Relational Inferencecited by 0
  38. 2023Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adaptercited by 0

50 most cited of 90 All 90 (JSON)