# DanmakuTPPBench: A Multi-modal Benchmark for Temporal Point Process Modeling and Understanding

arXiv: https://arxiv.org/abs/2505.18411 · HTML: https://mlsanity.com/paper/2505.18411/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [Neural Spatio-Temporal Point Processes](https://mlsanity.com/paper/2011.04583/) (2020)
- [Transformer Embeddings of Irregularly Spaced Events and Their Participants](https://mlsanity.com/paper/2201.00044/) (2021)
- [Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement](https://mlsanity.com/paper/2503.01875/) (2025)
- [EasyTPP: Towards Open Benchmarking Temporal Point Processes](https://mlsanity.com/paper/2307.08097/) (2023)
- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://mlsanity.com/paper/2306.15681/) (2023)
- [Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning](https://mlsanity.com/paper/2305.16646/) (2023)
- [COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolution](https://mlsanity.com/paper/1507.02293/) (2015)
- [Expecting to be HIP: Hawkes Intensity Processes for Social Media Popularity](https://mlsanity.com/paper/1602.06033/) (2016)
- [CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generation](https://mlsanity.com/paper/2406.11451/) (2024)
- [Feature Driven and Point Process Approaches for Popularity Prediction](https://mlsanity.com/paper/1608.04862/) (2016)
- [TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Models](https://mlsanity.com/paper/2410.02062/) (2024)
- [Modeling Information Cascades with Self-exciting Processes via Generalized Epidemic Models](https://mlsanity.com/paper/1910.05451/) (2019)
- [Byte-token Enhanced Language Models for Temporal Point Processes Analysis](https://mlsanity.com/paper/2502.07139/) (2025)

## Built on by (most cited first, 7 of 7)

- [Self-Evolving Multi-Agent Systems via Textual Backpropagation](https://mlsanity.com/paper/2506.09046/) (2025)
- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)
- [SatireDecoder: Visual Cascaded Decoupling for Enhancing Satirical Image Comprehension](https://mlsanity.com/paper/2512.00582/) (2025)
- [MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue](https://mlsanity.com/paper/2606.00622/) (2026)
- [Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models](https://mlsanity.com/paper/2512.13618/) (2025)
- [Long-range Modeling and Processing of Multimodal Event Sequences](https://mlsanity.com/paper/2602.01125/) (2026)
- [Latent Block-Diffusion Temporal Point Processes: A Semi-Autoregressive Framework for Asynchronous Event Sequence Generation](https://mlsanity.com/paper/2606.24982/) (2026)

## Abstract (arXiv)

We introduce DanmakuTPPBench, a comprehensive benchmark designed to advance multi-modal Temporal Point Process (TPP) modeling in the era of Large Language Models (LLMs). While TPPs have been widely studied for modeling temporal event sequences, existing datasets are predominantly unimodal, hindering progress in models that require joint reasoning over temporal, textual, and visual information. To address this gap, DanmakuTPPBench comprises two complementary components: (1) DanmakuTPP-Events, a novel dataset derived from the Bilibili video platform, where user-generated bullet comments (Danmaku) naturally form multi-modal events annotated with precise timestamps, rich textual content, and corresponding video frames; (2) DanmakuTPP-QA, a challenging question-answering dataset constructed via a novel multi-agent pipeline powered by state-of-the-art LLMs and multi-modal LLMs (MLLMs), targeting complex temporal-textual-visual reasoning. We conduct extensive evaluations using both classical TPP models and recent MLLMs, revealing significant performance gaps and limitations in current methods' ability to model multi-modal event dynamics. Our benchmark establishes strong baselines and calls for further integration of TPP modeling into the multi-modal language modeling landscape. Project page: https://github.com/FRENKIE-CHIANG/DanmakuTPPBench