# Discovering Intrinsic Spatial-Temporal Logic Rules to Explain Human Actions

arXiv: https://arxiv.org/abs/2306.12244 · HTML: https://mlsanity.com/paper/2306.12244/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion](https://mlsanity.com/paper/2203.13777/) (2022)
- [DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs](https://mlsanity.com/paper/1911.00055/) (2019)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2204.08770/) (2022)
- [RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2010.04029/) (2020)
- [Neural Spatio-Temporal Point Processes](https://mlsanity.com/paper/2011.04583/) (2020)
- [Boolean Decision Rules via Column Generation](https://mlsanity.com/paper/1805.09901/) (2018)
- [SocialVAE: Human Trajectory Prediction using Timewise Latents](https://mlsanity.com/paper/2203.08207/) (2022)
- [Human Trajectory Prediction via Neural Social Physics](https://mlsanity.com/paper/2207.10435/) (2022)
- [Collaborative Motion Prediction via Neural Motion Message Passing](https://mlsanity.com/paper/2003.06594/) (2020)
- [CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations](https://mlsanity.com/paper/2008.02792/) (2020)
- [Neural Point Process for Learning Spatiotemporal Event Dynamics](https://mlsanity.com/paper/2112.06351/) (2021)
- [A Review of Self-Exciting Spatio-Temporal Point Processes and Their Applications](https://mlsanity.com/paper/1708.02647/) (2017)
- [Social-Implicit: Rethinking Trajectory Prediction Evaluation and The Effectiveness of Implicit Maximum Likelihood Estimation](https://mlsanity.com/paper/2203.03057/) (2022)
- [Collaborative Uncertainty in Multi-Agent Trajectory Forecasting](https://mlsanity.com/paper/2110.13947/) (2021)
- [Action-based Contrastive Learning for Trajectory Prediction](https://mlsanity.com/paper/2207.08664/) (2022)
- [baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling](https://mlsanity.com/paper/2102.03291/) (2021)
- [Factorized Point Process Intensities: A Spatial Analysis of Professional Basketball](https://mlsanity.com/paper/1401.0942/) (2014)

## Built on by (most cited first, 3 of 3)

- [Improving Trajectory Prediction in Dynamic Multi-Agent Environment by Dropping Waypoints](https://mlsanity.com/paper/2309.17338/) (2023)
- [Enhancing Trajectory Prediction through Self-Supervised Waypoint Noise Prediction](https://mlsanity.com/paper/2312.09466/) (2023)
- [Explainable Human Activity Recognition: A Unified Review of Concepts and Mechanisms](https://mlsanity.com/paper/2604.09799/) (2026)

## Abstract (arXiv)

We propose a logic-informed knowledge-driven modeling framework for human movements by analyzing their trajectories. Our approach is inspired by the fact that human actions are usually driven by their intentions or desires, and are influenced by environmental factors such as the spatial relationships with surrounding objects. In this paper, we introduce a set of spatial-temporal logic rules as knowledge to explain human actions. These rules will be automatically discovered from observational data. To learn the model parameters and the rule content, we design an expectation-maximization (EM) algorithm, which treats the rule content as latent variables. The EM algorithm alternates between the E-step and M-step: in the E-step, the posterior distribution over the latent rule content is evaluated; in the M-step, the rule generator and model parameters are jointly optimized by maximizing the current expected log-likelihood. Our model may have a wide range of applications in areas such as sports analytics, robotics, and autonomous cars, where understanding human movements are essential. We demonstrate the model's superior interpretability and prediction performance on pedestrian and NBA basketball player datasets, both achieving promising results.