# Learning 3D-aware Egocentric Spatial-Temporal Interaction via Graph Convolutional Networks

arXiv: https://arxiv.org/abs/1909.09272 · HTML: https://mlsanity.com/paper/1909.09272/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2019.

## Built on by (most cited first, 22 of 22)

- [Graph-based Topology Reasoning for Driving Scenes](https://mlsanity.com/paper/2304.05277/) (2023)
- [AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition](https://mlsanity.com/paper/2102.05775/) (2021)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction](https://mlsanity.com/paper/2111.06123/) (2021)
- [SCENE: Reasoning about Traffic Scenes using Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2301.03512/) (2023)
- [RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding](https://mlsanity.com/paper/2304.08600/) (2023)
- [roadscene2vec: A Tool for Extracting and Embedding Road Scene-Graphs](https://mlsanity.com/paper/2109.01183/) (2021)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Traffic](https://mlsanity.com/paper/2404.08561/) (2024)
- [RiskBench: A Scenario-based Benchmark for Risk Identification](https://mlsanity.com/paper/2312.01659/) (2023)
- [Survey on video anomaly detection in dynamic scenes with moving cameras](https://mlsanity.com/paper/2308.07050/) (2023)
- [DROID: Driver-centric Risk Object Identification](https://mlsanity.com/paper/2106.13201/) (2021)
- [Explainable Action Prediction through Self-Supervision on Scene Graphs](https://mlsanity.com/paper/2302.03477/) (2023)
- [Ego-Vehicle Action Recognition based on Semi-Supervised Contrastive Learning](https://mlsanity.com/paper/2303.00977/) (2023)
- [Learning Road Scene-level Representations via Semantic Region Prediction](https://mlsanity.com/paper/2301.00714/) (2023)
- [MEAT: Maneuver Extraction from Agent Trajectories](https://mlsanity.com/paper/2206.05158/) (2022)
- [RSG-Net: Towards Rich Sematic Relationship Prediction for Intelligent Vehicle in Complex Environments](https://mlsanity.com/paper/2207.12321/) (2022)
- [Potential Field as Scene Affordance for Behavior Change-Based Visual Risk Object Identification](https://mlsanity.com/paper/2409.15846/) (2024)
- [Transfer-LMR: Heavy-Tail Driving Behavior Recognition in Diverse Traffic Scenarios](https://mlsanity.com/paper/2405.05354/) (2024)
- [Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle](https://mlsanity.com/paper/2309.06313/) (2023)
- [Hazard-Aware Traffic Scene Graph Generation](https://mlsanity.com/paper/2603.03584/) (2026)
- [Uncertainty-Aware Vision-based Risk Object Identification via Conformal Risk Tube Prediction](https://mlsanity.com/paper/2603.23919/) (2026)

## Abstract (arXiv)

To enable intelligent automated driving systems, a promising strategy is to understand how human drives and interacts with road users in complicated driving situations. In this paper, we propose a 3D-aware egocentric spatial-temporal interaction framework for automated driving applications. Graph convolution networks (GCN) is devised for interaction modeling. We introduce three novel concepts into GCN. First, we decompose egocentric interactions into ego-thing and ego-stuff interaction, modeled by two GCNs. In both GCNs, ego nodes are introduced to encode the interaction between thing objects (e.g., car and pedestrian), and interaction between stuff objects (e.g., lane marking and traffic light). Second, objects' 3D locations are explicitly incorporated into GCN to better model egocentric interactions. Third, to implement ego-stuff interaction in GCN, we propose a MaskAlign operation to extract features for irregular objects. We validate the proposed framework on tactical driver behavior recognition. Extensive experiments are conducted using Honda Research Institute Driving Dataset, the largest dataset with diverse tactical driver behavior annotations. Our framework demonstrates substantial performance boost over baselines on the two experimental settings by 3.9% and 6.0%, respectively. Furthermore, we visualize the learned affinity matrices, which encode ego-thing and ego-stuff interactions, to showcase the proposed framework can capture interactions effectively.