# Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective

arXiv: https://arxiv.org/abs/2111.14820 · HTML: https://mlsanity.com/paper/2111.14820/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 98 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 52)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning](https://mlsanity.com/paper/1702.08811/) (2017)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)

## Built on by (most cited first, 30 of 40)

- [SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation](https://mlsanity.com/paper/2304.10417/) (2023)
- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion](https://mlsanity.com/paper/2312.04466/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning](https://mlsanity.com/paper/2301.05169/) (2023)
- [Causal Composition Diffusion Model for Closed-loop Traffic Generation](https://mlsanity.com/paper/2412.17920/) (2024)
- [MotionScript: Natural Language Descriptions for Expressive 3D Human Motions](https://mlsanity.com/paper/2312.12634/) (2023)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory](https://mlsanity.com/paper/2403.10052/) (2024)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Causal Discovery of Dynamic Models for Predicting Human Spatial Interactions](https://mlsanity.com/paper/2210.16535/) (2022)
- [A Review of the Role of Causality in Developing Trustworthy AI Systems](https://mlsanity.com/paper/2302.06975/) (2023)
- [Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting](https://mlsanity.com/paper/2302.08635/) (2023)
- [Safety-compliant Generative Adversarial Networks for Human Trajectory Forecasting](https://mlsanity.com/paper/2209.12243/) (2022)
- [Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting](https://mlsanity.com/paper/2211.03165/) (2022)
- [Intention-aware Denoising Diffusion Model for Trajectory Prediction](https://mlsanity.com/paper/2403.09190/) (2024)
- [Spatio-temporal neural structural causal models for bike flow prediction](https://mlsanity.com/paper/2301.07843/) (2023)
- [Curb Your Attention: Causal Attention Gating for Robust Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2410.07191/) (2024)
- [Vision-based Multi-future Trajectory Prediction: A Survey](https://mlsanity.com/paper/2302.10463/) (2023)
- [CaT-GNN: Enhancing Credit Card Fraud Detection via Causal Temporal Graph Neural Networks](https://mlsanity.com/paper/2402.14708/) (2024)
- [Vision Paper: Causal Inference for Interpretable and Robust Machine Learning in Mobility Analysis](https://mlsanity.com/paper/2210.10010/) (2022)
- [Vision-Language Models Assisted Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2409.14109/) (2024)
- [Multimodal Clickbait Detection by De-confounding Biases Using Causal Representation Inference](https://mlsanity.com/paper/2410.07673/) (2024)
- [A Game-Theoretic Framework for Joint Forecasting and Planning](https://mlsanity.com/paper/2308.06137/) (2023)
- [AdapTraj: A Multi-Source Domain Generalization Framework for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2312.14394/) (2023)
- [Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representations](https://mlsanity.com/paper/2312.04540/) (2023)
- [LoRD: Adapting Differentiable Driving Policies to Distribution Shifts](https://mlsanity.com/paper/2410.09681/) (2024)
- [Experimental Evaluation of ROS-Causal in Real-World Human-Robot Spatial Interaction Scenarios](https://mlsanity.com/paper/2406.04955/) (2024)
- [Towards Generalizable PDE Dynamics Forecasting via Physics-Guided Invariant Learning](https://mlsanity.com/paper/2509.24332/) (2025)
- [CILF:Causality Inspired Learning Framework for Out-of-Distribution Vehicle Trajectory Prediction](https://mlsanity.com/paper/2307.05624/) (2023)

## Abstract (arXiv)

Learning behavioral patterns from observational data has been a de-facto approach to motion forecasting. Yet, the current paradigm suffers from two shortcomings: brittle under distribution shifts and inefficient for knowledge transfer. In this work, we propose to address these challenges from a causal representation perspective. We first introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables, namely invariant variables, style confounders, and spurious features. We then introduce a learning framework that treats each group separately: (i) unlike the common practice mixing datasets collected from different locations, we exploit their subtle distinctions by means of an invariance loss encouraging the model to suppress spurious correlations; (ii) we devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a sparse causal graph; (iii) we introduce a style contrastive loss that not only enforces the structure of style representations but also serves as a self-supervisory signal for test-time refinement on the fly. Experiments on synthetic and real datasets show that our proposed method improves the robustness and reusability of learned motion representations, significantly outperforming prior state-of-the-art motion forecasting models for out-of-distribution generalization and low-shot transfer.