# Generative Hybrid Representations for Activity Forecasting with No-Regret Learning

arXiv: https://arxiv.org/abs/1904.06250 · HTML: https://mlsanity.com/paper/1904.06250/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 18 of 18)

- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [EPIC Fields: Marrying 3D Geometry and Video Understanding](https://mlsanity.com/paper/2306.08731/) (2023)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Multi-label affordance mapping from egocentric vision](https://mlsanity.com/paper/2309.02120/) (2023)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Generative Adversarial Network for Future Hand Segmentation from Egocentric Video](https://mlsanity.com/paper/2203.11305/) (2022)
- [HYPER: Learned Hybrid Trajectory Prediction via Factored Inference and Adaptive Sampling](https://mlsanity.com/paper/2110.02344/) (2021)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction](https://mlsanity.com/paper/2402.12238/) (2024)
- [Rich Action-semantic Consistent Knowledge for Early Action Prediction](https://mlsanity.com/paper/2201.09169/) (2022)
- [Efficient Human Vision Inspired Action Recognition using Adaptive Spatiotemporal Sampling](https://mlsanity.com/paper/2207.05249/) (2022)
- [Egocentric Activity Recognition and Localization on a 3D Map](https://mlsanity.com/paper/2105.09544/) (2021)
- [Unified Simulation, Perception, and Generation of Human Behavior](https://mlsanity.com/paper/2204.13678/) (2022)
- [Finding Islands of Predictability in Action Forecasting](https://mlsanity.com/paper/2210.07354/) (2022)
- [Learning Socio-Temporal Graphs for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2312.14373/) (2023)
- [Explainable Artificial Intelligence for Quantifying Interfering and High-Risk Behaviors in Autism Spectrum Disorder in a Real-World Classroom Environment Using Privacy-Preserving Video Analysis](https://mlsanity.com/paper/2407.21691/) (2024)

## Abstract (arXiv)

Automatically reasoning about future human behaviors is a difficult problem but has significant practical applications to assistive systems. Part of this difficulty stems from learning systems' inability to represent all kinds of behaviors. Some behaviors, such as motion, are best described with continuous representations, whereas others, such as picking up a cup, are best described with discrete representations. Furthermore, human behavior is generally not fixed: people can change their habits and routines. This suggests these systems must be able to learn and adapt continuously. In this work, we develop an efficient deep generative model to jointly forecast a person's future discrete actions and continuous motions. On a large-scale egocentric dataset, EPIC-KITCHENS, we observe our method generates high-quality and diverse samples while exhibiting better generalization than related generative models. Finally, we propose a variant to continually learn our model from streaming data, observe its practical effectiveness, and theoretically justify its learning efficiency.