# Trajectory prediction for heterogeneous agents: A performance analysis on small and imbalanced datasets

arXiv: https://arxiv.org/abs/2510.03776 · HTML: https://mlsanity.com/paper/2510.03776/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions](https://mlsanity.com/paper/1812.04767/) (2018)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [TH"OR: Human-Robot Navigation Data Collection and Accurate Motion Trajectories Dataset](https://mlsanity.com/paper/1909.04403/) (2019)
- [Joint Inference of Groups, Events and Human Roles in Aerial Videos](https://mlsanity.com/paper/1505.05957/) (2015)
- [MoGaze: A Dataset of Full-Body Motions that Includes Workspace Geometry and Eye-Gaze](https://mlsanity.com/paper/2011.11552/) (2020)
- [Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty](https://mlsanity.com/paper/2104.12446/) (2021)
- [Unlimited Neighborhood Interaction for Heterogeneous Trajectory Prediction](https://mlsanity.com/paper/2108.00238/) (2021)
- [Safety-compliant Generative Adversarial Networks for Human Trajectory Forecasting](https://mlsanity.com/paper/2209.12243/) (2022)
- [Heterogeneous Trajectory Forecasting via Risk and Scene Graph Learning](https://mlsanity.com/paper/2211.00848/) (2022)
- [The Magni Human Motion Dataset: Accurate, Complex, Multi-Modal, Natural, Semantically-Rich and Contextualized](https://mlsanity.com/paper/2208.14925/) (2022)
- [The Atlas Benchmark: an Automated Evaluation Framework for Human Motion Prediction](https://mlsanity.com/paper/2207.09830/) (2022)
- [Semantics-STGCNN: A Semantics-guided Spatial-Temporal Graph Convolutional Network for Multi-class Trajectory Prediction](https://mlsanity.com/paper/2108.04740/) (2021)
- [Learning World Transition Model for Socially Aware Robot Navigation](https://mlsanity.com/paper/2011.03922/) (2020)
- [CLiFF-LHMP: Using Spatial Dynamics Patterns for Long-Term Human Motion Prediction](https://mlsanity.com/paper/2309.07066/) (2023)
- [Vehicle Motion Forecasting using Prior Information and Semantic-assisted Occupancy Grid Maps](https://mlsanity.com/paper/2308.04303/) (2023)
- [A Data-Efficient Approach for Long-Term Human Motion Prediction Using Maps of Dynamics](https://mlsanity.com/paper/2306.03617/) (2023)

## Abstract (arXiv)

Robots and other intelligent systems navigating in complex dynamic environments should predict future actions and intentions of surrounding agents to reach their goals efficiently and avoid collisions. The dynamics of those agents strongly depends on their tasks, roles, or observable labels. Class-conditioned motion prediction is thus an appealing way to reduce forecast uncertainty and get more accurate predictions for heterogeneous agents. However, this is hardly explored in the prior art, especially for mobile robots and in limited data applications. In this paper, we analyse different class-conditioned trajectory prediction methods on two datasets. We propose a set of conditional pattern-based and efficient deep learning-based baselines, and evaluate their performance on robotics and outdoors datasets (TH"OR-MAGNI and Stanford Drone Dataset). Our experiments show that all methods improve accuracy in most of the settings when considering class labels. More importantly, we observe that there are significant differences when learning from imbalanced datasets, or in new environments where sufficient data is not available. In particular, we find that deep learning methods perform better on balanced datasets, but in applications with limited data, e.g., cold start of a robot in a new environment, or imbalanced classes, pattern-based methods may be preferable.