# Encoding Agent Trajectories as Representations with Sequence Transformers

arXiv: https://arxiv.org/abs/2410.09204 · HTML: https://mlsanity.com/paper/2410.09204/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Graph Neural Network for Traffic Forecasting: A Survey](https://mlsanity.com/paper/2101.11174/) (2021)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [OSMnx: New Methods for Acquiring, Constructing, Analyzing, and Visualizing Complex Street Networks](https://mlsanity.com/paper/1611.01890/) (2016)
- [A Survey on Deep Learning for Human Mobility](https://mlsanity.com/paper/2012.02825/) (2020)
- [Leveraging Language Foundation Models for Human Mobility Forecasting](https://mlsanity.com/paper/2209.05479/) (2022)
- [MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction](https://mlsanity.com/paper/2110.01401/) (2021)
- [SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation](https://mlsanity.com/paper/2210.12213/) (2022)
- [RNTrajRec: Road Network Enhanced Trajectory Recovery with Spatial-Temporal Transformer](https://mlsanity.com/paper/2211.13234/) (2022)
- [Artificial Neural Networks Applied to Taxi Destination Prediction](https://mlsanity.com/paper/1508.00021/) (2015)
- [How do you go where? Improving next location prediction by learning travel mode information using transformers](https://mlsanity.com/paper/2210.04095/) (2022)
- [Revisiting Mobility Modeling with Graph: A Graph Transformer Model for Next Point-of-Interest Recommendation](https://mlsanity.com/paper/2310.01224/) (2023)
- [Predicting Vehicles Trajectories in Urban Scenarios with Transformer Networks and Augmented Information](https://mlsanity.com/paper/2106.00559/) (2021)
- [Cross-attention Spatio-temporal Context Transformer for Semantic Segmentation of Historical Maps](https://mlsanity.com/paper/2310.12616/) (2023)
- [PlanetSense: A Real-time Streaming and Spatio-temporal Analytics Platform for Gathering Geo-spatial Intelligence from Open Source Data](https://mlsanity.com/paper/1507.05245/) (2015)

## Abstract (arXiv)

Spatiotemporal data faces many analogous challenges to natural language text including the ordering of locations (words) in a sequence, long range dependencies between locations, and locations having multiple meanings. In this work, we propose a novel model for representing high dimensional spatiotemporal trajectories as sequences of discrete locations and encoding them with a Transformer-based neural network architecture. Similar to language models, our Sequence Transformer for Agent Representation Encodings (STARE) model can learn representations and structure in trajectory data through both supervisory tasks (e.g., classification), and self-supervisory tasks (e.g., masked modelling). We present experimental results on various synthetic and real trajectory datasets and show that our proposed model can learn meaningful encodings that are useful for many downstream tasks including discriminating between labels and indicating similarity between locations. Using these encodings, we also learn relationships between agents and locations present in spatiotemporal data.