# Pre-trained Transformer Uncovers Meaningful Patterns in Human Mobility Data

arXiv: https://arxiv.org/abs/2406.04029 · HTML: https://mlsanity.com/paper/2406.04029/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting](https://mlsanity.com/paper/2210.08964/) (2022)
- [Where Would I Go Next? Large Language Models as Human Mobility Predictors](https://mlsanity.com/paper/2308.15197/) (2023)
- [On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2304.06798/) (2023)
- [Leveraging Language Foundation Models for Human Mobility Forecasting](https://mlsanity.com/paper/2209.05479/) (2022)
- [Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding](https://mlsanity.com/paper/2001.11101/) (2020)
- [Exploring Large Language Models for Human Mobility Prediction under Public Events](https://mlsanity.com/paper/2311.17351/) (2023)
- [Large Language Models for Spatial Trajectory Patterns Mining](https://mlsanity.com/paper/2310.04942/) (2023)
- [Hex2vec -- Context-Aware Embedding H3 Hexagons with OpenStreetMap Tags](https://mlsanity.com/paper/2111.00970/) (2021)
- [Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models](https://mlsanity.com/paper/2309.17319/) (2023)
- [Towards A Foundation Model For Trajectory Intelligence](https://mlsanity.com/paper/2312.00076/) (2023)
- [Trajectory-User Linking Is Easier Than You Think](https://mlsanity.com/paper/2212.07081/) (2022)
- [Where You Are Is What You Do: On Inferring Offline Activities From Location Data](https://mlsanity.com/paper/2301.13537/) (2023)

## Built on by (most cited first, 1 of 1)

- [Embodied AI-Enhanced IoMT Edge Computing: UAV Trajectory Optimization and Task Offloading with Mobility Prediction](https://mlsanity.com/paper/2512.20902/) (2025)

## Abstract (arXiv)

We empirically demonstrate that a transformer pre-trained on country-scale unlabeled human mobility data learns embeddings capable, through fine-tuning, of developing a deep understanding of the target geography and its corresponding mobility patterns. Utilizing an adaptation framework, we evaluate the performance of our pre-trained embeddings in encapsulating a broad spectrum of concepts directly and indirectly related to human mobility. This includes basic notions, such as geographic location and distance, and extends to more complex constructs, such as administrative divisions and land cover. Our extensive empirical analysis reveals a substantial performance boost gained from pre-training, reaching up to 38% in tasks such as tree-cover regression. We attribute this result to the ability of the pre-training to uncover meaningful patterns hidden in the raw data, beneficial for modeling relevant high-level concepts. The pre-trained embeddings emerge as robust representations of regions and trajectories, potentially valuable for a wide range of downstream applications.