# Pre-training Contextual Location Embeddings in Personal Trajectories via Efficient Hierarchical Location Representations

arXiv: https://arxiv.org/abs/2310.01252 · HTML: https://mlsanity.com/paper/2310.01252/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DeepMove: Learning Place Representations through Large Scale Movement Data](https://mlsanity.com/paper/1807.04241/) (2018)
- [Time-Dependent Representation for Neural Event Sequence Prediction](https://mlsanity.com/paper/1708.00065/) (2017)
- [Learning Fine Grained Place Embeddings with Spatial Hierarchy from Human Mobility Trajectories](https://mlsanity.com/paper/2002.02058/) (2020)
- [Hierarchical Multitask Learning Approach for BERT](https://mlsanity.com/paper/2011.04451/) (2020)
- [GT-SEER: Geo-Temporal SEquential Embedding Rank for Point-of-interest Recommendation](https://mlsanity.com/paper/1606.05859/) (2016)

## Built on by (most cited first, 3 of 3)

- [Self-Supervised Representation Learning for Geospatial Objects: A Survey](https://mlsanity.com/paper/2408.12133/) (2024)
- [FedGeo: Privacy-Preserving User Next Location Prediction with Federated Learning](https://mlsanity.com/paper/2312.04594/) (2023)
- [NLP-enabled Trajectory Map-matching in Urban Road Networks using a Transformer-based Encoder-decoder](https://mlsanity.com/paper/2404.12460/) (2024)

## Abstract (arXiv)

Pre-training the embedding of a location generated from human mobility data has become a popular method for location based services. In practice, modeling the location embedding is too expensive, due to the large number of locations to be trained in situations with fine-grained resolution or extensive target regions. Previous studies have handled less than ten thousand distinct locations, which is insufficient in the real-world applications. To tackle this problem, we propose a Geo-Tokenizer, designed to efficiently reduce the number of locations to be trained by representing a location as a combination of several grids at different scales. In the Geo-Tokenizer, a grid at a larger scale shares the common set of grids at smaller scales, which is a key factor in reducing the size of the location vocabulary. The sequences of locations preprocessed with the Geo-Tokenizer are utilized by a causal location embedding model to capture the temporal dependencies of locations. This model dynamically calculates the embedding vector of a target location, which varies depending on its trajectory. In addition, to efficiently pre-train the location embedding model, we propose the Hierarchical Auto-regressive Location Model objective to effectively train decomposed locations in the Geo-Tokenizer. We conducted experiments on two real-world user trajectory datasets using our pre-trained location model. The experimental results show that our model significantly improves the performance of downstream tasks with fewer model parameters compared to existing location embedding methods.