# MM-Path: Multi-modal, Multi-granularity Path Representation Learning -- Extended Version

arXiv: https://arxiv.org/abs/2411.18428 · HTML: https://mlsanity.com/paper/2411.18428/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction](https://mlsanity.com/paper/2402.11838/) (2024)
- [DiffTraj: Generating GPS Trajectory with Diffusion Probabilistic Model](https://mlsanity.com/paper/2304.11582/) (2023)
- [Self-supervised Trajectory Representation Learning with Temporal Regularities and Travel Semantics](https://mlsanity.com/paper/2211.09510/) (2022)
- [ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion Model](https://mlsanity.com/paper/2404.15380/) (2024)
- [Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding](https://mlsanity.com/paper/2001.11101/) (2020)
- [Contrastive Trajectory Similarity Learning with Dual-Feature Attention](https://mlsanity.com/paper/2210.05155/) (2022)
- [LightPath: Lightweight and Scalable Path Representation Learning](https://mlsanity.com/paper/2307.10171/) (2023)
- [i-Code: An Integrative and Composable Multimodal Learning Framework](https://mlsanity.com/paper/2205.01818/) (2022)
- [More Than Routing: Joint GPS and Route Modeling for Refine Trajectory Representation Learning](https://mlsanity.com/paper/2402.16915/) (2024)
- [LightTR: A Lightweight Framework for Federated Trajectory Recovery](https://mlsanity.com/paper/2405.03409/) (2024)
- [FlashST: A Simple and Universal Prompt-Tuning Framework for Traffic Prediction](https://mlsanity.com/paper/2405.17898/) (2024)
- [Weakly-supervised Temporal Path Representation Learning with Contrastive Curriculum Learning -- Extended Version](https://mlsanity.com/paper/2203.16110/) (2022)
- [Learning to Route with Sparse Trajectory Sets---Extended Version](https://mlsanity.com/paper/1802.07980/) (2018)

## Built on by (most cited first, 14 of 14)

- [FlexiReg: Flexible Urban Region Representation Learning](https://mlsanity.com/paper/2503.09128/) (2025)
- [Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review](https://mlsanity.com/paper/2506.01364/) (2025)
- [ARROW: An Adaptive Rollout and Routing Method for Global Weather Forecasting](https://mlsanity.com/paper/2510.09734/) (2025)
- [STAR: Boosting Time Series Foundation Models for Anomaly Detection through State-aware Adapter](https://mlsanity.com/paper/2510.16014/) (2025)
- [Spatio-Temporal Trajectory Foundation Model - Recent Advances and Future Directions](https://mlsanity.com/paper/2511.20729/) (2025)
- [Data Driven Decision Making with Time Series and Spatio-temporal Data](https://mlsanity.com/paper/2503.08473/) (2025)
- [MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification](https://mlsanity.com/paper/2509.17809/) (2025)
- [TiWeaver: Unified Temporal Dynamics Modeling via Contextual Patching](https://mlsanity.com/paper/2606.03121/) (2026)
- [Learning Generalized and Flexible Trajectory Models from Omni-Semantic Supervision](https://mlsanity.com/paper/2505.17437/) (2025)
- [REFINE: Trajectory Representation Learning via Closed-Loop Transcription -- Extended Version](https://mlsanity.com/paper/2609.07206/) (2026)
- [DGCPath: Distribution-Aware Generative Contrastive Framework for Self-supervised Path Representation Learning -- Extended Version](https://mlsanity.com/paper/2609.07316/) (2026)
- [TrafficClaw: A Generalizable LLM Agent in the Unified Physical Environment for Urban Traffic Control](https://mlsanity.com/paper/2604.17456/) (2026)
- [GeoMetric: Injecting Metric Geographic Structure into Worldwide Image Geo-Localization](https://mlsanity.com/paper/2606.08918/) (2026)
- [When Correlations Mislead: Confounder-Aware Multi-View Urban Region Representation Learning](https://mlsanity.com/paper/2609.15305/) (2026)

## Abstract (arXiv)

Developing effective path representations has become increasingly essential across various fields within intelligent transportation. Although pre-trained path representation learning models have shown improved performance, they predominantly focus on the topological structures from single modality data, i.e., road networks, overlooking the geometric and contextual features associated with path-related images, e.g., remote sensing images. Similar to human understanding, integrating information from multiple modalities can provide a more comprehensive view, enhancing both representation accuracy and generalization. However, variations in information granularity impede the semantic alignment of road network-based paths (road paths) and image-based paths (image paths), while the heterogeneity of multi-modal data poses substantial challenges for effective fusion and utilization. In this paper, we propose a novel Multi-modal, Multi-granularity Path Representation Learning Framework (MM-Path), which can learn a generic path representation by integrating modalities from both road paths and image paths. To enhance the alignment of multi-modal data, we develop a multi-granularity alignment strategy that systematically associates nodes, road sub-paths, and road paths with their corresponding image patches, ensuring the synchronization of both detailed local information and broader global contexts. To address the heterogeneity of multi-modal data effectively, we introduce a graph-based cross-modal residual fusion component designed to comprehensively fuse information across different modalities and granularities. Finally, we conduct extensive experiments on two large-scale real-world datasets under two downstream tasks, validating the effectiveness of the proposed MM-Path. The code is available at: https://github.com/decisionintelligence/MM-Path.