# Learning Neighborhood Representation from Multi-Modal Multi-Graph: Image, Text, Mobility Graph and Beyond

arXiv: https://arxiv.org/abs/2105.02489 · HTML: https://mlsanity.com/paper/2105.02489/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 15 of 15)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Presence-Only Geographical Priors for Fine-Grained Image Classification](https://mlsanity.com/paper/1906.05272/) (2019)
- [Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US](https://mlsanity.com/paper/1702.06683/) (2017)
- [Geo-Aware Networks for Fine-Grained Recognition](https://mlsanity.com/paper/1906.01737/) (2019)
- [Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding](https://mlsanity.com/paper/2001.11101/) (2020)
- [ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery](https://mlsanity.com/paper/2011.05479/) (2020)
- [Effective Data Fusion with Generalized Vegetation Index: Evidence from Land Cover Segmentation in Agriculture](https://mlsanity.com/paper/2005.03743/) (2020)
- [A Survey on Spatial and Spatiotemporal Prediction Methods](https://mlsanity.com/paper/2012.13384/) (2020)

## Built on by (most cited first, 9 of 9)

- [C2-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System](https://mlsanity.com/paper/2201.02732/) (2022)
- [Region Embedding with Intra and Inter-View Contrastive Learning](https://mlsanity.com/paper/2211.08975/) (2022)
- [Urban Region Representation Learning with Attentive Fusion](https://mlsanity.com/paper/2312.04606/) (2023)
- [Self-Supervised Representation Learning for Geospatial Objects: A Survey](https://mlsanity.com/paper/2408.12133/) (2024)
- [CityPulse: Fine-Grained Assessment of Urban Change with Street View Time Series](https://mlsanity.com/paper/2401.01107/) (2024)
- [Detecting Neighborhood Gentrification at Scale via Street-level Visual Data](https://mlsanity.com/paper/2301.01842/) (2023)
- [CureGraph: Contrastive Multi-Modal Graph Representation Learning for Urban Living Circle Health Profiling and Prediction](https://mlsanity.com/paper/2501.07157/) (2025)
- [UrbanVerse: Learning Urban Region Representation Across Cities and Tasks](https://mlsanity.com/paper/2602.15750/) (2026)
- [Multimodal Contrastive Learning of Urban Space Representations from POI Data](https://mlsanity.com/paper/2411.06229/) (2024)

## Abstract (arXiv)

Recent urbanization has coincided with the enrichment of geotagged data, such as street view and point-of-interest (POI). Region embedding enhanced by the richer data modalities has enabled researchers and city administrators to understand the built environment, socioeconomics, and the dynamics of cities better. While some efforts have been made to simultaneously use multi-modal inputs, existing methods can be improved by incorporating different measures of 'proximity' in the same embedding space - leveraging not only the data that characterizes the regions (e.g., street view, local businesses pattern) but also those that depict the relationship between regions (e.g., trips, road network). To this end, we propose a novel approach to integrate multi-modal geotagged inputs as either node or edge features of a multi-graph based on their relations with the neighborhood region (e.g., tiles, census block, ZIP code region, etc.). We then learn the neighborhood representation based on a contrastive-sampling scheme from the multi-graph. Specifically, we use street view images and POI features to characterize neighborhoods (nodes) and use human mobility to characterize the relationship between neighborhoods (directed edges). We show the effectiveness of the proposed methods with quantitative downstream tasks as well as qualitative analysis of the embedding space: The embedding we trained outperforms the ones using only unimodal data as regional inputs.