# Multi-modal Representation Learning for Social Post Location Inference

arXiv: https://arxiv.org/abs/2306.07935 · HTML: https://mlsanity.com/paper/2306.07935/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [A Survey of Location Prediction on Twitter](https://mlsanity.com/paper/1705.03172/) (2017)
- [Tagvisor: A Privacy Advisor for Sharing Hashtags](https://mlsanity.com/paper/1802.04122/) (2018)
- [End-to-end Network for Twitter Geolocation Prediction and Hashing](https://mlsanity.com/paper/1710.04802/) (2017)
- [Geo-distinctive Visual Element Matching for Location Estimation of Images](https://mlsanity.com/paper/1601.07884/) (2016)
- [Fine-grained Geolocation Prediction of Tweets with Human Machine Collaboration](https://mlsanity.com/paper/2106.13411/) (2021)

## Abstract (arXiv)

Inferring geographic locations via social posts is essential for many practical location-based applications such as product marketing, point-of-interest recommendation, and infector tracking for COVID-19. Unlike image-based location retrieval or social-post text embedding-based location inference, the combined effect of multi-modal information (i.e., post images, text, and hashtags) for social post positioning receives less attention. In this work, we collect real datasets of social posts with images, texts, and hashtags from Instagram and propose a novel Multi-modal Representation Learning Framework (MRLF) capable of fusing different modalities of social posts for location inference. MRLF integrates a multi-head attention mechanism to enhance location-salient information extraction while significantly improving location inference compared with single domain-based methods. To overcome the noisy user-generated textual content, we introduce a novel attention-based character-aware module that considers the relative dependencies between characters of social post texts and hashtags for flexible multi-model information fusion. The experimental results show that MRLF can make accurate location predictions and open a new door to understanding the multi-modal data of social posts for online inference tasks.