# Point-of-Interest Type Prediction using Text and Images

arXiv: https://arxiv.org/abs/2109.00602 · HTML: https://mlsanity.com/paper/2109.00602/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!](https://mlsanity.com/paper/2010.06572/) (2020)
- [Multimodal Named Entity Recognition for Short Social Media Posts](https://mlsanity.com/paper/1802.07862/) (2018)
- [Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts](https://mlsanity.com/paper/1904.09073/) (2019)
- [Automatically Identifying Complaints in Social Media](https://mlsanity.com/paper/1906.03890/) (2019)
- [Analyzing Political Parody in Social Media](https://mlsanity.com/paper/2004.13878/) (2020)
- [CITE: A Corpus of Image-Text Discourse Relations](https://mlsanity.com/paper/1904.06286/) (2019)
- [Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings](https://mlsanity.com/paper/2011.01565/) (2020)
- [Analyzing Online Political Advertisements](https://mlsanity.com/paper/2105.04047/) (2021)
- [Modeling the Severity of Complaints in Social Media](https://mlsanity.com/paper/2103.12428/) (2021)
- [Complaint Identification in Social Media with Transformer Networks](https://mlsanity.com/paper/2010.10910/) (2020)
- [Extracting human emotions at different places based on facial expressions and spatial clustering analysis](https://mlsanity.com/paper/1905.01817/) (2019)
- [Point-of-Interest Type Inference from Social Media Text](https://mlsanity.com/paper/2009.14734/) (2020)

## Built on by (most cited first, 10 of 10)

- [Detecting Dementia from Speech and Transcripts using Transformers](https://mlsanity.com/paper/2110.14769/) (2021)
- [Automatic Identification and Classification of Bragging in Social Media](https://mlsanity.com/paper/2203.05840/) (2022)
- [Combining Humor and Sarcasm for Improving Political Parody Detection](https://mlsanity.com/paper/2205.03313/) (2022)
- [Understanding Social Media Cross-Modality Discourse in Linguistic Space](https://mlsanity.com/paper/2302.13311/) (2023)
- [Identifying and Characterizing Active Citizens who Refute Misinformation in Social Media](https://mlsanity.com/paper/2204.10080/) (2022)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification](https://mlsanity.com/paper/2303.15016/) (2023)
- [Multimodal Detection of Bots on X (Twitter) using Transformers](https://mlsanity.com/paper/2308.14484/) (2023)
- [A Multimodal Analysis of Influencer Content on Twitter](https://mlsanity.com/paper/2309.03064/) (2023)
- [Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks](https://mlsanity.com/paper/2309.07794/) (2023)

## Abstract (arXiv)

Point-of-interest (POI) type prediction is the task of inferring the type of a place from where a social media post was shared. Inferring a POI's type is useful for studies in computational social science including sociolinguistics, geosemiotics, and cultural geography, and has applications in geosocial networking technologies such as recommendation and visualization systems. Prior efforts in POI type prediction focus solely on text, without taking visual information into account. However in reality, the variety of modalities, as well as their semiotic relationships with one another, shape communication and interactions in social media. This paper presents a study on POI type prediction using multimodal information from text and images available at posting time. For that purpose, we enrich a currently available data set for POI type prediction with the images that accompany the text messages. Our proposed method extracts relevant information from each modality to effectively capture interactions between text and image achieving a macro F1 of 47.21 across eight categories significantly outperforming the state-of-the-art method for POI type prediction based on text-only methods. Finally, we provide a detailed analysis to shed light on cross-modal interactions and the limitations of our best performing model.