# ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps

arXiv: https://arxiv.org/abs/2203.09127 · HTML: https://mlsanity.com/paper/2203.09127/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [A Survey of Point-of-interest Recommendation in Location-based Social Networks](https://mlsanity.com/paper/1607.00647/) (2016)
- [SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering](https://mlsanity.com/paper/1910.11559/) (2019)
- [C-Watcher: A Framework for Early Detection of High-Risk Neighborhoods Ahead of COVID-19 Outbreak](https://mlsanity.com/paper/2012.12169/) (2020)
- [GEDIT: Geographic-Enhanced and Dependency-Guided Tagging for Joint POI and Accessibility Extraction at Baidu Maps](https://mlsanity.com/paper/2108.09104/) (2021)
- [Understanding the Impact of the COVID-19 Pandemic on Transportation-related Behaviors with Human Mobility Data](https://mlsanity.com/paper/2202.12264/) (2022)

## Built on by (most cited first, 25 of 25)

- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web](https://mlsanity.com/paper/2310.18340/) (2023)
- [K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization](https://mlsanity.com/paper/2306.05064/) (2023)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook](https://mlsanity.com/paper/2402.19348/) (2024)
- [CityGPT: Empowering Urban Spatial Cognition of Large Language Models](https://mlsanity.com/paper/2406.13948/) (2024)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [T-RAG: Lessons from the LLM Trenches](https://mlsanity.com/paper/2402.07483/) (2024)
- [Poly2Vec: Polymorphic Fourier-Based Encoding of Geospatial Objects for GeoAI Applications](https://mlsanity.com/paper/2408.14806/) (2024)
- [MGeo: Multi-Modal Geographic Pre-Training Method](https://mlsanity.com/paper/2301.04283/) (2023)
- [QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search](https://mlsanity.com/paper/2306.06707/) (2023)
- [LAMP: A Language Model on the Map](https://mlsanity.com/paper/2403.09059/) (2024)
- [Self-Supervised Representation Learning for Geospatial Objects: A Survey](https://mlsanity.com/paper/2408.12133/) (2024)
- [LDMapNet-U: An End-to-End System for City-Scale Lane-Level Map Updating](https://mlsanity.com/paper/2501.02763/) (2025)
- [Evaluation of Geographical Distortions in Language Models](https://mlsanity.com/paper/2404.17401/) (2024)
- [G2PTL: A Pre-trained Model for Delivery Address and its Applications in Logistics System](https://mlsanity.com/paper/2304.01559/) (2023)
- [Survey on Foundation Models for Prognostics and Health Management in Industrial Cyber-Physical Systems](https://mlsanity.com/paper/2312.06261/) (2023)
- [GeoDecoder: Empowering Multimodal Map Understanding](https://mlsanity.com/paper/2401.15118/) (2024)
- [AddrLLM: Address Rewriting via Large Language Model on Nationwide Logistics Data](https://mlsanity.com/paper/2411.13584/) (2024)
- [Spacetime-GR: A Spacetime-Aware Generative Model for Large Scale Online POI Recommendation](https://mlsanity.com/paper/2508.16126/) (2025)
- [Geo-Encoder: A Chunk-Argument Bi-Encoder Framework for Chinese Geographic Re-Ranking](https://mlsanity.com/paper/2309.01606/) (2023)
- [GeoResponder: Towards Building Geospatial LLMs for Time-Critical Disaster Response](https://mlsanity.com/paper/2509.19354/) (2025)
- [DuIVRS-2: An LLM-based Interactive Voice Response System for Large-scale POI Attribute Acquisition](https://mlsanity.com/paper/2605.17900/) (2026)
- [GenHAR: Generalizing Cross-domain Human Activity Recognition for Last-mile Delivery](https://mlsanity.com/paper/2605.22086/) (2026)

## Abstract (arXiv)

Pre-trained models (PTMs) have become a fundamental backbone for downstream tasks in natural language processing and computer vision. Despite initial gains that were obtained by applying generic PTMs to geo-related tasks at Baidu Maps, a clear performance plateau over time was observed. One of the main reasons for this plateau is the lack of readily available geographic knowledge in generic PTMs. To address this problem, in this paper, we present ERNIE-GeoL, which is a geography-and-language pre-trained model designed and developed for improving the geo-related tasks at Baidu Maps. ERNIE-GeoL is elaborately designed to learn a universal representation of geography-language by pre-training on large-scale data generated from a heterogeneous graph that contains abundant geographic knowledge. Extensive quantitative and qualitative experiments conducted on large-scale real-world datasets demonstrate the superiority and effectiveness of ERNIE-GeoL. ERNIE-GeoL has already been deployed in production at Baidu Maps since April 2021, which significantly benefits the performance of various downstream tasks. This demonstrates that ERNIE-GeoL can serve as a fundamental backbone for a wide range of geo-related tasks.