mlsanityData to 7 Oct 2026

SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation

arXiv 2210.12213 ↗First posted 21 Oct 2022cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Named geographic entities (geo-entities for short) are the building blocks of many geographic datasets. Characterizing geo-entities is integral to various application domains, such as geo-intelligence and map comprehension, while a key challenge is to capture the spatial-varying context of an entity. We hypothesize that we shall know the characteristics of a geo-entity by its surrounding entities, similar to knowing word meanings by their linguistic context. Accordingly, we propose a novel spatial language model, SpaBERT, which provides a general-purpose geo-entity representation based on neighboring entities in geospatial data. SpaBERT extends BERT to capture linearized spatial context, while incorporating a spatial coordinate embedding mechanism to preserve spatial relations of entities in the 2-dimensional space. SpaBERT is pretrained with masked language modeling and masked entity prediction tasks to learn spatial dependencies. We apply SpaBERT to two downstream tasks: geo-entity typing and geo-entity linking. Compared with the existing language models that do not use spatial context, SpaBERT shows significant performance improvement on both tasks. We also analyze the entity representation from SpaBERT in various settings and the effect of spatial coordinate embedding.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 41 of its 60 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

41 of 60 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
29 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2019Publicly Available Clinical BERT Embeddingscited by 995
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  7. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  8. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  9. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  12. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  13. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  14. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  15. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  16. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  17. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  18. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  19. 2021MATE: Multi-view Attention for Table Transformer Efficiencycited by 54
  20. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  21. 2020Code and Named Entity Recognition in StackOverflowcited by 39
  22. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  23. 2022Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learningcited by 32
  24. 2022StruBERT: Structure-aware BERT for Table Search and Matchingcited by 24
  25. 2020Learning to Represent Image and Text with Denotation Graphcited by 13
  26. 2021Prix-LM: Pretraining for Multilingual Knowledge Base Constructioncited by 12
  27. 2020DETECT: Deep Trajectory Clustering for Mobility-Behavior Analysiscited by 11
  28. 2021Visually Grounded Concept Compositioncited by 5
  29. 2021Building Autocorrelation-Aware Representations for Fine-Scale Spatiotemporal Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 7; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
  3. 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Modelscited by 29
  4. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  5. 2023City Foundation Models for Learning General Purpose Representations from OpenStreetMapcited by 21
  6. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  7. 2025GAIR: Location-Aware Self-Supervised Contrastive Pre-Training with Geo-Aligned Implicit Representationscited by 9
  8. 2024Self-Supervised Representation Learning for Geospatial Objects: A Surveycited by 8
  9. 2023GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmarkcited by 5
  10. 2024Where on Earth Do Users Say They Are?: Geo-Entity Linking for Noisy Multilingual User Inputcited by 4
  11. 2025MoRA: Mobility as the Backbone for Geospatial Representation Learning at Scalecited by 4
  12. 2026Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movementcited by 3
11 more
  1. 2024Context-Aware Trajectory Anomaly Detectioncited by 2
  2. 2025AdaptGOT: A Pre-trained Model for Adaptive Contextual POI Representation Learningcited by 1
  3. 2025GeoResponder: Towards Building Geospatial LLMs for Time-Critical Disaster Responsecited by 1
  4. 2024Encoding Agent Trajectories as Representations with Sequence Transformerscited by 0
  5. 2024Leveraging Large Language Models for Generating Labeled Mineral Site Record Linkage Datacited by 0
  6. 2025GPS-MTM: Capturing Pattern of Normalcy in GPS-Trajectories with self-supervised learningcited by 0
  7. 2025Markovian Reeb Graphs for Simulating Spatiotemporal Patterns of Lifecited by 0
  8. 2026NARA: Anchor-Conditioned Representation Learning for Heterogeneous Vector Geoentitiescited by 0
  9. 2026Earth Science Foundation Models: From Perception to Reasoning and Discoverycited by 0
  10. 2026From Location Phrases to Geographic Entities: Task-Adapted Retrieval for People Searchcited by 0
  11. 2026Location-Aware Language Models via Secondary Embeddingscited by 0