mlsanityData to 7 Oct 2026

Large Language Models are Geographically Biased

arXiv 2402.02680 ↗First posted 5 Feb 2024cs.CL cs.AI cs.CYcited by 91

On the map: World models → World models for language agents

Abstract (arXiv)

Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is particularly powerful as there is ground truth for the numerous aspects of human life that are meaningfully projected onto geographic space such as culture, race, language, politics, and religion. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions. Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's ρ of up to 0.89). We then show that LLMs exhibit common biases across a range of objective and subjective topics. In particular, LLMs are clearly biased against locations with lower socioeconomic conditions (e.g. most of Africa) on a variety of sensitive subjective topics such as attractiveness, morality, and intelligence (Spearman's ρ of up to 0.70). Finally, we introduce a bias score to quantify this and find that there is significant variation in the magnitude of bias across existing LLMs. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 40 of its 61 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

40 of 61 references.

References · 40

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2024Mixtral of Expertscited by 1,785
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  9. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
28 more
  1. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  2. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  3. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  4. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  5. 2023Language Modeling Is Compressioncited by 225
  6. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  7. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  8. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  9. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  10. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  11. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  12. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  13. 2023On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligencecited by 88
  14. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  15. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  16. 2020Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Biascited by 82
  17. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  18. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  19. 2021SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learningcited by 53
  20. 2023GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPTcited by 40
  21. 2020A survey of bias in Machine Learning through the prism of Statistical Parity for the Adult Data Setcited by 35
  22. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  23. 2022Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Resultscited by 17
  24. 2023Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articlescited by 16
  25. 2021Assessing the Reliability of Word Embedding Gender Bias Measurescited by 14
  26. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  27. 2017Poverty Prediction with Public Landsat 7 Satellite Imagery and Machine Learningcited by 9
  28. 2024Global-Liar: Factuality of LLMs over Time and Geographic Regionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 32; 2026: 34.

91 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 56
  3. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  4. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  5. 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Modelscited by 29
  6. 2024TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learningcited by 29
  7. 2024CityBench: Evaluating the Capabilities of Large Language Models for Urban Taskscited by 26
  8. 2024AgentMove: A Large Language Model based Agentic Framework for Zero-shot Next Location Predictioncited by 25
  9. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  10. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 23
  11. 2024Geolocation Representation from Large Language Models are Generic Enhancers for Spatio-Temporal Learningcited by 15
  12. 2025Spatial-RAG: Spatial Retrieval Augmented Generation for Real-World Geospatial Reasoning Questionscited by 13
38 more
  1. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  2. 2025ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learningcited by 10
  3. 2024Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendationscited by 9
  4. 2025GAIR: Location-Aware Self-Supervised Contrastive Pre-Training with Geo-Aligned Implicit Representationscited by 9
  5. 2024Attributing Culture-Conditioned Generations to Pretraining Corporacited by 8
  6. 2025OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligencecited by 7
  7. 2024Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approachcited by 6
  8. 2025B-score: Detecting biases in large language models using response historycited by 6
  9. 2025Around the World in 24 Hours: Probing LLM Knowledge of Time and Placecited by 6
  10. 2025Whose Truth? Pluralistic Geo-Alignment for (Agentic) AIcited by 6
  11. 2025Mitigating Biases in Language Models via Bias Unlearningcited by 6
  12. 2024Retrieval Augmented Thought Process for Private Data Handling in Healthcarecited by 5
  13. 2024It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Humancited by 5
  14. 2024Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLMcited by 5
  15. 2024TravelAgent: Generative Agents in the Built Environmentcited by 5
  16. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  17. 2024Responsible AI for Earth Observationcited by 4
  18. 2024Quantifying Geospatial in the Common Crawl Corpuscited by 4
  19. 2025Uncovering Cultural Representation Disparities in Vision-Language Modelscited by 4
  20. 2025CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulationcited by 4
  21. 2026Auditing Preferences for Brands and Cultures in LLMscited by 4
  22. 2024Hybrid Querying Over Relational Databases and Large Language Modelscited by 3
  23. 2025Massive-STEPS: Massive Semantic Trajectories for Understanding POI Check-ins -- Dataset and Benchmarkscited by 3
  24. 2025On the Performance of LLMs for Real Estate Appraisalcited by 3
  25. 2025Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Languagecited by 3
  26. 2025Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligningcited by 3
  27. 2025Urban-R1: Reinforced MLLMs Mitigate Geospatial Biases for Urban General Intelligencecited by 3
  28. 2025Generating the Modal Worker: A Cross-Model Audit of Race and Gender in LLM-Generated Personas Across 41 Occupationscited by 3
  29. 2025LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoningcited by 3
  30. 2026In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generationscited by 3
  31. 2025Breaking Down Bias: On The Limits of Generalizable Pruning Strategiescited by 2
  32. 2025Where Should I Study? Biased Language Models Decide! Evaluating Fairness in LMs for Academic Recommendationscited by 2
  33. 2025GeoBS: Information-Theoretic Quantification of Geographic Bias in AI Modelscited by 2
  34. 2025Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queriescited by 2
  35. 2026GPSBench: Do Large Language Models Understand GPS Coordinates?cited by 2
  36. 2026Emergence of Biased Consensus in Multi-Agent LLM Debatescited by 2
  37. 2024LoFTI: Localization and Factuality Transfer to Indian Localescited by 1
  38. 2025Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rightscited by 1

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Distortions in Judged Spatial Relations in Large Language Models
  2. 2024With a Grain of SALT: Are LLMs Fair Across Social Dimensions?
  3. 2024A Multi-LLM Debiasing Framework
  4. 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Models
  6. 2023Gender bias and stereotypes in Large Language Models
  7. 2025Geospatial Mechanistic Interpretability of Large Language Models
  8. 2025Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution