mlsanityData to 7 Oct 2026

Fumbling in Babel: An Investigation into ChatGPT's Language Identification Ability

arXiv 2311.09696 ↗First posted 16 Nov 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

ChatGPT has recently emerged as a powerful NLP tool that can carry out a variety of tasks. However, the range of languages ChatGPT can handle remains largely a mystery. To uncover which languages ChatGPT `knows', we investigate its language identification (LID) abilities. For this purpose, we compile Babel-670, a benchmark comprising 670 languages representing 24 language families spoken in five continents. Languages in Babel-670 run the gamut from the very high-resource to the very low-resource. We then study ChatGPT's (both GPT-3.5 and GPT-4) ability to (i) identify language names and language codes (ii) under zero- and few-shot conditions (iii) with and without provision of a label set. When compared to smaller finetuned LID tools, we find that ChatGPT lags behind. For example, it has poor performance on African languages. We conclude that current large language models would benefit from further development before they can sufficiently serve diverse communities.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

15 of 30 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  7. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  9. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  10. 2023An Open Dataset and Model for Language Identificationcited by 49
  11. 2022Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Gocited by 46
  12. 2018Automatic Language Identification in Texts: A Surveycited by 40
3 more
  1. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  2. 2021Improved Language Identification Through Cross-Lingual Self-Supervised Learningcited by 20
  3. 2023Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languagescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  2. 2025Where Are We? Evaluating LLM Performance on African Languagescited by 9
  3. 2026Robust Language Identification for Romansh Varietiescited by 2
  4. 2025Improving Informally Romanized Language Identificationcited by 1
  5. 2026What Language is This? Ask Your Tokenizercited by 1
  6. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0