# Far Out: Evaluating Language Models on Slang in Australian and Indian English

arXiv: https://arxiv.org/abs/2602.15373 · HTML: https://mlsanity.com/paper/2602.15373/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024)
- [Evaluation of African American Language Bias in Natural Language Generation](https://mlsanity.com/paper/2305.14291/) (2023)
- [DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures](https://mlsanity.com/paper/2403.14651/) (2024)
- [Evaluating Dialect Robustness of Language Models via Conversation Understanding](https://mlsanity.com/paper/2405.05688/) (2024)
- [Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models](https://mlsanity.com/paper/2409.00358/) (2024)

## Abstract (arXiv)

Language models exhibit systematic performance gaps when processing text in non-standard language varieties, yet their ability to comprehend variety-specific slang remains underexplored for several languages. We present a comprehensive evaluation of slang awareness in Indian English (en-IN) and Australian English (en-AU) across seven state-of-the-art language models. We construct two complementary datasets: WEB, containing 377 web-sourced usage examples from Urban Dictionary, and GEN, featuring 1,492 synthetically generated usages of these slang terms, across diverse scenarios. We assess language models on three tasks: target word prediction (TWP), guided target word prediction (TWP^*) and target word selection (TWS). Our results reveal four key findings: (1) Higher average model performance TWS versus TWP and TWP^*, with average accuracy score increasing from 0.03 to 0.49 respectively (2) Stronger average model performance on WEB versus GEN datasets, with average similarity score increasing by 0.03 and 0.05 across TWP and TWP^* tasks respectively (3) en-IN tasks outperform en-AU when averaged across all models and datasets, with TWS demonstrating the largest disparity, increasing average accuracy from 0.44 to 0.54. These findings underscore fundamental asymmetries between generative and discriminative competencies for variety-specific language, particularly in the context of slang expressions despite being in a technologically rich language such as English.