# IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

arXiv: https://arxiv.org/abs/2602.16832 · HTML: https://mlsanity.com/paper/2602.16832/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 21 of 21)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data](https://mlsanity.com/paper/2406.15053/) (2024)
- [Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture](https://mlsanity.com/paper/2407.07342/) (2024)
- [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022)
- [Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy](https://mlsanity.com/paper/2412.17759/) (2024)
- [SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use](https://mlsanity.com/paper/2505.17332/) (2025)
- [Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems](https://mlsanity.com/paper/2505.18366/) (2025)
- [Enhancing Document AI Data Generation Through Graph-Based Synthetic Layouts](https://mlsanity.com/paper/2412.03590/) (2024)
- [Tokenization Matters: Improving Zero-Shot NER for Indic Languages](https://mlsanity.com/paper/2504.16977/) (2025)
- [Clinical QA 2.0: Multi-Task Learning for Answer Extraction and Categorization](https://mlsanity.com/paper/2502.13108/) (2025)
- [IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages](https://mlsanity.com/paper/2506.02573/) (2025)
- [Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation](https://mlsanity.com/paper/2506.02097/) (2025)
- [Review of Tools for Zero-Code LLM Based Application Development](https://mlsanity.com/paper/2510.19747/) (2025)
- [LLM-Guided Lifecycle-Aware Clustering of Multi-Turn Customer Support Conversations](https://mlsanity.com/paper/2601.04388/) (2026)

## Built on by (most cited first, 5 of 5)

- [IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia](https://mlsanity.com/paper/2603.17915/) (2026)
- [Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms](https://mlsanity.com/paper/2608.22335/) (2026)
- [MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models](https://mlsanity.com/paper/2606.07706/) (2026)
- [LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review](https://mlsanity.com/paper/2608.14626/) (2026)
- [Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs](https://mlsanity.com/paper/2610.01428/) (2026)

## Abstract (arXiv)

Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities understudied. We introduce Indic Jailbreak Robustness (IJR), a judge-free benchmark for adversarial safety across 12 Indic and South Asian languages (2.1 Billion speakers), covering 45216 prompts in JSON (contract-bound) and Free (naturalistic) tracks. IJR reveals three patterns. (1) Contracts inflate refusals but do not stop jailbreaks: in JSON, LLaMA and Sarvam exceed 0.92 JSR, and in Free all models reach 1.0 with refusals collapsing. (2) English to Indic attacks transfer strongly, with format wrappers often outperforming instruction wrappers. (3) Orthography matters: romanized or mixed inputs reduce JSR under JSON, with correlations to romanization share and tokenization (approx 0.28 to 0.32) indicating systematic effects. Human audits confirm detector reliability, and lite-to-full comparisons preserve conclusions. IJR offers a reproducible multilingual stress test revealing risks hidden by English-only, contract-focused evaluations, especially for South Asian users who frequently code-switch and romanize.