mlsanityData to 7 Oct 2026

IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

arXiv 2602.16832 ↗First posted 18 Feb 2026cs.AI cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities understudied. We introduce Indic Jailbreak Robustness (IJR), a judge-free benchmark for adversarial safety across 12 Indic and South Asian languages (2.1 Billion speakers), covering 45216 prompts in JSON (contract-bound) and Free (naturalistic) tracks. IJR reveals three patterns. (1) Contracts inflate refusals but do not stop jailbreaks: in JSON, LLaMA and Sarvam exceed 0.92 JSR, and in Free all models reach 1.0 with refusals collapsing. (2) English to Indic attacks transfer strongly, with format wrappers often outperforming instruction wrappers. (3) Orthography matters: romanized or mixed inputs reduce JSR under JSON, with correlations to romanization share and tokenization (approx 0.28 to 0.32) indicating systematic effects. Human audits confirm detector reliability, and lite-to-full comparisons preserve conclusions. IJR offers a reproducible multilingual stress test revealing risks hidden by English-only, contract-focused evaluations, especially for South Asian users who frequently code-switch and romanize.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 38 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

21 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture10 Jul

Other references · 20

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2019Defending Against Neural Fake Newscited by 553
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  8. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 55
  9. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  10. 2022Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Userscited by 19
  11. 2024Survey of Large Multimodal Model Datasets, Application Categories and Taxonomycited by 17
  12. 2025SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Usecited by 15
8 more
  1. 2025Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systemscited by 14
  2. 2024Enhancing Document AI Data Generation Through Graph-Based Synthetic Layoutscited by 12
  3. 2025Tokenization Matters: Improving Zero-Shot NER for Indic Languagescited by 12
  4. 2025Clinical QA 2.0: Multi-Task Learning for Answer Extraction and Categorizationcited by 8
  5. 2025IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languagescited by 7
  6. 2025Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptationcited by 6
  7. 2025Review of Tools for Zero-Code LLM Based Application Developmentcited by 4
  8. 2026LLM-Guided Lifecycle-Aware Clustering of Multi-Turn Customer Support Conversationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asiacited by 3
  2. 2026Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harmscited by 1
  3. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Modelscited by 0
  4. 2026LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Reviewcited by 0
  5. 2026Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
  2. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  3. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  4. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models
  5. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  6. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  7. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  8. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy