mlsanityData to 14 Oct 2026

SUTRA: Scalable Multilingual Language Model Architecture

arXiv 2405.06694 ↗First posted 7 May 2024cs.CL cs.AIcited by 11

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this paper, we introduce SUTRA, multilingual Large Language Model architecture capable of understanding, reasoning, and generating text in over 50 languages. SUTRA's design uniquely decouples core conceptual understanding from language-specific processing, which facilitates scalable and efficient multilingual alignment and learning. Employing a Mixture of Experts framework both in language and concept processing, SUTRA demonstrates both computational efficiency and responsiveness. Through extensive evaluations, SUTRA is demonstrated to surpass existing models like GPT-3.5, Llama2 by 20-30% on leading Massive Multitask Language Understanding (MMLU) benchmarks for multilingual tasks. SUTRA models are also online LLMs that can use knowledge from the internet to provide hallucination-free, factual and up-to-date responses while retaining their multilingual capabilities. Furthermore, we explore the broader implications of its architecture for the future of multilingual AI, highlighting its potential to democratize access to AI technology globally and to improve the equity and utility of AI in regions with predominantly non-English languages. Our findings suggest that SUTRA not only fills pivotal gaps in multilingual model capabilities but also establishes a new benchmark for operational efficiency and scalability in AI applications.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 38 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

30 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performance23 May
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Models7 Oct

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2023Mistral 7Bcited by 3,164
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
16 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2017Six Challenges for Neural Machine Translationcited by 335
  6. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  7. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  8. 2021Internet-Augmented Dialogue Generationcited by 217
  9. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  10. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  11. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  12. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  13. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  14. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  15. 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
  16. 2024RakutenAI-7B: Extending Large Language Models for Japanesecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 4.

11 papers.

  1. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  2. 2024Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languagescited by 11
  3. 2025Chitrarth: Bridging Vision and Language for a Billion Peoplecited by 10
  4. 2025Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translationcited by 6
  5. 2024Performance Evaluation of Tokenizers in Large Language Models for the Assamese Languagecited by 3
  6. 2025AMBEDKAR-A Multi-level Bias Elimination through a Decoding Approach with Knowledge Augmentation for Robust Constitutional Alignment of Language Modelscited by 2
  7. 2025BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languagescited by 2
  8. 2026BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_basecited by 2
  9. 2026Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languagescited by 0
  10. 2026Perspectives on Cross-Lingual Consistency in LLMs for Medical Questionscited by 0
  11. 2026Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Propositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment
  2. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact
  3. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
  4. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
  5. 2023YAYI 2: Multilingual Open-Source Large Language Models
  6. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
  7. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMs
  8. 2026Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations