mlsanityData to 14 Oct 2026

MEGA: Multilingual Evaluation of Generative AI

arXiv 2303.12528 ↗First posted 22 Mar 2023cs.CLcited by 293

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important question being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative LLMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and tasks and discuss challenges in improving the performance of generative LLMs on low-resource languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

Key facts. Cited by 293 papers in the map (data through 14 Oct 2026). Builds on 67 of its 100 references that are in the map. First posted on arXiv 22 Mar 2023.

Builds on

67 of 100 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
38 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  13. 2019How multilingual is Multilingual BERT?cited by 645
  14. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  15. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  16. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  17. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  18. 2018Gender Bias in Coreference Resolutioncited by 442
  19. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  20. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  21. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  22. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  23. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  24. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  25. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  26. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  27. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  28. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  29. 2019Evaluating Gender Bias in Machine Translationcited by 249
  30. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  31. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  32. 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
  33. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  34. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  35. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  36. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  37. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
  38. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 52; 2024: 125; 2025: 62; 2026: 54.

293 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  4. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  5. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  6. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  7. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  8. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  9. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  10. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  11. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  12. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
38 more
  1. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  2. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  3. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  4. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  5. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  6. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  7. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  8. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  9. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  10. 2023Do Multilingual Language Models Think Better in English?cited by 88
  11. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  12. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  13. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  14. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  15. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 66
  16. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  17. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  18. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  19. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 53
  20. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
  21. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  22. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 50
  23. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 46
  24. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  25. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  26. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  27. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  28. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 40
  29. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  30. 2024Generative Artificial Intelligence: A Systematic Review and Applicationscited by 37
  31. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  32. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 36
  33. 2023COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuancescited by 35
  34. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 35
  35. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  36. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  37. 2023This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Modelscited by 32
  38. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32

50 most cited of 293 All 293 (JSON)