mlsanityData to 14 Oct 2026

On the Calibration of Massively Multilingual Language Models

arXiv 2210.12265 ↗First posted 21 Oct 2022cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in evaluating these models for their performance on a variety of tasks and languages, little attention has been paid on how well calibrated these models are with respect to the confidence in their predictions. We first investigate the calibration of MMLMs in the zero-shot setting and observe a clear case of miscalibration in low-resource languages or those which are typologically diverse from English. Next, we empirically show that calibration methods like temperature scaling and label smoothing do reasonably well towards improving calibration in the zero-shot scenario. We also find that few-shot examples in the language can further help reduce the calibration errors, often substantially. Overall, our work contributes towards building more reliable multilingual models by highlighting the issue of their miscalibration, understanding what language and model specific factors influence it, and pointing out the strategies to improve the same.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
19 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  2. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  3. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  4. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  5. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  6. 2020Calibration of Pre-trained Transformerscited by 315
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  8. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  9. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  10. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  11. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  12. 2020The Multilingual Amazon Reviews Corpuscited by 143
  13. 2020Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptationcited by 43
  14. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  15. 2020Parameter Space Factorization for Zero-Shot Learning across Tasks and Languagescited by 22
  16. 2020Unsupervised Calibration under Covariate Shiftcited by 15
  17. 2020Are Chess Discussions Racist? An Adversarial Hate Speech Data Setcited by 10
  18. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10
  19. 2022Highly Generalizable Models for Multilingual Hate Speech Detectioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 2; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  2. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  3. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  4. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  5. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  6. 2024The Zeno's Paradox of `Low-Resource' Languagescited by 12
  7. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 10
  8. 2023On the Calibration of Multilingual Question Answering LLMscited by 6
  9. 2024Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Modelscited by 6
  10. 2024When Quantization Affects Confidence of Large Language Models?cited by 5
  11. 2024Predict the Next Word: Humans exhibit uncertainty in this task and language models _____cited by 4
  12. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
7 more
  1. 2023Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibrationcited by 2
  2. 2024SSP: Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Modelscited by 1
  3. 2026Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Modelscited by 1
  4. 2025How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigmcited by 0
  5. 2026Choosing the Right Language Mode at Inference Time for Multilingual Reliabilitycited by 0
  6. 2026Language Distances are Practical for Equitable Cross-Lingual Transfercited by 0
  7. 2026Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languagescited by 0