mlsanityData to 7 Oct 2026

Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models

arXiv 2510.03136 ↗First posted 3 Oct 2025cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Confidence calibration, the alignment of a model's predicted confidence with its actual accuracy, is crucial for the reliable deployment of Large Language Models (LLMs). However, this critical property remains largely under-explored in multilingual contexts. In this work, we conduct the first large-scale, systematic studies of multilingual calibration across six model families and over 100 languages, revealing that non-English languages suffer from systematically worse calibration. To diagnose this, we investigate the model's internal representations and find that the final layer, biased by English-centric training, provides a poor signal for multilingual confidence. In contrast, our layer-wise analysis uncovers a key insight that late-intermediate layers consistently offer a more reliable and better-calibrated signal. Building on this, we introduce a suite of training-free methods, including Language-Aware Confidence Ensemble (LACE), which adaptively selects an optimal ensemble of layers for each specific language. Our study highlights the hidden costs of English-centric alignment and offer a new path toward building more globally equitable and trustworthy LLMs by looking beyond the final layer.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

35 of 54 references.

References · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  9. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  10. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  11. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
23 more
  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2019Depth-Adaptive Transformercited by 227
  3. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  4. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  5. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  6. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  7. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelitycited by 47
  8. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  9. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 39
  10. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  11. 2024Atomic Calibration of LLMs in Long-Form Generationscited by 23
  12. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  13. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  14. 2022On the Calibration of Massively Multilingual Language Modelscited by 18
  15. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  16. 2024The Hidden Space of Transformer Language Adapterscited by 16
  17. 2024Do We Need Language-Specific Fact-Checking Models? The Case of Chinesecited by 15
  18. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  19. 2024The harms of class imbalance corrections for machine learning based prediction models: a simulation studycited by 10
  20. 2023On the Calibration of Multilingual Question Answering LLMscited by 6
  21. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5
  22. 2024MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Modelscited by 4
  23. 2022A Comparative Study of Confidence Calibration in Deep Learning: From Computer Vision to Medical Imagingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026Confidence Estimation for LLMs in Multi-turn Interactionscited by 12
  2. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  3. 2026When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMscited by 2
  4. 2026Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMscited by 1
  5. 2026Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Modelscited by 1
  6. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluatorscited by 0
  7. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?cited by 0
  8. 2026Choosing the Right Language Mode at Inference Time for Multilingual Reliabilitycited by 0
  9. 2026Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agentscited by 0