mlsanityData to 14 Oct 2026

Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages

arXiv 2610.14112 ↗First posted 9 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Calibrated typed-decision ("System One") models return a label and a probability in a single forward pass at a small fraction of the cost of large language models (LLMs), yet their evaluation has been almost entirely English. We report a pre-registered evaluation of the commercial Jev model and the open Laya-multilingual model on Indonesian regional languages, using the parallel NusaX sentiment (12 languages) and NusaTranslation sentiment and emotion (7 languages) test sets. Jev is more often correct on sentences that are lexically closer to Indonesian in all three datasets, and presenting the human Indonesian translation instead raises its accuracy by up to 37.2 points, most in languages distant from Indonesian. Its expected calibration error does not grow on regional languages, but its ability to rank its own errors degrades in all three datasets. Retrained classifiers need more than 500 labels per language to match zero-shot Jev in 9 of 10 NusaX regional languages. A confidence-gated cascade that sends Jev's 10-30% least confident decisions to a local IndoBERT classifier matches or exceeds the better of the two systems, and routing to an LLM instead matches that LLM at 24-77% lower API cost. Of six further commercial LLMs (three Claude and three GPT models), five are 4-14 points more accurate than Jev, at several hundred times the API cost where it is reported, while the smallest (Claude Haiku) is less accurate. We release the pre-registration, code and all predictions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 48 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

26 of 48 references.

References · 26

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  10. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  11. 2021How Many Data Points is a Prompt Worth?cited by 198
  12. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
14 more
  1. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 113
  2. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  3. 2020Thinking Fast and Slow in AIcited by 70
  4. 2024Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classificationcited by 70
  5. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  6. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  7. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 58
  8. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 45
  9. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 41
  10. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
  11. 2022On the Calibration of Massively Multilingual Language Modelscited by 19
  12. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 11
  13. 2026Evaluating and Benchmarking the System One Model Jevcited by 9
  14. 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 5

Built on by

Not cited yet.