Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages
Not on the map. Why?
Abstract (arXiv)
Calibrated typed-decision ("System One") models return a label and a probability in a single forward pass at a small fraction of the cost of large language models (LLMs), yet their evaluation has been almost entirely English. We report a pre-registered evaluation of the commercial Jev model and the open Laya-multilingual model on Indonesian regional languages, using the parallel NusaX sentiment (12 languages) and NusaTranslation sentiment and emotion (7 languages) test sets. Jev is more often correct on sentences that are lexically closer to Indonesian in all three datasets, and presenting the human Indonesian translation instead raises its accuracy by up to 37.2 points, most in languages distant from Indonesian. Its expected calibration error does not grow on regional languages, but its ability to rank its own errors degrades in all three datasets. Retrained classifiers need more than 500 labels per language to match zero-shot Jev in 9 of 10 NusaX regional languages. A confidence-gated cascade that sends Jev's 10-30% least confident decisions to a local IndoBERT classifier matches or exceeds the better of the two systems, and routing to an LLM instead matches that LLM at 24-77% lower API cost. Of six further commercial LLMs (three Claude and three GPT models), five are 4-14 points more accurate than Jev, at several hundred times the API cost where it is reported, while the smallest (Claude Haiku) is less accurate. We release the pre-registration, code and all predictions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 48 references that are in the map. First posted on arXiv 9 Oct 2026.
Builds on
26 of 48 references.
References · 26
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
- 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
- 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
- 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
- 2021How Many Data Points is a Prompt Worth?cited by 198
- 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
14 more
- 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 113
- 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
- 2020Thinking Fast and Slow in AIcited by 70
- 2024Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classificationcited by 70
- 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
- 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
- 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 58
- 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 45
- 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 41
- 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
- 2022On the Calibration of Massively Multilingual Language Modelscited by 19
- 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 11
- 2026Evaluating and Benchmarking the System One Model Jevcited by 9
- 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 5
Built on by
Not cited yet.