mlsanityData to 14 Oct 2026

ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring

arXiv 2610.06744 ↗First posted 5 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

ufakzeka-karar is an open Turkish decision model with 182,494,466 parameters. Given a Turkish text and questions of a fixed answer type (a choice, a level on an ordered scale, or yes or no), it returns a temperature-scaled probability for every option and an expected error that serves as a "not sure" signal, without generating text and in one CPU forward pass for up to ten options. Built on the lab's ufakzeka-1-base, its head scores each option blind to the others at shared positions, so the answer does not depend on option order. A sequential head trained with shuffled options was about as accurate but changed 2.3 to 2.8 percent of its answers when only the option order changed; REINFORCE lost 10.2 points (0.102) of macro F1 to cross-entropy. On the open set of HakemBench v1.0 (4,275 questions, 7 tracks) the released model ranks 7th of 16 rows with a composite of 0.660 (95% interval 0.642 to 0.677). Temperature scaling lowers calibration error (smooth ECE) on the development set but raises it on held-out support questions, from 0.027 to 0.045 for the first scored run, which never trained on them; the released model later trained on them, so its 0.036 to 0.064 is not an unseen-question test. The released model is the last of three runs scored on HakemBench, and its numbers are not blind. The second run's new training data was aimed at the first run's errors on the full test set in guardrails, moderation and customer support, and the released run was trained after the second run's guardrail results on the full test set were read, under a protocol fixed in writing before any of its data, code or runs. All its numbers come after these readings; its guardrail, moderation and customer support numbers carry the flag "shaped by reading the test results". With every model scored on the other four tracks only, its composite is 0.678, 6th of 16. Weights and code are under Apache-2.0.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

24 of 46 references.

References · 24

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2017Decoupled Weight Decay Regularizationcited by 1,831
  6. 2017Selective Classification for Deep Neural Networkscited by 852
  7. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  8. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  9. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  10. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  11. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  12. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 91
12 more
  1. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 63
  2. 2021DOCTOR: A Simple Method for Detecting Misclassification Errorscited by 53
  3. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 37
  4. 2021MFAQ: a Multilingual FAQ Datasetcited by 17
  5. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 17
  6. 2025Set-LLM: A Permutation-Invariant LLMcited by 11
  7. 2025WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrievalcited by 10
  8. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learningcited by 3
  9. 2026MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculumcited by 2
  10. 2026ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratchcited by 2
  11. 2026A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluationcited by 1
  12. 2026HakemBench: A Turkish Benchmark of Typed Decisionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish
  2. 2024UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions
  3. 2024Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training
  4. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
  5. 2024Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking
  6. 2026Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
  7. 2024Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think
  8. 2025PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains