# IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesian

arXiv: https://arxiv.org/abs/2507.22159 · HTML: https://mlsanity.com/paper/2507.22159/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 14 of 14)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation](https://mlsanity.com/paper/2102.00287/) (2021)
- [IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP](https://mlsanity.com/paper/2011.00677/) (2020)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [M-RewardBench: Evaluating Reward Models in Multilingual Settings](https://mlsanity.com/paper/2410.15522/) (2024)
- [RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs](https://mlsanity.com/paper/2407.02552/) (2024)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [R3: Robust Rubric-Agnostic Reward Models](https://mlsanity.com/paper/2505.13388/) (2025)
- [ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models](https://mlsanity.com/paper/2406.09334/) (2024)
- [Liputan6: A Large-scale Indonesian Dataset for Text Summarization](https://mlsanity.com/paper/2011.00679/) (2020)

## Built on by (most cited first, 1 of 1)

- [mR3: Multilingual Rubric-Agnostic Reward Reasoning Models](https://mlsanity.com/paper/2510.01146/) (2025)

## Abstract (arXiv)

Over 200 million people speak Indonesian, yet the language remains significantly underrepresented in preference-based research for large language models (LLMs). Most existing multilingual datasets are derived from English translations, often resulting in content that lacks cultural and linguistic authenticity. To address this gap, we introduce IndoPref, the first fully human-authored and multi-domain Indonesian preference dataset designed to evaluate the naturalness and quality of LLM-generated text. The dataset contains 522 prompts and yields 4,099 human-annotated pairwise preferences from comparisons across five instruction-tuned LLMs. All annotations are natively written in Indonesian with strong inter-annotator agreement, measured by Krippendorff's alpha. Our benchmark spans 10 diverse categories, enabling practitioners to identify LLMs' fine-grained strengths and weaknesses.