# NativQA: Multilingual Culturally-Aligned Natural Query for LLMs

arXiv: https://arxiv.org/abs/2407.09823 · HTML: https://mlsanity.com/paper/2407.09823/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 13 Jul 2024.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](https://mlsanity.com/paper/2311.09528/) (2023)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [GooAQ: Open Question Answering with Diverse Answer Types](https://mlsanity.com/paper/2104.08727/) (2021)

## Built on by (most cited first, 30 of 30)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture](https://mlsanity.com/paper/2509.19274/) (2025)
- [MemeIntel: Explainable Detection of Propagandistic and Hateful Memes](https://mlsanity.com/paper/2502.16612/) (2025)
- [TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking](https://mlsanity.com/paper/2502.11187/) (2025)
- [SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs](https://mlsanity.com/paper/2505.19163/) (2025)
- [CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models](https://mlsanity.com/paper/2505.19108/) (2025)
- [CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis](https://mlsanity.com/paper/2509.10886/) (2025)
- [Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs](https://mlsanity.com/paper/2604.19292/) (2026)
- [Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub](https://mlsanity.com/paper/2506.00482/) (2025)
- [Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants](https://mlsanity.com/paper/2510.24328/) (2025)
- [MENASpeechBank: A Reference Voice Bank with Persona-Conditioned Multi-Turn Conversations for AudioLLMs](https://mlsanity.com/paper/2602.07036/) (2026)
- [Fanar-Sadiq: A Multi-Agent Architecture for Grounded Islamic QA](https://mlsanity.com/paper/2603.08501/) (2026)
- [Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation](https://mlsanity.com/paper/2604.05083/) (2026)
- [CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation](https://mlsanity.com/paper/2508.17324/) (2025)
- [CALM: Culturally Self-Aware Language Models](https://mlsanity.com/paper/2601.03483/) (2026)
- [Pseudo-Knowledge Graph: Meta-Path Guided Retrieval and In-Graph Text for RAG-Equipped LLM](https://mlsanity.com/paper/2503.00309/) (2025)
- [XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering](https://mlsanity.com/paper/2508.16139/) (2025)
- [No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding](https://mlsanity.com/paper/2602.03709/) (2026)
- [CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks](https://mlsanity.com/paper/2604.19262/) (2026)
- [Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs](https://mlsanity.com/paper/2604.21751/) (2026)
- [Learning to Route Languages for Multilingual Policy Optimization](https://mlsanity.com/paper/2605.25360/) (2026)
- [NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledge](https://mlsanity.com/paper/2504.05995/) (2025)
- [The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models](https://mlsanity.com/paper/2602.01161/) (2026)
- [ExCAM: Explainable Cultural Awareness Metrics](https://mlsanity.com/paper/2605.29897/) (2026)
- [When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models](https://mlsanity.com/paper/2605.30481/) (2026)
- [Improving Answer Extraction in Context-based Question Answering Systems Using LLMs](https://mlsanity.com/paper/2606.06197/) (2026)
- [MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark](https://mlsanity.com/paper/2607.00724/) (2026)
- [PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding](https://mlsanity.com/paper/2608.21853/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work has been done in parallel, there is a notable lack of a framework and large scale region-specific datasets queried by native users in their own languages. This gap hinders the effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of ~64k manually annotated QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark open- and closed-source LLMs with the MultiNativQA dataset. We made the MultiNativQA dataset(https://huggingface.co/datasets/QCRI/MultiNativQA), and other experimental scripts(https://gitlab.com/nativqa/multinativqa) publicly available for the community.