# NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledge

arXiv: https://arxiv.org/abs/2504.05995 · HTML: https://mlsanity.com/paper/2504.05995/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 25 of 25)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [SeaLLMs -- Large Language Models for Southeast Asia](https://mlsanity.com/paper/2312.00738/) (2023)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages](https://mlsanity.com/paper/2410.16153/) (2024)
- [GooAQ: Open Question Answering with Diverse Answer Types](https://mlsanity.com/paper/2104.08727/) (2021)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese](https://mlsanity.com/paper/2402.17302/) (2024)
- [PALO: A Polyglot Large Multimodal Model for 5B People](https://mlsanity.com/paper/2402.14818/) (2024)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [NativQA: Multilingual Culturally-Aligned Natural Query for LLMs](https://mlsanity.com/paper/2407.09823/) (2024)
- [Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation](https://mlsanity.com/paper/2501.03225/) (2025)
- [ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?](https://mlsanity.com/paper/2403.17368/) (2024)
- [Think Fast, Think Slow, Think Critical: Designing an Automated Propaganda Detection Tool](https://mlsanity.com/paper/2402.19135/) (2024)
- [TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking](https://mlsanity.com/paper/2502.11187/) (2025)
- [SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set](https://mlsanity.com/paper/1707.08435/) (2017)

## Abstract (arXiv)

The rapid progress of large language models (LLMs) raises concerns about cultural bias, fairness, and performance in diverse languages and underrepresented regions. Addressing these gaps requires large-scale resources grounded in multilingual, local, and cultural contexts. We systematize and extend the earlier NativQA framework to multimodality by adding image, audio, and video support, enabling scalable construction of culturally and regionally aligned QA datasets in native languages. Given user-defined seed queries, the framework uses search engines to collect location-specific everyday information. We evaluate it across 39 locations in 24 countries and 7 languages, spanning extremely low-resource to high-resource settings, and collect over ~300K text QA pairs, ~312K images, and ~29K videos with associated audio. The developed resources can be used for LLMs benchmarking and further fine-tuning. The framework has been made publicly available for the community (https://gitlab.com/nativqa/nativqa-framework). Demo video is available here: hrefhttps://shorturl.at/DAVn9https://shorturl.at/DAVn9.