# VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

arXiv: https://arxiv.org/abs/2608.03095 · HTML: https://mlsanity.com/paper/2608.03095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Emotion Recognition for Vietnamese Social Media Text](https://mlsanity.com/paper/1911.09339/) (2019)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding](https://mlsanity.com/paper/2411.05049/) (2024)
- [Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine](https://mlsanity.com/paper/2406.02394/) (2024)
- [Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection](https://mlsanity.com/paper/2405.09279/) (2024)
- [VinaLLaMA: LLaMA-based Vietnamese Foundation Model](https://mlsanity.com/paper/2312.11011/) (2023)
- [Creative and Context-Aware Translation of East Asian Idioms with GPT-4](https://mlsanity.com/paper/2410.00988/) (2024)
- [KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context](https://mlsanity.com/paper/2412.07251/) (2024)
- [Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training](https://mlsanity.com/paper/2403.15470/) (2024)
- [GreenMind: A Next-Generation Vietnamese Large Language Model for Structured and Logical Reasoning](https://mlsanity.com/paper/2504.16832/) (2025)
- [From Course to Skill: Evaluating LLM Performance in Curricular Analytics](https://mlsanity.com/paper/2505.02324/) (2025)
- [A Large-Scale Benchmark for Vietnamese Sentence Paraphrases](https://mlsanity.com/paper/2502.07188/) (2025)

## Abstract (arXiv)

We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese. VIVID comprises 1,636 idioms and proverbs annotated with five complexity traits (literal expressions, pragmatic nuances, Sino-Vietnamese terms, uncommon vocabulary, folk knowledge) and seven semantic themes. We establish an evaluation framework combining generative and discriminative tasks, proposing an LLM-as-a-Judge approach with aspect-based prompting validated against human judgment (Cohen's kappa = 0.792). Evaluating eight state-of-the-art models reveals critical gaps: Vietnamese-specialized models drastically underperform multilingual systems (VinaLLaMA-7B: 0.13 vs. GPT-4o: 2.46), and even top models achieve less than 50% of maximum scores. Notably, few-shot prompting does not universally improve performance, with GPT-4o exhibiting degradation due to stylistic overfitting. Our analysis exposes systematic failures including literal over-interpretation, lexical gaps, and pragmatic flattening, demonstrating that current models lack cultural competence for nuanced figurative interpretation. VIVID provides an essential tool for advancing figurative language understanding in culturally rich contexts.