# BRoverbs -- Measuring how much LLMs understand Portuguese proverbs

arXiv: https://arxiv.org/abs/2509.08960 · HTML: https://mlsanity.com/paper/2509.08960/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 30 of 35)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [DynaSent: A Dynamic Benchmark for Sentiment Analysis](https://mlsanity.com/paper/2012.15349/) (2020)
- [PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data](https://mlsanity.com/paper/2406.15053/) (2024)
- [Cabrita: closing the gap for foreign languages](https://mlsanity.com/paper/2308.11878/) (2023)
- [Building a Sentiment Corpus of Tweets in Brazilian Portuguese](https://mlsanity.com/paper/1712.08917/) (2017)
- [HateBR: A Large Expert Annotated Corpus of Brazilian Instagram Comments for Offensive Language and Hate Speech Detection](https://mlsanity.com/paper/2103.14972/) (2021)
- [BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams](https://mlsanity.com/paper/2307.05410/) (2023)
- [ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding](https://mlsanity.com/paper/2411.05049/) (2024)

## Built on by (most cited first, 3 of 3)

- [PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese](https://mlsanity.com/paper/2511.17808/) (2025)
- [MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs](https://mlsanity.com/paper/2601.22050/) (2026)
- [Progressing beyond Art Masterpieces or Touristic Clich'es: how to assess your LLMs for cultural alignment?](https://mlsanity.com/paper/2604.25654/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit significant performance variations depending on the linguistic and cultural context in which they are applied. This disparity signals the necessity of mature evaluation frameworks that can assess their capabilities in specific regional settings. In the case of Portuguese, existing evaluations remain limited, often relying on translated datasets that may not fully capture linguistic nuances or cultural references. Meanwhile, native Portuguese-language datasets predominantly focus on structured national exams or sentiment analysis of social media interactions, leaving gaps in evaluating broader linguistic understanding. To address this limitation, we introduce BRoverbs, a dataset specifically designed to assess LLM performance through Brazilian proverbs. Proverbs serve as a rich linguistic resource, encapsulating cultural wisdom, figurative expressions, and complex syntactic structures that challenge the model comprehension of regional expressions. BRoverbs aims to provide a new evaluation tool for Portuguese-language LLMs, contributing to advancing regionally informed benchmarking. The benchmark is available at https://huggingface.co/datasets/Tropic-AI/BRoverbs.