# GhazalBench: Canonical Verse Access in LLMs across Persian Ghazals and Shakespearean Sonnets

arXiv: https://arxiv.org/abs/2603.09979 · HTML: https://mlsanity.com/paper/2603.09979/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?](https://mlsanity.com/paper/2404.06644/) (2024)
- [Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets](https://mlsanity.com/paper/2406.18906/) (2024)
- [Evaluating Diversity in Automatic Poetry Generation](https://mlsanity.com/paper/2406.15267/) (2024)
- [Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs](https://mlsanity.com/paper/2505.18152/) (2025)
- [Does ChatGPT Have a Poetic Style?](https://mlsanity.com/paper/2410.15299/) (2024)
- [ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data](https://mlsanity.com/paper/2504.14452/) (2025)
- [Beyond Tools: Understanding How Heavy Users Integrate LLMs into Everyday Tasks and Decision-Making](https://mlsanity.com/paper/2502.15395/) (2025)
- [Evaluating the Creativity of LLMs in Persian Literary Text Generation](https://mlsanity.com/paper/2509.18401/) (2025)
- [Improving Instruct Models for Free: A Study on Partial Adaptation](https://mlsanity.com/paper/2504.11626/) (2025)
- [FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Language](https://mlsanity.com/paper/2601.20105/) (2026)
- [MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs](https://mlsanity.com/paper/2601.22050/) (2026)

## Built on by (most cited first, 1 of 1)

- [PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions](https://mlsanity.com/paper/2605.27015/) (2026)

## Abstract (arXiv)

Persian poetry plays an active role in Iranian cultural practice, where verses by canonical poets such as Hafez and Saadi are frequently quoted, paraphrased, or completed from incomplete cues. Supporting such interactions requires language models to reliably access canonical verses from semantic and lexical information. We introduce GhazalBench, a benchmark for evaluating how large language models (LLMs) access the canonical surface forms of Persian ghazal under usage-grounded conditions. Unlike prior work that primarily treats memorization as a liability, GhazalBench studies settings in which access to exact wording is functionally important. The benchmark evaluates completion and recognition under varied semantic and lexical cues. Across proprietary and open-weight multilingual LLMs, exact completion remains challenging, while recognition is substantially stronger. Performance also varies across poets and model families. Parallel experiments on Shakespearean sonnets yield markedly higher completion for several models, consistent with differences in exposure to canonical texts. {Additional experiments suggest that post-training may reduce direct continuation-based access to canonical text.} Our findings motivate evaluation frameworks that distinguish production from recognition and assess access to culturally significant canonical texts. GhazalBench is available at https://github.com/kalhorghazal/GhazalBench.