# Exploring Large Language Models for Relevance Judgments in Tetun

arXiv: https://arxiv.org/abs/2406.07299 · HTML: https://mlsanity.com/paper/2406.07299/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2024.

## Built on by (most cited first, 3 of 3)

- [Low-resource Machine Translation: what for? who for? An observational study on a dedicated Tetun language translation service](https://mlsanity.com/paper/2411.12262/) (2024)
- [Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks](https://mlsanity.com/paper/2602.06526/) (2026)
- [When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search](https://mlsanity.com/paper/2507.02139/) (2025)

## Abstract (arXiv)

The Cranfield paradigm has served as a foundational approach for developing test collections, with relevance judgments typically conducted by human assessors. However, the emergence of large language models (LLMs) has introduced new possibilities for automating these tasks. This paper explores the feasibility of using LLMs to automate relevance assessments, particularly within the context of low-resource languages. In our study, LLMs are employed to automate relevance judgment tasks, by providing a series of query-document pairs in Tetun as the input text. The models are tasked with assigning relevance scores to each pair, where these scores are then compared to those from human annotators to evaluate the inter-annotator agreement levels. Our investigation reveals results that align closely with those reported in studies of high-resource languages.