# L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages

arXiv: https://arxiv.org/abs/2509.02503 · HTML: https://mlsanity.com/paper/2509.02503/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024)
- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024)
- [IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?](https://mlsanity.com/paper/2410.02611/) (2024)

## Built on by (most cited first, 1 of 1)

- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)

## Abstract (arXiv)

Semantic evaluation in low-resource languages remains a major challenge in NLP. While sentence transformers have shown strong performance in high-resource settings, their effectiveness in Indic languages is underexplored due to a lack of high-quality benchmarks. To bridge this gap, we introduce L3Cube-IndicHeadline-ID, a curated headline identification dataset spanning ten low-resource Indic languages: Marathi, Hindi, Tamil, Gujarati, Odia, Kannada, Malayalam, Punjabi, Telugu, Bengali and English. Each language includes 20,000 news articles paired with four headline variants: the original, a semantically similar version, a lexically similar version, and an unrelated one, designed to test fine-grained semantic understanding. The task requires selecting the correct headline from the options using article-headline similarity. We benchmark several sentence transformers, including multilingual and language-specific models, using cosine similarity. Results show that multilingual models consistently perform well, while language-specific models vary in effectiveness. Given the rising use of similarity models in Retrieval-Augmented Generation (RAG) pipelines, this dataset also serves as a valuable resource for evaluating and improving semantic understanding in such applications. Additionally, the dataset can be repurposed for multiple-choice question answering, headline classification, or other task-specific evaluations of LLMs, making it a versatile benchmark for Indic NLP. The dataset is shared publicly at https://github.com/l3cube-pune/indic-nlp