LEGAL-UQA: A Low-Resource Urdu-English Dataset for Legal Question Answering
Not on the map. Why?
Abstract (arXiv)
We present LEGAL-UQA, the first Urdu legal question-answering dataset derived from Pakistan's constitution. This parallel English-Urdu dataset includes 619 question-answer pairs, each with corresponding legal article contexts, addressing the need for domain-specific NLP resources in low-resource languages. We describe the dataset creation process, including OCR extraction, manual refinement, and GPT-4-assisted translation and generation of QA pairs. Our experiments evaluate the latest generalist language and embedding models on LEGAL-UQA, with Claude-3.5-Sonnet achieving 99.19% human-evaluated accuracy. We fine-tune mt5-large-UQA-1.0, highlighting the challenges of adapting multilingual models to specialized domains. Additionally, we assess retrieval performance, finding OpenAI's text-embedding-3-large outperforms Mistral's mistral-embed. LEGAL-UQA bridges the gap between global NLP advancements and localized applications, particularly in constitutional law, and lays the foundation for improved legal information access in Pakistan.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 29 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
11 of 29 references.
References · 11
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
- 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 122
- 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
- 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
- 2024UQA: Corpus for Urdu Question Answeringcited by 8
- 2021UQuAD1.0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehensioncited by 6
- 2023UTRNet: High-Resolution Urdu Text Recognition In Printed Documentscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.