Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model

arXiv 2609.24177 ↗First posted 21 Sep 2026cs.CLcited by 0

On the map: Retrieval-augmented generation → Legal and financial RAG

Abstract (arXiv)

Legal information in Bangladesh is inaccessible to most citizens. Statutory text is English-only, trained lawyers are concentrated in urban centres, and cloud-dependent AI fails where mobile connectivity is unreliable, a setting in which hallucinated legal text causes direct harm. The system addresses statutory interpretation only; queries that require judicial precedent or case-law reasoning fall outside its scope. We target the statutory access gap by compressing a 9-billion-parameter Gemma-2 teacher into a 2-billion-parameter student through two-phase progressive knowledge distillation. Phase 1 performs supervised fine-tuning on 9,429 quality-gated legal question-answer pairs (65% acceptance from 14,514 generated queries); Phase 2 minimises sparse Kullback-Leibler divergence against the teacher's top-50 per-token logits at temperature tau = 4.0, implemented via QLoRA (4-bit NF4, rank-32 LoRA adapters). Prior legal language models target general legal English; this system specialises in Bangladeshi statutory law. Every response is grounded through hybrid retrieval combining dense semantic search (60%) and BM25 (40%) across 36,029 statutory passages from the Bangladesh Constitution and national legislation. On a 50-query English benchmark, the distilled model reaches ROUGE-L 0.4715 and BERTScore F1 0.5679, a 103% ROUGE-L and 143% BERTScore gain over the retrieval-augmented undistilled baseline (ROUGE-L 0.2323, BERTScore 0.2340). The adapter quantises to 1.6 GB (GGUF Q4_K_M) and runs at 4-8 tokens per second on a Pixel 6 with no network access. Cross-lingual evaluation on 50 Bangla queries yields ROUGE-L 0.4083 and BERTScore 0.8133, showing effective retrieval from Bangla input against an English-only corpus. In a single-evaluator pilot, a practising lawyer rated 50 responses at a weighted mean of 4.16/5 (90% rated 4 or 5), supporting utility beyond text-overlap metrics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 24 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

19 of 24 references.

Closest ancestors · 2

Same direction, newest first

Other references · 17

2015Distilling the Knowledge in a Neural Networkcited by 10,076
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 33
2025A Hybrid Approach to Information Retrieval and Answer Generation for Regulatory Textscited by 8
2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instructcited by 7

5 more

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way