# Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh

arXiv: https://arxiv.org/abs/2607.23446 · HTML: https://mlsanity.com/paper/2607.23446/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Builds on (most cited first, 16 of 16)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024)
- [Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates](https://mlsanity.com/paper/2402.18540/) (2024)
- [IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning](https://mlsanity.com/paper/2407.05399/) (2024)
- [Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?](https://mlsanity.com/paper/2507.23248/) (2025)
- [KoBLEX: Open Legal Question Answering with Multi-hop Reasoning](https://mlsanity.com/paper/2509.01324/) (2025)
- [LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval](https://mlsanity.com/paper/2504.16121/) (2025)
- [Exploring Possibilities of AI-Powered Legal Assistance in Bangladesh through Large Language Modeling](https://mlsanity.com/paper/2410.17210/) (2024)
- [Assessing the Reliability of Large Language Models in the Bengali Legal Context: A Comparative Evaluation Using LLM-as-Judge and Legal Experts](https://mlsanity.com/paper/2511.05627/) (2025)
- [Mina: A Multilingual LLM-Powered Legal Assistant Agent for Bangladesh for Empowering Access to Justice](https://mlsanity.com/paper/2511.08605/) (2025)

## Abstract (arXiv)

A small language model can receive the governing statutory provision and still answer incorrectly. We test whether fine-tuning on examples containing relevant law improves later use of retrieved law. We curate 2,165 bilingual QA records from six Bangladeshi acts and three schedules, then fine-tune Qwen3.5 at 0.8B, 2B, and 4B. Evaluation uses the 2022 and 2023 Bangladesh Bar Council exams in Bangla and machine-translated English, with no retrieval, BM25, or FAISS, scored by strict consistency over three seeded runs. At 0.8B, fine-tuning raises the 2022 English FAISS score from 2 to 34 of 100. Gains at 0.8B and 2B survive paired testing, but the 4B model has no detectable net gain: Bangla improves while several English conditions regress. Fine-tuning also reduces answers that drift from Bangla into mostly English from 44.0--53.2% to 0.2--0.7%, with adjusted p<.001 at every scale. Retrieval quality is therefore not the only bottleneck. Small bilingual legal models also differ in how they use supplied law and whether they answer in the requested language. The dataset is publicly available at https://huggingface.co/datasets/momahadi/bangladesh-legal-qa-dataset.