# A Llama walks into the 'Bar': Efficient Supervised Fine-Tuning for Legal Reasoning in the Multi-state Bar Exam arXiv: https://arxiv.org/abs/2504.04945 ยท HTML: https://mlsanity.com/paper/2504.04945/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. - Builds on 29 of its 79 references that are in the map. - First posted on arXiv 7 Apr 2025. ## Placed in - LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/) ## Builds on (most cited first, 29 of 29) - [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023) - [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023) - [MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics](https://mlsanity.com/paper/2109.00110/) (2021) - [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020) - [Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs](https://mlsanity.com/paper/2210.12283/) (2022) - [Autoformalization with Large Language Models](https://mlsanity.com/paper/2205.12615/) (2022) - [ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics](https://mlsanity.com/paper/2302.12433/) (2023) - [PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition](https://mlsanity.com/paper/2407.11214/) (2024) - [Baldur: Whole-Proof Generation and Repair with Large Language Models](https://mlsanity.com/paper/2303.04910/) (2023) - [LEGO-Prover: Neural Theorem Proving with Growing Libraries](https://mlsanity.com/paper/2310.00656/) (2023) - [Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers](https://mlsanity.com/paper/2205.10893/) (2022) - [An In-Context Learning Agent for Formal Theorem-Proving](https://mlsanity.com/paper/2310.04353/) (2023) - [Formal Mathematical Reasoning: A New Frontier in AI](https://mlsanity.com/paper/2412.16075/) (2024) - [STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving](https://mlsanity.com/paper/2502.00212/) (2025) - [First Experiments with Neural Translation of Informal to Formal Mathematics](https://mlsanity.com/paper/1805.06502/) (2018) - [Learning Formal Mathematics From Intrinsic Motivation](https://mlsanity.com/paper/2407.00695/) (2024) - [Learning to Infer Program Sketches](https://mlsanity.com/paper/1902.06349/) (2019) - [A formal proof of the Kepler conjecture](https://mlsanity.com/paper/1501.02155/) (2015) - [LeanAgent: Lifelong Learning for Formal Theorem Proving](https://mlsanity.com/paper/2410.06209/) (2024) - [Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification](https://mlsanity.com/paper/2412.14063/) (2024) - [First Neural Conjecturing Datasets and Experiments](https://mlsanity.com/paper/2005.14664/) (2020) - [Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions](https://mlsanity.com/paper/2502.12065/) (2025) - [Learning Program Synthesis for Integer Sequences from Scratch](https://mlsanity.com/paper/2202.11908/) (2022) - [Abstraction boundaries and spec driven development in pure mathematics](https://mlsanity.com/paper/2309.14870/) (2023) - [Proof-Pattern Recognition and Lemma Discovery in ACL2](https://mlsanity.com/paper/1308.1780/) (2013) - [Theory Exploration Powered By Deductive Synthesis](https://mlsanity.com/paper/2009.04826/) (2020) - [Template-Based Conjecturing for Automated Induction in Isabelle/HOL](https://mlsanity.com/paper/2212.11151/) (2022) - [Learning Conjecturing from Scratch](https://mlsanity.com/paper/2503.01389/) (2025) - [Hipster: Integrating Theory Exploration in a Proof Assistant](https://mlsanity.com/paper/1405.3426/) (2014) ## Built on by (most cited first, 3 of 3) - [LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis](https://mlsanity.com/paper/2510.17602/) (2025) - [How Does Prefix Matter in Reasoning Model Tuning?](https://mlsanity.com/paper/2601.01624/) (2026) - [CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data](https://mlsanity.com/paper/2607.16122/) (2026) ## Abstract (arXiv) Legal reasoning tasks present unique challenges for large language models (LLMs) due to the complexity of domain-specific knowledge and reasoning processes. This paper investigates how effectively smaller language models (Llama 2 7B and Llama 3 8B) can be fine-tuned with a limited dataset of 1,514 Multi-state Bar Examination (MBE) questions to improve legal question answering accuracy. We evaluate these models on the 2022 MBE questions licensed from JD Advising, the same dataset used in the 'GPT-4 passes the Bar exam' study. Our methodology involves collecting approximately 200 questions per legal domain across 7 domains. We distill the dataset using Llama 3 (70B) to transform explanations into a structured IRAC (Issue, Rule, Application, Conclusion) format as a guided reasoning process to see if it results in better performance over the non-distilled dataset. We compare the non-fine-tuned models against their supervised fine-tuned (SFT) counterparts, trained for different sample sizes per domain, to study the effect on accuracy and prompt adherence. We also analyse option selection biases and their mitigation following SFT. In addition, we consolidate the performance across multiple variables: prompt type (few-shot vs zero-shot), answer ordering (chosen-option first vs generated-explanation first), response format (Numbered list vs Markdown vs JSON), and different decoding temperatures. Our findings show that domain-specific SFT helps some model configurations achieve close to human baseline performance, despite limited computational resources and a relatively small dataset. We release both the gathered SFT dataset and the family of Supervised Fine-tuned (SFT) adapters optimised for MBE performance. This establishes a practical lower bound on resources needed towards achieving effective legal question answering in smaller LLMs.