# SaulLM-7B: A pioneering Large Language Model for Law

arXiv: https://arxiv.org/abs/2403.03883 · HTML: https://mlsanity.com/paper/2403.03883/

## Key facts

- Cited by 142 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 76 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 30 of 55)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)

## Built on by (most cited first, 30 of 142)

- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Synthetic continued pretraining](https://mlsanity.com/paper/2409.07431/) (2024)
- [SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)](https://mlsanity.com/paper/2406.17975/) (2024)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models](https://mlsanity.com/paper/2407.09424/) (2024)
- [SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain](https://mlsanity.com/paper/2407.19584/) (2024)
- [GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory](https://mlsanity.com/paper/2406.11149/) (2024)
- [LawLLM: Law Large Language Model for the US Legal System](https://mlsanity.com/paper/2407.21065/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025)
- [ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?](https://mlsanity.com/paper/2507.05639/) (2025)
- [WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs](https://mlsanity.com/paper/2503.05683/) (2025)
- [Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications](https://mlsanity.com/paper/2409.05314/) (2024)
- [Demystifying Domain-adaptive Post-training for Financial LLMs](https://mlsanity.com/paper/2501.04961/) (2025)
- [CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models](https://mlsanity.com/paper/2407.17467/) (2024)
- [Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training](https://mlsanity.com/paper/2502.11191/) (2025)
- [Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report](https://mlsanity.com/paper/2508.01059/) (2025)
- [Sparsing Law: Towards Large Language Models with Greater Activation Sparsity](https://mlsanity.com/paper/2411.02335/) (2024)
- [Can Large Language Models Generate High-quality Patent Claims?](https://mlsanity.com/paper/2406.19465/) (2024)
- [FinMTEB: Finance Massive Text Embedding Benchmark](https://mlsanity.com/paper/2502.10990/) (2025)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models](https://mlsanity.com/paper/2508.09874/) (2025)
- [Do We Need Domain-Specific Embedding Models? An Empirical Investigation](https://mlsanity.com/paper/2409.18511/) (2024)
- [Automatic Legal Writing Evaluation of LLMs](https://mlsanity.com/paper/2504.21202/) (2025)
- [Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement](https://mlsanity.com/paper/2408.03092/) (2024)
- [Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains](https://mlsanity.com/paper/2501.14431/) (2025)
- [LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain](https://mlsanity.com/paper/2404.02127/) (2024)
- [The Factuality of Large Language Models in the Legal Domain](https://mlsanity.com/paper/2409.11798/) (2024)

## Abstract (arXiv)

In this paper, we introduce SaulLM-7B, a large language model (LLM) tailored for the legal domain. With 7 billion parameters, SaulLM-7B is the first LLM designed explicitly for legal text comprehension and generation. Leveraging the Mistral 7B architecture as its foundation, SaulLM-7B is trained on an English legal corpus of over 30 billion tokens. SaulLM-7B exhibits state-of-the-art proficiency in understanding and processing legal documents. Additionally, we present a novel instructional fine-tuning method that leverages legal datasets to further enhance SaulLM-7B's performance in legal tasks. SaulLM-7B is released under the MIT License.