mlsanityData to 14 Oct 2026

Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts

arXiv 2410.12001 ↗First posted 15 Oct 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper aims to offer AI & Law researchers and practitioners a more detailed understanding of whether and how continued pre-training and instruction fine-tuning (IFT) of large language models (LLMs) on legal corpora increases their utilization of human-defined legal concepts when developing global contextual representations of input sequences. We compared three models: Mistral 7B, SaulLM-7B-Base (Mistral 7B with continued pre-training on legal corpora), and SaulLM-7B-Instruct (with further IFT). This preliminary assessment examined 7 distinct text sequences from recent AI & Law literature, each containing a human-defined legal concept. We first compared the proportions of total attention the models allocated to subsets of tokens representing the legal concepts. We then visualized patterns of raw attention score alterations, evaluating whether legal training introduced novel attention patterns corresponding to structures of human legal knowledge. This inquiry revealed that (1) the impact of legal training was unevenly distributed across the various human-defined legal concepts, and (2) the contextual representations of legal knowledge learned during legal training did not coincide with structures of human-defined legal concepts. We conclude with suggestions for further investigation into the dynamics of legal LLM training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

18 of 30 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Attention is not Explanationcited by 929
  3. 2019Attention is not not Explanationcited by 636
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  6. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  7. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  9. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  10. 2024To Believe or Not to Believe Your LLMcited by 120
  11. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  12. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
6 more
  1. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  2. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 36
  3. 2023Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Textscited by 33
  4. 2023Using Large Language Models to Support Thematic Analysis in Empirical Legal Studiescited by 25
  5. 2023Harnessing GPT-3.5-turbo for Rhetorical Role Prediction in Legal Casescited by 8
  6. 2024Extending Token Computation for LLM Reasoningcited by 3

Built on by

Not cited yet.