# LoFiT: Localized Fine-tuning on LLM Representations

arXiv: https://arxiv.org/abs/2406.01563 · HTML: https://mlsanity.com/paper/2406.01563/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 58 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)

## Built on by (most cited first, 30 of 57)

- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Steer LLM Latents for Hallucination Detection](https://mlsanity.com/paper/2503.01917/) (2025)
- [Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents](https://mlsanity.com/paper/2502.06975/) (2025)
- [The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations](https://mlsanity.com/paper/2509.12886/) (2025)
- [To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Models](https://mlsanity.com/paper/2510.13290/) (2025)
- [When One LLM Drools, Multi-LLM Collaboration Rules](https://mlsanity.com/paper/2502.04506/) (2025)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning](https://mlsanity.com/paper/2507.04250/) (2025)
- [Spherical Steering: Geometry-Aware Activation Rotation for Language Models](https://mlsanity.com/paper/2602.08169/) (2026)
- [Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers](https://mlsanity.com/paper/2505.13737/) (2025)
- [Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models](https://mlsanity.com/paper/2508.10599/) (2025)
- [Position: Agentic Evolution is the Path to Evolving LLMs](https://mlsanity.com/paper/2602.00359/) (2026)
- [LinEAS: End-to-end Learning of Activation Steering with a Distributional Loss](https://mlsanity.com/paper/2503.10679/) (2025)
- [Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Models](https://mlsanity.com/paper/2602.01654/) (2026)
- [KV Cache Steering for Controlling Frozen LLMs](https://mlsanity.com/paper/2507.08799/) (2025)
- [Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes](https://mlsanity.com/paper/2410.16930/) (2024)
- [Understanding Synthetic Context Extension via Retrieval Heads](https://mlsanity.com/paper/2410.22316/) (2024)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [Joint Localization and Activation Editing for Low-Resource Fine-Tuning](https://mlsanity.com/paper/2502.01179/) (2025)
- [Permissioned LLMs: Enforcing Access Control in Large Language Models](https://mlsanity.com/paper/2505.22860/) (2025)
- [Probe-Free Low-Rank Activation Intervention](https://mlsanity.com/paper/2502.04043/) (2025)
- [DISCO: Disentangled Communication Steering for Large Language Models](https://mlsanity.com/paper/2509.16820/) (2025)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models](https://mlsanity.com/paper/2504.11381/) (2025)
- [DenseLoRA: Dense Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2505.23808/) (2025)
- [Learning Distribution-Wise Control in Representation Space for Language Models](https://mlsanity.com/paper/2506.06686/) (2025)
- [Learning Composable Chains-of-Thought](https://mlsanity.com/paper/2505.22635/) (2025)
- [Dynamic Weight Grafting: Localizing Finetuned Factual Knowledge in Transformers](https://mlsanity.com/paper/2506.20746/) (2025)

## Abstract (arXiv)

Recent work in interpretability shows that large language models (LLMs) can be adapted for new tasks in a learning-free way: it is possible to intervene on LLM representations to elicit desired behaviors for alignment. For instance, adding certain bias vectors to the outputs of certain attention heads is reported to boost the truthfulness of models. In this work, we show that localized fine-tuning serves as an effective alternative to such representation intervention methods. We introduce a framework called Localized Fine-Tuning on LLM Representations (LoFiT), which identifies a subset of attention heads that are most important for learning a specific task, then trains offset vectors to add to the model's hidden representations at those selected heads. LoFiT localizes to a sparse set of heads (3%-10%) and learns the offset vectors from limited training data, comparable to the settings used for representation intervention. For truthfulness and reasoning tasks, we find that LoFiT's intervention vectors are more effective for LLM adaptation than vectors from representation intervention methods such as Inference-time Intervention. We also find that the localization step is important: selecting a task-specific set of attention heads can lead to higher performance than intervening on heads selected for a different task. Finally, across 7 tasks we study, LoFiT achieves comparable performance to other parameter-efficient fine-tuning methods such as LoRA, despite modifying 20x-200x fewer parameters than these methods.