# Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages

arXiv: https://arxiv.org/abs/2310.04799 · HTML: https://mlsanity.com/paper/2310.04799/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 7 Oct 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [Composing Parameter-Efficient Modules with Arithmetic Operations](https://mlsanity.com/paper/2306.14870/) (2023)
- [Elastic Weight Removal for Faithful and Abstractive Dialogue Generation](https://mlsanity.com/paper/2303.17574/) (2023)

## Built on by (most cited first, 30 of 54)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [What Matters for Model Merging at Scale?](https://mlsanity.com/paper/2410.03617/) (2024)
- [Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing](https://mlsanity.com/paper/2407.08770/) (2024)
- [Personality Vector: Modulating Personality of Large Language Models by Model Merging](https://mlsanity.com/paper/2509.19727/) (2025)
- [Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning](https://mlsanity.com/paper/2502.11019/) (2025)
- [Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition](https://mlsanity.com/paper/2406.02925/) (2024)
- [Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach](https://mlsanity.com/paper/2412.11679/) (2024)
- [Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging](https://mlsanity.com/paper/2412.19512/) (2024)
- [Fingerprint Vector: Enabling Scalable and Efficient Model Fingerprint Transfer via Vector Addition](https://mlsanity.com/paper/2409.08846/) (2024)
- [Shadow-FT: Tuning Instruct Model via Training on Paired Base Model](https://mlsanity.com/paper/2505.12716/) (2025)
- [Adapting Chat Language Models Using Only Target Unlabeled Language Data](https://mlsanity.com/paper/2412.11704/) (2024)
- [Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge](https://mlsanity.com/paper/2502.20186/) (2025)
- [Efficient Model Development through Fine-tuning Transfer](https://mlsanity.com/paper/2503.20110/) (2025)
- [Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors](https://mlsanity.com/paper/2504.20106/) (2025)
- [LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agents](https://mlsanity.com/paper/2505.21963/) (2025)
- [DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging](https://mlsanity.com/paper/2407.01470/) (2024)
- [Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models](https://mlsanity.com/paper/2506.06844/) (2025)
- [Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy](https://mlsanity.com/paper/2502.11533/) (2025)
- [Investigating Task Arithmetic for Zero-Shot Information Retrieval](https://mlsanity.com/paper/2505.00649/) (2025)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [Persona Cartography: Charting Language Model Personality Traits in Weight Space](https://mlsanity.com/paper/2607.07916/) (2026)
- [In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models](https://mlsanity.com/paper/2503.01611/) (2025)
- [V-Agent: An Interactive Video Search System Using Vision-Language Models](https://mlsanity.com/paper/2512.16925/) (2025)
- [Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis](https://mlsanity.com/paper/2512.18699/) (2025)
- [A Survey on Training-free Alignment of Large Language Models](https://mlsanity.com/paper/2508.09016/) (2025)
- [Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance](https://mlsanity.com/paper/2511.02451/) (2025)
- [Bus-Conditioned Zero-Shot Trajectory Generation via Task Arithmetic](https://mlsanity.com/paper/2602.13071/) (2026)
- [PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation](https://mlsanity.com/paper/2503.01233/) (2025)
- [Detoxification of Large Language Models through Output-layer Fusion with a Calibration Model](https://mlsanity.com/paper/2506.01266/) (2025)

## Abstract (arXiv)

Recently, the development of open-source large language models (LLMs) has advanced rapidly. Nevertheless, due to data constraints, the capabilities of most open-source LLMs are primarily focused on English. To address this issue, we introduce the concept of textitchat vector to equip pre-trained language models with instruction following and human value alignment via simple model arithmetic. The chat vector is derived by subtracting the weights of a pre-trained base model (e.g. LLaMA2) from those of its corresponding chat model (e.g. LLaMA2-chat). By simply adding the chat vector to a continual pre-trained model's weights, we can endow the model with chat capabilities in new languages without the need for further training. Our empirical studies demonstrate the superior efficacy of the chat vector from three different aspects: instruction following, toxicity mitigation, and multi-turn dialogue. Moreover, to showcase the adaptability of our approach, we extend our experiments to encompass various languages, base models, and chat vectors. The results underscore the chat vector's simplicity, effectiveness, and wide applicability, making it a compelling solution for efficiently enabling conversational capabilities in pre-trained language models. Our code is available at https://github.com/aqweteddy/ChatVector.