# TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA

arXiv: https://arxiv.org/abs/2510.04682 · HTML: https://mlsanity.com/paper/2510.04682/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Knowledge Fusion of Large Language Models](https://mlsanity.com/paper/2401.10491/) (2024)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [Train No Evil: Selective Masking for Task-Guided Pre-Training](https://mlsanity.com/paper/2004.09733/) (2020)
- [Evaluating Language Models as Synthetic Data Generators](https://mlsanity.com/paper/2412.03679/) (2024)
- [Small Scale Data-Free Knowledge Distillation](https://mlsanity.com/paper/2406.07876/) (2024)
- [textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuning](https://mlsanity.com/paper/2405.17258/) (2024)
- [Personalized LLM Decoding via Contrasting Personal Preference](https://mlsanity.com/paper/2506.12109/) (2025)
- [KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation](https://mlsanity.com/paper/2410.20777/) (2024)
- [Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning](https://mlsanity.com/paper/2407.16920/) (2024)
- [ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining](https://mlsanity.com/paper/2505.19893/) (2025)

## Built on by (most cited first, 2 of 2)

- [RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents](https://mlsanity.com/paper/2609.23466/) (2026)
- [UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists](https://mlsanity.com/paper/2608.20918/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are widely applied in real world scenarios, yet fine-tuning them comes with significant computational and storage costs. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA mitigate these costs; however, the adapted parameters are dependent on the base model and cannot be transferred across different backbones. One way to address this issue is through knowledge distillation, but its effectiveness inherently depends on training data. Recent work such as TransLoRA avoids this by generating synthetic data; nevertheless, this adds complexity since it requires training an additional discriminator model. In this paper, we propose TiTok, a new framework that enables effective LoRA Transplantation through Token-level knowledge transfer. Specifically, TiTok captures task-relevant information through a token-wise contrastive excess between a source model with and without LoRA. This excess highlights informative tokens and enables selective filtering of synthetic data, all without additional models or overhead. Through experiments on three benchmarks across multiple transfer settings, we demonstrate that TiTok is consistently effective, achieving average performance gains of +4~10% compared to baselines overall.