# Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching

arXiv: https://arxiv.org/abs/2507.05617 · HTML: https://mlsanity.com/paper/2507.05617/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023)
- [Less is More: Task-aware Layer-wise Distillation for Language Model Compression](https://mlsanity.com/paper/2210.01351/) (2022)
- [How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://mlsanity.com/paper/2303.00293/) (2023)
- [Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty](https://mlsanity.com/paper/2308.02019/) (2023)
- [Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.10708/) (2025)
- [Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint](https://mlsanity.com/paper/2401.06081/) (2024)
- [Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories](https://mlsanity.com/paper/2306.05406/) (2023)
- [Distilling Large Vision-Language Model with Out-of-Distribution Generalizability](https://mlsanity.com/paper/2307.03135/) (2023)
- [KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning](https://mlsanity.com/paper/2403.17486/) (2024)
- [Modeling User Behavior with Graph Convolution for Personalized Product Search](https://mlsanity.com/paper/2202.06081/) (2022)
- [Flexible and Adaptable Summarization via Expertise Separation](https://mlsanity.com/paper/2406.05360/) (2024)
- [Large Language Models for Failure Mode Classification: An Investigation](https://mlsanity.com/paper/2309.08181/) (2023)
- [Enhancing In-Context Learning via Implicit Demonstration Augmentation](https://mlsanity.com/paper/2407.00100/) (2024)
- [RPM-Oriented Query Rewriting Framework for E-commerce Keyword-Based Sponsored Search](https://mlsanity.com/paper/1910.12527/) (2019)
- [Improving Zero-Shot Text Matching for Financial Auditing with Large Language Models](https://mlsanity.com/paper/2308.06111/) (2023)
- [Multi-Intent Attribute-Aware Text Matching in Searching](https://mlsanity.com/paper/2402.07788/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [Interfaze: The Future of AI is built on Task-Specific Small Models](https://mlsanity.com/paper/2602.04101/) (2026)
- [One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems](https://mlsanity.com/paper/2606.09293/) (2026)

## Abstract (arXiv)

Knowledge distillation typically involves transferring knowledge from a Large Language Model (LLM) to a Smaller Language Model (SLM). However, in tasks such as text matching, fine-tuned smaller models often yield more effective domain-specific representations, as they focus on optimizing the similarity of input pairs. To leverage both the specialized strengths of small models and the rich semantic understanding of LLMs, we introduce a flipped knowledge distillation paradigm, where LLM learns from SLM. Specifically, we address the architectural gap between decoder-only LLMs and smaller encoder-based models by reinterpreting LLMs in an encoder-decoder manner using LoRA. The encoder generates compressed representations, while the decoder maps them to the output space. During training, the encoder produces representations and their similarities, which are then aligned with the similarity scores produced by the teacher, using our proposed Margin-aware Contrastive Learning (MCL) approach. The MCL ensures accurate similarity for both positive and negative pairs, and adaptively handles the internal differences within positive and negative samples. Our paradigm requires only a reasonably good-performing SLM, allowing the LLM to achieve improved performance. Experiments on financial and healthcare benchmarks, as well as real-world applications, confirm its effectiveness, and the model has been fully deployed in an online environment.