# Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue Summarization

arXiv: https://arxiv.org/abs/2401.15496 · HTML: https://mlsanity.com/paper/2401.15496/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 27 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [NEFTune: Noisy Embeddings Improve Instruction Finetuning](https://mlsanity.com/paper/2310.05914/) (2023)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization](https://mlsanity.com/paper/2105.12544/) (2021)
- [CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization](https://mlsanity.com/paper/2108.13139/) (2021)
- [Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling](https://mlsanity.com/paper/2012.07311/) (2020)
- [Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders](https://mlsanity.com/paper/2012.07300/) (2020)
- [Other Roles Matter! Enhancing Role-Oriented Dialogue Summarization via Role Interactions](https://mlsanity.com/paper/2205.13190/) (2022)

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain](https://mlsanity.com/paper/2503.18360/) (2025)
- [Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization](https://mlsanity.com/paper/2507.02145/) (2025)
- [Practical Applications of Advanced Cloud Services and Generative AI Systems in Medical Image Analysis](https://mlsanity.com/paper/2403.17549/) (2024)
- [Financial Text Classification Based On rLoRA Finetuning On Qwen3-8B model](https://mlsanity.com/paper/2512.00630/) (2025)
- [Instruction Finetuning LLaMA-3-8B Model Using LoRA for Financial Named Entity Recognition](https://mlsanity.com/paper/2601.10043/) (2026)
- [Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization](https://mlsanity.com/paper/2608.09043/) (2026)

## Abstract (arXiv)

Large language models (LLMs) like Llama, Baichuan and Bloom models show remarkable ability with instruction fine-tuning in many natural language tasks. Nevertheless, for the dialogue summarization task, which aims to generate summaries for different roles in dialogue, most of the state-of-the-art methods conduct on small models (e.g Bart and Bert). Existing methods try to add task specified optimization on small models like adding global-local centrality score to models. In this paper, we propose an instruction fine-tuning model: Baichuan2-Sum, for role-oriented diaglouge summarization. By setting different instructions for different roles, the model can learn from the dialogue interactions and output the expected summaries. Furthermore, we applied NEFTune technique to add suitable noise during training to improve the results. The experiments demonstrate that the proposed model achieves the new state-of-the-art results on two public dialogue summarization datasets: CSDS and SAMSUM. We release our model and related codes to facilitate future studies on dialogue summarization task.