# AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization

arXiv: https://arxiv.org/abs/2407.11591 · HTML: https://mlsanity.com/paper/2407.11591/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [Investigating Efficiently Extending Transformers for Long Input Summarization](https://mlsanity.com/paper/2208.04347/) (2022)
- [Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models](https://mlsanity.com/paper/2310.10449/) (2023)
- [Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?](https://mlsanity.com/paper/2402.00841/) (2024)

## Built on by (most cited first, 9 of 9)

- [AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation](https://mlsanity.com/paper/2503.19693/) (2025)
- [DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization](https://mlsanity.com/paper/2410.15687/) (2024)
- [Evaluating Large Language Models on Financial Report Summarization: An Empirical Study](https://mlsanity.com/paper/2411.06852/) (2024)
- [Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guarantees](https://mlsanity.com/paper/2508.20736/) (2025)
- [Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents](https://mlsanity.com/paper/2510.14453/) (2025)
- [DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization](https://mlsanity.com/paper/2510.05858/) (2025)
- [An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift](https://mlsanity.com/paper/2601.05882/) (2026)
- [Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization](https://mlsanity.com/paper/2509.01314/) (2025)
- [Generating Realistic, Protocol-Compliant Maritime Radio Dialogues using Self-Instruct and Low-Rank Adaptation](https://mlsanity.com/paper/2603.04423/) (2026)

## Abstract (arXiv)

Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.