# Efficient Few-Shot Fine-Tuning for Opinion Summarization

arXiv: https://arxiv.org/abs/2205.02170 · HTML: https://mlsanity.com/paper/2205.02170/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 63 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://mlsanity.com/paper/1808.08858/) (2018)
- [Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation](https://mlsanity.com/paper/1904.03396/) (2019)
- [Unsupervised Opinion Summarization as Copycat-Review Generation](https://mlsanity.com/paper/1911.02247/) (2019)
- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [Aspect-Controllable Opinion Summarization](https://mlsanity.com/paper/2109.03171/) (2021)
- [Unsupervised Opinion Summarization with Noising and Denoising](https://mlsanity.com/paper/2004.10150/) (2020)
- [OpinionDigest: A Simple Framework for Opinion Summarization](https://mlsanity.com/paper/2005.01901/) (2020)
- [Controllable Neural Dialogue Summarization with Personal Named Entity Planning](https://mlsanity.com/paper/2109.13070/) (2021)

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation](https://mlsanity.com/paper/2305.14647/) (2023)
- [OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization](https://mlsanity.com/paper/2310.18122/) (2023)
- [Bias in Opinion Summarisation from Pre-training to Adaptation: A Case Study in Political Bias](https://mlsanity.com/paper/2402.00322/) (2024)
- [Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets](https://mlsanity.com/paper/2310.13340/) (2023)
- [Beyond Opinion Mining: Summarizing Opinions of Customer Reviews](https://mlsanity.com/paper/2206.01543/) (2022)
- [Self-Improving Customer Review Response Generation Based on LLMs](https://mlsanity.com/paper/2405.03845/) (2024)
- [Noisy Pairing and Partial Supervision for Stylized Opinion Summarization](https://mlsanity.com/paper/2211.08723/) (2022)
- [OpineSum: Entailment-based self-training for abstractive opinion summarization](https://mlsanity.com/paper/2212.10791/) (2022)
- [Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning](https://mlsanity.com/paper/2305.12077/) (2023)
- [To Adapt or to Fine-tune: A Case Study on Abstractive Summarization](https://mlsanity.com/paper/2208.14559/) (2022)
- [Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization](https://mlsanity.com/paper/2303.11660/) (2023)
- [Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions](https://mlsanity.com/paper/2508.17610/) (2025)
- [PELMS: Pre-training for Effective Low-Shot Multi-Document Summarization](https://mlsanity.com/paper/2311.09836/) (2023)
- [TL;DR Progress: Multi-faceted Literature Exploration in Text Summarization](https://mlsanity.com/paper/2402.06913/) (2024)
- [ReflectSumm: A Benchmark for Course Reflection Summarization](https://mlsanity.com/paper/2403.19012/) (2024)

## Abstract (arXiv)

Abstractive summarization models are typically pre-trained on large amounts of generic texts, then fine-tuned on tens or hundreds of thousands of annotated samples. However, in opinion summarization, large annotated datasets of reviews paired with reference summaries are not available and would be expensive to create. This calls for fine-tuning methods robust to overfitting on small datasets. In addition, generically pre-trained models are often not accustomed to the specifics of customer reviews and, after fine-tuning, yield summaries with disfluencies and semantic mistakes. To address these problems, we utilize an efficient few-shot method based on adapters which, as we show, can easily store in-domain knowledge. Instead of fine-tuning the entire model, we add adapters and pre-train them in a task-specific way on a large corpus of unannotated customer reviews, using held-out reviews as pseudo summaries. Then, fine-tune the adapters on the small available human-annotated dataset. We show that this self-supervised adapter pre-training improves summary quality over standard fine-tuning by 2.0 and 1.3 ROUGE-L points on the Amazon and Yelp datasets, respectively. Finally, for summary personalization, we condition on aspect keyword queries, automatically created from generic datasets. In the same vein, we pre-train the adapters in a query-based manner on customer reviews and then fine-tune them on annotated datasets. This results in better-organized summary content reflected in improved coherence and fewer redundancies.