# Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models

arXiv: https://arxiv.org/abs/2203.02652 · HTML: https://mlsanity.com/paper/2203.02652/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 5 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2010.03546/) (2020)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Conversational Semantic Parsing](https://mlsanity.com/paper/2009.13655/) (2020)
- [Generating Synthetic Data for Task-Oriented Semantic Parsing with Hierarchical Representations](https://mlsanity.com/paper/2011.02050/) (2020)

## Built on by (most cited first, 4 of 4)

- [Federated Few-Shot Learning for Mobile NLP](https://mlsanity.com/paper/2212.05974/) (2022)
- [Multi-Task Domain Adaptation for Language Grounding with 3D Objects](https://mlsanity.com/paper/2407.02846/) (2024)
- [Memorization of Named Entities in Fine-tuned BERT Models](https://mlsanity.com/paper/2212.03749/) (2022)
- [The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing](https://mlsanity.com/paper/2305.15108/) (2023)

## Abstract (arXiv)

Semantic parsing is a key NLP task that maps natural language to structured meaning representations. As in many other NLP tasks, SOTA performance in semantic parsing is now attained by fine-tuning a large pretrained language model (PLM). While effective, this approach is inefficient in the presence of multiple downstream tasks, as a new set of values for all parameters of the PLM needs to be stored for each task separately. Recent work has explored methods for adapting PLMs to downstream tasks while keeping most (or all) of their parameters frozen. We examine two such promising techniques, prefix tuning and bias-term tuning, specifically on semantic parsing. We compare them against each other on two different semantic parsing datasets, and we also compare them against full and partial fine-tuning, both in few-shot and conventional data settings. While prefix tuning is shown to do poorly for semantic parsing tasks off the shelf, we modify it by adding special token embeddings, which results in very strong performance without compromising parameter savings.