mlsanityData to 7 Oct 2026

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

arXiv 2311.00408 ↗First posted 1 Nov 2023cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of few-shot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent's effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

40 of 55 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
28 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  6. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  7. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  8. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  9. 2021True Few-Shot Learning with Language Modelscited by 343
  10. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  11. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  12. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  13. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  14. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  15. 2022Efficient Few-Shot Learning Without Promptscited by 135
  16. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  17. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  18. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  19. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  20. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 44
  21. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  22. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  23. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  24. 2018Evaluation of sentence embeddings in downstream and linguistic probing taskscited by 28
  25. 2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 22
  26. 2021Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understandingcited by 12
  27. 2022Masked Autoencoders As The Unified Learners For Pre-Trained Sentence Representationcited by 5
  28. 2022Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrievalcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  2. 2024REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Modelscited by 1
  3. 2025Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generationcited by 1