mlsanityData to 14 Oct 2026

Applying SoftTriple Loss for Supervised Language Model Fine Tuning

arXiv 2112.08462 ↗First posted 15 Dec 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce a new loss function TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models based on cross-entropy and SoftTriple loss. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samples in the training dataset, the higher gain -- thus, for small-sized dataset it is 0.78%, for medium-sized -- 0.86% for large -- 0.20% and for extra-large 0.04%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

14 of 34 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 786
  9. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  10. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  11. 2017No Fuss Distance Metric Learning using Proxiescited by 226
  12. 2019SoftTriple Loss: Deep Metric Learning Without Triplet Samplingcited by 163
2 more
  1. 2020Pre-training Polish Transformer-based Language Models at Scalecited by 21
  2. 2020Metric learning by Similarity Network for Deep Semi-Supervised Learningcited by 1

Built on by

Not cited yet.