# MetricBERT: Text Representation Learning via Self-Supervised Triplet Training

arXiv: https://arxiv.org/abs/2208.06610 · HTML: https://mlsanity.com/paper/2208.06610/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 13 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Item2Vec: Neural Item Embedding for Collaborative Filtering](https://mlsanity.com/paper/1603.04259/) (2016)
- [Class Rectification Hard Mining for Imbalanced Deep Learning](https://mlsanity.com/paper/1712.03162/) (2017)
- [A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives](https://mlsanity.com/paper/2102.12982/) (2021)
- [Simple Applications of BERT for Ad Hoc Document Retrieval](https://mlsanity.com/paper/1903.10972/) (2019)
- [Explainable Recommendations via Attentive Multi-Persona Collaborative Filtering](https://mlsanity.com/paper/2010.07042/) (2020)
- [Caption Enriched Samples for Improving Hateful Memes Detection](https://mlsanity.com/paper/2109.10649/) (2021)
- [Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference](https://mlsanity.com/paper/2106.01186/) (2021)
- [Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding](https://mlsanity.com/paper/1908.05161/) (2019)
- [Bayesian Neural Word Embedding](https://mlsanity.com/paper/1603.06571/) (2016)
- [Neural Attentive Multiview Machines](https://mlsanity.com/paper/2002.07696/) (2020)
- [Bayesian Hierarchical Words Representation Learning](https://mlsanity.com/paper/2004.07126/) (2020)
- [Attentive Item2Vec: Neural Attentive User Representations](https://mlsanity.com/paper/2002.06205/) (2020)
- [RecoBERT: A Catalog Language Model for Text-Based Recommendations](https://mlsanity.com/paper/2009.13292/) (2020)
- [Cold Item Integration in Deep Hybrid Recommenders via Tunable Stochastic Gates](https://mlsanity.com/paper/2112.07615/) (2021)

## Built on by (most cited first, 3 of 3)

- [Efficient Discovery and Effective Evaluation of Visual Perceptual Similarity: A Benchmark and Beyond](https://mlsanity.com/paper/2308.14753/) (2023)
- [ESCL: Equivariant Self-Contrastive Learning for Sentence Representations](https://mlsanity.com/paper/2303.05143/) (2023)
- [FastDoc: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomy](https://mlsanity.com/paper/2306.06190/) (2023)

## Abstract (arXiv)

We present MetricBERT, a BERT-based model that learns to embed text under a well-defined similarity metric while simultaneously adhering to the ``traditional'' masked-language task. We focus on downstream tasks of learning similarities for recommendations where we show that MetricBERT outperforms state-of-the-art alternatives, sometimes by a substantial margin. We conduct extensive evaluations of our method and its different variants, showing that our training objective is highly beneficial over a traditional contrastive loss, a standard cosine similarity objective, and six other baselines. As an additional contribution, we publish a dataset of video games descriptions along with a test set of similarity annotations crafted by a domain expert.