mlsanityData to 7 Oct 2026

Can Contrastive Learning Refine Embeddings

arXiv 2404.08701 ↗First posted 11 Apr 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent advancements in contrastive learning have revolutionized self-supervised representation learning and achieved state-of-the-art performance on benchmark tasks. While most existing methods focus on applying contrastive learning to input data modalities such as images, natural language sentences, or networks, they overlook the potential of utilizing outputs from previously trained encoders. In this paper, we introduce SIMSKIP, a novel contrastive learning framework that specifically refines input embeddings for downstream tasks. Unlike traditional unsupervised learning approaches, SIMSKIP takes advantage of the output embeddings of encoder models as its input. Through theoretical analysis, we provide evidence that applying SIMSKIP does not result in larger upper bounds on downstream task errors than those of the original embeddings, which serve as SIMSKIP's input. Experimental results on various open datasets demonstrate that the embeddings produced by SIMSKIP improve performance on downstream tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

32 of 49 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2017Inductive Representation Learning on Large Graphscited by 6,284
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2020Supervised Contrastive Learningcited by 2,803
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  12. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
20 more
  1. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  2. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 929
  3. 2016Complex Embeddings for Simple Link Predictioncited by 897
  4. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  5. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2020BoxE: A Box Embedding Model for Knowledge Base Completioncited by 105
  9. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  10. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  11. 2019Transferable Contrastive Network for Generalized Zero-Shot Learningcited by 64
  12. 2022A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Predictioncited by 48
  13. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 47
  14. 2020FedE: Embedding Knowledge Graphs in Federated Settingcited by 46
  15. 2020KompaRe: A Knowledge Graph Comparative Reasoning Systemcited by 20
  16. 2024Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphscited by 20
  17. 2022Defending Graph Convolutional Networks against Dynamic Graph Perturbations via Bayesian Self-supervisioncited by 16
  18. 2023Understand Legal Documents with Contextualized Large Language Modelscited by 5
  19. 2023Conversational Question Answering with Reformulations over Knowledge Graphcited by 4
  20. 2021KGE-CL: Contrastive Learning of Tensor Decomposition Based Knowledge Graph Embeddingscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 4.

5 papers.

  1. 2024PyG-SSL: A Graph Self-Supervised Learning Toolkitcited by 14
  2. 2026Multi-hop Reasoning and Retrieval in Embedding Space: Leveraging Large Language Models with Knowledgecited by 2
  3. 2026Neural-Symbolic Logic Query Answering in Non-Euclidean Spacecited by 2
  4. 2026Prompt-tuning with Attribute Guidance for Low-resource Entity Matchingcited by 2
  5. 2026Outage Detection in Self-Healing Smart Grids Using Reinforcement Learning with Spectral Graph Neural Networkscited by 0