mlsanityData to 7 Oct 2026

ContraCLM: Contrastive Learning For Causal Language Model

arXiv 2210.01185 ↗First posted 3 Oct 2022cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Despite exciting progress in causal language models, the expressiveness of the representations is largely limited due to poor discrimination ability. To remedy this issue, we present ContraCLM, a novel contrastive learning framework at both token-level and sequence-level. We assess ContraCLM on a variety of downstream tasks. We show that ContraCLM enhances discrimination of the representations and bridges the gap with the encoder-only models, which makes causal language models better suited for tasks beyond language generation. Specifically, we attain 44% relative improvement on the Semantic Textual Similarity tasks and 34% on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of the representations, ContraCLM also boosts the source code generation capability with 9% relative improvement on execution accuracy on the HumanEval benchmark.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

47 of 64 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
35 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  12. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  13. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  14. 2020Understanding the Behaviour of Contrastive Losscited by 397
  15. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  16. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  17. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  18. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  19. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  20. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  21. 2022A Contrastive Framework for Neural Text Generationcited by 224
  22. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  23. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  24. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  25. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  26. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  27. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  28. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  29. 2021Supporting Clustering with Contrastive Learningcited by 90
  30. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  31. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  32. 2020Contrastive Learning with Adversarial Perturbations for Conditional Text Generationcited by 56
  33. 2022Fault-Aware Neural Code Rankerscited by 33
  34. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  35. 2022CoNT: Contrastive Neural Text Generationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  2. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  3. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  4. 2024Code Representation Learning At Scalecited by 26
  5. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  6. 2025CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generationcited by 9
  7. 2024Reducing Distraction in Long-Context Language Models by Focused Learningcited by 8
  8. 2024Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Modelscited by 6
  9. 2025On the Predictive Power of Representation Dispersion in Language Modelscited by 6
  10. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  11. 2024BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of Englishcited by 5
  12. 2024CodeFort: Robust Training for Code Generation Modelscited by 4
6 more
  1. 2024SetCSE: Set Operations using Contrastive Learning of Sentence Embeddingscited by 2
  2. 2024Building Decision Making Models Through Language Model Regimecited by 2
  3. 2024Efficient Training of Language Models with Compact and Consistent Next Token Distributionscited by 0
  4. 2024BioNCERE: Non-Contrastive Enhancement For Relation Extraction In Biomedical Textscited by 0
  5. 2025CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Cloudscited by 0
  6. 2026An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Codecited by 0