mlsanityData to 7 Oct 2026

Adversarial Contrastive Pre-training for Protein Sequences

arXiv 2102.00466 ↗First posted 31 Jan 2021cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent developments in Natural Language Processing (NLP) demonstrate that large-scale, self-supervised pre-training can be extremely beneficial for downstream tasks. These ideas have been adapted to other domains, including the analysis of the amino acid sequences of proteins. However, to date most attempts on protein sequences rely on direct masked language model style pre-training. In this work, we design a new, adversarial pre-training method for proteins, extending and specializing similar advances in NLP. We show compelling results in comparison to traditional MLM pre-training, though further development is needed to ensure the gains are worth the significant computational cost.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 31 Jan 2021.

Builds on

9 of 15 references.

References · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2019Evaluating Protein Transfer Learning with TAPEcited by 229
  7. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 69
  8. 2018Adversarial Contrastive Estimationcited by 23
  9. 2020Variance-reduced Language Pretraining via a Mask Proposal Networkcited by 4

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 2.

8 papers.

  1. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
  2. 2023Structure-informed Language Models Are Protein Designerscited by 68
  3. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 59
  4. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  5. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  6. 2021Structure Inducing Pre-Trainingcited by 10
  7. 2022A Survey on Protein Representation Learning: Retrospect and Prospectcited by 10
  8. 2022Adversarial Contrastive Learning by Permuting Cluster Assignmentscited by 2