# Protein sequence-to-structure learning: Is this the end(-to-end revolution)?

arXiv: https://arxiv.org/abs/2105.07407 · HTML: https://mlsanity.com/paper/2105.07407/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 192 references that are in the map.
- First posted on arXiv 16 May 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [A Convolutional Encoder Model for Neural Machine Translation](https://mlsanity.com/paper/1611.02344/) (2016)
- [N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials](https://mlsanity.com/paper/1803.01588/) (2018)
- [ProteinNet: a standardized data set for machine learning of protein structure](https://mlsanity.com/paper/1902.00249/) (2019)
- [Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model](https://mlsanity.com/paper/1609.00680/) (2016)
- [Protein model quality assessment using rotation-equivariant, hierarchical neural networks](https://mlsanity.com/paper/2011.13557/) (2020)
- [Dilated Convolutions for Modeling Long-Distance Genomic Dependencies](https://mlsanity.com/paper/1710.01278/) (2017)
- [Spherical convolutions on molecular graphs for protein model quality assessment](https://mlsanity.com/paper/2011.07980/) (2020)
- [Accurate Protein Structure Prediction by Embeddings and Deep Learning Representations](https://mlsanity.com/paper/1911.05531/) (2019)
- [Protein Contact Prediction by Integrating Joint Evolutionary Coupling Analysis and Supervised Learning](https://mlsanity.com/paper/1312.2988/) (2013)
- [Hierarchical, rotation-equivariant neural networks to select structural models of protein complexes](https://mlsanity.com/paper/2006.09275/) (2020)

## Built on by (most cited first, 3 of 3)

- [Equivariant Graph Neural Networks for 3D Macromolecular Structure](https://mlsanity.com/paper/2106.03843/) (2021)
- [6DCNN with roto-translational convolution filters for volumetric data processing](https://mlsanity.com/paper/2107.12078/) (2021)
- [Protein Structure Prediction: From Evolutionary Constraints to Generative Modeling](https://mlsanity.com/paper/2608.16094/) (2026)

## Abstract (arXiv)

The potential of deep learning has been recognized in the protein structure prediction community for some time, and became indisputable after CASP13. In CASP14, deep learning has boosted the field to unanticipated levels reaching near-experimental accuracy. This success comes from advances transferred from other machine learning areas, as well as methods specifically designed to deal with protein sequences and structures, and their abstractions. Novel emerging approaches include (i) geometric learning, i.e. learning on representations such as graphs, 3D Voronoi tessellations, and point clouds; (ii) pre-trained protein language models leveraging attention; (iii) equivariant architectures preserving the symmetry of 3D space; (iv) use of large meta-genome databases; (v) combinations of protein representations; (vi) and finally truly end-to-end architectures, i.e. differentiable models starting from a sequence and returning a 3D structure. Here, we provide an overview and our opinion of the novel deep learning approaches developed in the last two years and widely used in CASP14.