mlsanityData to 14 Oct 2026

Learning with Latent Structures in Natural Language Processing: A Survey

arXiv 2201.00490 ↗First posted 3 Jan 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

While end-to-end learning with fully differentiable models has enabled tremendous success in natural language process (NLP) and machine learning, there have been significant recent interests in learning with latent discrete structures to incorporate better inductive biases for improved end-task performance and better interpretability. This paradigm, however, is not straightforwardly amenable to the mainstream gradient-based optimization methods. This work surveys three main families of methods to learn such models: surrogate gradients, continuous relaxation, and marginal likelihood maximization via sampling. We conclude with a review of applications of these methods and an inspection of the learned latent structure that they induce.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 62 of its 103 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

62 of 103 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  11. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  12. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
38 more
  1. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
  2. 2016Rationalizing Neural Predictionscited by 281
  3. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  4. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  5. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  6. 2019Sparse Sequence-to-Sequence Modelscited by 170
  7. 2014A* Samplingcited by 151
  8. 2017Encoding Sentences with Graph Convolutional Networks for Semantic Role Labelingcited by 137
  9. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  10. 2019Interpretable Neural Predictions with Differentiable Binary Variablescited by 126
  11. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 117
  12. 2016Recurrent Neural Network Grammarscited by 109
  13. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 104
  14. 2017Structured Attention Networkscited by 95
  15. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  16. 2018SparseMAP: Differentiable Sparse Structured Inferencecited by 62
  17. 2018Learning Neural Templates for Text Generationcited by 61
  18. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  19. 2018Differentiable Dynamic Programming for Structured Prediction and Attentioncited by 58
  20. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  21. 2017Learning to Compose Task-Specific Tree Structurescited by 48
  22. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  23. 2016Linguistic Input Features Improve Neural Machine Translationcited by 38
  24. 2017A Regularized Framework for Sparse and Structured Neural Attentioncited by 38
  25. 2016A Fast Unified Model for Parsing and Sentence Understandingcited by 34
  26. 2016Learning to Compose Words into Sentences with Reinforcement Learningcited by 30
  27. 2016Tree-to-Sequence Attentional Neural Machine Translationcited by 28
  28. 2016Language as a Latent Variable: Discrete Generative Models for Sentence Compressioncited by 28
  29. 2019Cooperative Learning of Disjoint Syntax and Semanticscited by 27
  30. 2015Training Deep Neural Networks via Direct Loss Minimizationcited by 25
  31. 2017Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMscited by 24
  32. 2017Learning Structured Text Representationscited by 24
  33. 2018StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsingcited by 22
  34. 2018Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencodercited by 19
  35. 2021R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modelingcited by 19
  36. 2020Infusing Finetuning with Semantic Dependenciescited by 18
  37. 2020LP-SparseMAP: Differentiable Relaxed Optimization for Sparse Structured Predictioncited by 17
  38. 2018On Tree-Based Neural Sentence Modelingcited by 16

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023On Learning Latent Models with Multi-Instance Weak Supervisioncited by 13
  2. 2023Efficient Beam Tree Recursioncited by 4