# Structured Prediction in NLP -- A survey

arXiv: https://arxiv.org/abs/2110.02057 · HTML: https://mlsanity.com/paper/2110.02057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Structured Attention Networks](https://mlsanity.com/paper/1702.00887/) (2017)
- [Structured Prediction Energy Networks](https://mlsanity.com/paper/1511.06350/) (2015)
- [End-to-End Learning for Structured Prediction Energy Networks](https://mlsanity.com/paper/1703.05667/) (2017)
- [TaxiNLI: Taking a Ride up the NLU Hill](https://mlsanity.com/paper/2009.14505/) (2020)
- [Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT](https://mlsanity.com/paper/1908.04943/) (2019)
- [Dialogue Act Sequence Labeling using Hierarchical encoder with CRF](https://mlsanity.com/paper/1709.04250/) (2017)
- [Learning Approximate Inference Networks for Structured Prediction](https://mlsanity.com/paper/1803.03376/) (2018)
- [Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs](https://mlsanity.com/paper/1703.04363/) (2017)
- [Deep Structured Prediction with Nonlinear Output Transformations](https://mlsanity.com/paper/1811.00539/) (2018)
- [Graph Structured Prediction Energy Networks](https://mlsanity.com/paper/1910.14670/) (2019)
- [Cascaded Text Generation with Markov Transformers](https://mlsanity.com/paper/2006.01112/) (2020)
- [Improving Tree-LSTM with Tree Attention](https://mlsanity.com/paper/1901.00066/) (2019)
- [Autoregressive Text Generation Beyond Feedback Loops](https://mlsanity.com/paper/1908.11658/) (2019)
- [Syntax-driven Iterative Expansion Language Models for Controllable Text Generation](https://mlsanity.com/paper/2004.02211/) (2020)
- [Search-Guided, Lightly-supervised Training of Structured Prediction Energy Networks](https://mlsanity.com/paper/1812.09603/) (2018)

## Abstract (arXiv)

Over the last several years, the field of Structured prediction in NLP has had seen huge advancements with sophisticated probabilistic graphical models, energy-based networks, and its combination with deep learning-based approaches. This survey provides a brief of major techniques in structured prediction and its applications in the NLP domains like parsing, sequence labeling, text generation, and sequence to sequence tasks. We also deep-dived into energy-based and attention-based techniques in structured prediction, identified some relevant open issues and gaps in the current state-of-the-art research, and have come up with some detailed ideas for future research in these fields.