# Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT

arXiv: https://arxiv.org/abs/1908.04943 · HTML: https://mlsanity.com/paper/1908.04943/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2019.

## Built on by (most cited first, 19 of 19)

- [Self-Attention Networks Can Process Bounded Hierarchical Languages](https://mlsanity.com/paper/2105.11115/) (2021)
- [Rethinking Explainability as a Dialogue: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01875/) (2022)
- [The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders](https://mlsanity.com/paper/2109.06939/) (2021)
- [Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning](https://mlsanity.com/paper/2105.10195/) (2021)
- [Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL](https://mlsanity.com/paper/2405.06410/) (2024)
- [Evaluation of Unsupervised Entity and Event Salience Estimation](https://mlsanity.com/paper/2104.06924/) (2021)
- [Learning compositional structures for semantic graph parsing](https://mlsanity.com/paper/2106.04398/) (2021)
- [Transition-based Semantic Role Labeling with Pointer Networks](https://mlsanity.com/paper/2205.10023/) (2022)
- [Weakly Supervised Concept Map Generation through Task-Guided Graph Translation](https://mlsanity.com/paper/2110.15720/) (2021)
- [Predicting Music Hierarchies with a Graph-Based Neural Decoder](https://mlsanity.com/paper/2306.16955/) (2023)
- [Modeling Label Correlations for Second-Order Semantic Dependency Parsing with Mean-Field Inference](https://mlsanity.com/paper/2204.03619/) (2022)
- [Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing](https://mlsanity.com/paper/2302.02275/) (2023)
- [Structured Prediction in NLP -- A survey](https://mlsanity.com/paper/2110.02057/) (2021)
- [A Higher-Order Semantic Dependency Parser](https://mlsanity.com/paper/2201.11312/) (2022)
- [Schema-Free Dependency Parsing via Sequence Generation](https://mlsanity.com/paper/2201.12407/) (2022)
- [Semantic Tagging with LSTM-CRF](https://mlsanity.com/paper/2301.12206/) (2023)
- [Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals](https://mlsanity.com/paper/2306.04187/) (2023)
- [Auxiliary Tasks to Boost Biaffine Semantic Dependency Parsing](https://mlsanity.com/paper/2402.07682/) (2024)
- [Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models](https://mlsanity.com/paper/2508.17153/) (2025)

## Abstract (arXiv)

This paper presents new state-of-the-art models for three tasks, part-of-speech tagging, syntactic parsing, and semantic parsing, using the cutting-edge contextualized embedding framework known as BERT. For each task, we first replicate and simplify the current state-of-the-art approach to enhance its model efficiency. We then evaluate our simplified approaches on those three tasks using token embeddings generated by BERT. 12 datasets in both English and Chinese are used for our experiments. The BERT models outperform the previously best-performing models by 2.5% on average (7.5% for the most significant case). Moreover, an in-depth analysis on the impact of BERT embeddings is provided using self-attention, which helps understanding in this rich yet representation. All models and source codes are available in public so that researchers can improve upon and utilize them to establish strong baselines for the next decade.