# Better than BERT but Worse than Baseline

arXiv: https://arxiv.org/abs/2105.05915 · HTML: https://mlsanity.com/paper/2105.05915/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Coreference Resolution as Query-based Span Prediction](https://mlsanity.com/paper/1911.01746/) (2019)
- [What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation](https://mlsanity.com/paper/2010.14678/) (2020)
- [Biomedical named entity recognition using BERT in the machine reading comprehension framework](https://mlsanity.com/paper/2009.01560/) (2020)
- [Read, Retrospect, Select: An MRC Framework to Short Text Entity Linking](https://mlsanity.com/paper/2101.02394/) (2021)
- [Probing and Fine-tuning Reading Comprehension Models for Few-shot Event Extraction](https://mlsanity.com/paper/2010.11325/) (2020)

## Abstract (arXiv)

This paper compares BERT-SQuAD and Ab3P on the Abbreviation Definition Identification (ADI) task. ADI inputs a text and outputs short forms (abbreviations/acronyms) and long forms (expansions). BERT with reranking improves over BERT without reranking but fails to reach the Ab3P rule-based baseline. What is BERT missing? Reranking introduces two new features: charmatch and freq. The first feature identifies opportunities to take advantage of character constraints in acronyms and the second feature identifies opportunities to take advantage of frequency constraints across documents.