# BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching

arXiv: https://arxiv.org/abs/2111.02188 · HTML: https://mlsanity.com/paper/2111.02188/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 3 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Bilateral Multi-Perspective Matching for Natural Language Sentences](https://mlsanity.com/paper/1702.03814/) (2017)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [ParsBERT: Transformer-based Model for Persian Language Understanding](https://mlsanity.com/paper/2005.12515/) (2020)
- [Convolutional Neural Network Architectures for Matching Natural Language Sentences](https://mlsanity.com/paper/1503.03244/) (2015)
- [Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data](https://mlsanity.com/paper/2009.09139/) (2020)
- [Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding](https://mlsanity.com/paper/1904.09482/) (2019)
- [ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs](https://mlsanity.com/paper/1512.05193/) (2015)
- [Simple and Effective Text Matching with Richer Alignment Features](https://mlsanity.com/paper/1908.00300/) (2019)
- [Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information](https://mlsanity.com/paper/1805.11360/) (2018)
- [Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering](https://mlsanity.com/paper/1806.04330/) (2018)
- [FarsTail: A Persian Natural Language Inference Dataset](https://mlsanity.com/paper/2009.08820/) (2020)
- [Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference](https://mlsanity.com/paper/1801.00102/) (2017)
- [Neural Paraphrase Identification of Questions with Noisy Pretraining](https://mlsanity.com/paper/1704.04565/) (2017)
- [Sentence Embeddings in NLI with Iterative Refinement Encoders](https://mlsanity.com/paper/1808.08762/) (2018)
- [Explicit Contextual Semantics for Text Comprehension](https://mlsanity.com/paper/1809.02794/) (2018)

## Built on by (most cited first, 3 of 3)

- [Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding](https://mlsanity.com/paper/2204.02922/) (2022)
- [A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on n-dimensional Classes](https://mlsanity.com/paper/2404.06104/) (2024)
- [Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecture](https://mlsanity.com/paper/2410.04454/) (2024)

## Abstract (arXiv)

This paper presents a deep neural architecture, for Natural Language Sentence Matching (NLSM) by adding a deep recursive encoder to BERT so called BERT with Deep Recursive Encoder (BERT-DRE). Our analysis of model behavior shows that BERT still does not capture the full complexity of text, so a deep recursive encoder is applied on top of BERT. Three Bi-LSTM layers with residual connection are used to design a recursive encoder and an attention module is used on top of this encoder. To obtain the final vector, a pooling layer consisting of average and maximum pooling is used. We experiment our model on four benchmarks, SNLI, FarsTail, MultiNLI, SciTail, and a novel Persian religious questions dataset. This paper focuses on improving the BERT results in the NLSM task. In this regard, comparisons between BERT-DRE and BERT are conducted, and it is shown that in all cases, BERT-DRE outperforms BERT. The BERT algorithm on the religious dataset achieved an accuracy of 89.70%, and BERT-DRE architectures improved to 90.29% using the same dataset.