# TocBERT: Medical Document Structure Extraction Using Bidirectional Transformers

arXiv: https://arxiv.org/abs/2406.19526 · HTML: https://mlsanity.com/paper/2406.19526/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [Text Segmentation by Cross Segment Attention](https://mlsanity.com/paper/2004.14535/) (2020)
- [Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation](https://mlsanity.com/paper/2001.00891/) (2020)
- [Unsupervised Topic Segmentation of Meetings with BERT Embeddings](https://mlsanity.com/paper/2106.12978/) (2021)
- [Text Segmentation based on Semantic Word Embeddings](https://mlsanity.com/paper/1503.05543/) (2015)
- [Document Summarization with Text Segmentation](https://mlsanity.com/paper/2301.08817/) (2023)
- [Anatomy of Neural Language Models](https://mlsanity.com/paper/2401.03797/) (2024)

## Built on by (most cited first, 3 of 3)

- [Bridging the Domain Divide: Supervised vs. Zero-Shot Clinical Section Segmentation from MIMIC-III to Obstetrics](https://mlsanity.com/paper/2602.17513/) (2026)
- [Clinical Document Metadata Extraction: A Scoping Review](https://mlsanity.com/paper/2601.09730/) (2025)
- [Towards Multidisciplinary Summarization of Hospital Stays: Efficient Sentence-Level Clinical Provenance Categorization](https://mlsanity.com/paper/2606.02487/) (2026)

## Abstract (arXiv)

Text segmentation holds paramount importance in the field of Natural Language Processing (NLP). It plays an important role in several NLP downstream tasks like information retrieval and document summarization. In this work, we propose a new solution, namely TocBERT, for segmenting texts using bidirectional transformers. TocBERT represents a supervised solution trained on the detection of titles and sub-titles from their semantic representations. This task was formulated as a named entity recognition (NER) problem. The solution has been applied on a medical text segmentation use-case where the Bio-ClinicalBERT model is fine-tuned to segment discharge summaries of the MIMIC-III dataset. The performance of TocBERT has been evaluated on a human-labeled ground truth corpus of 250 notes. It achieved an F1-score of 84.6% when evaluated on a linear text segmentation problem and 72.8% on a hierarchical text segmentation problem. It outperformed a carefully designed rule-based solution, particularly in distinguishing titles from subtitles.