# Multimodal Approach for Metadata Extraction from German Scientific Publications

arXiv: https://arxiv.org/abs/2111.05736 · HTML: https://mlsanity.com/paper/2111.05736/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation](https://mlsanity.com/paper/1807.03121/) (2018)
- [Vision-Based Layout Detection from Scientific Literature using Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/2010.11727/) (2020)
- [MexPub: Deep Transfer Learning for Metadata Extraction from German Publications](https://mlsanity.com/paper/2106.07359/) (2021)
- [BiblioDAP: The 1st Workshop on Bibliographic Data Analysis and Processing](https://mlsanity.com/paper/2106.12320/) (2021)

## Abstract (arXiv)

Nowadays, metadata information is often given by the authors themselves upon submission. However, a significant part of already existing research papers have missing or incomplete metadata information. German scientific papers come in a large variety of layouts which makes the extraction of metadata a non-trivial task that requires a precise way to classify the metadata extracted from the documents. In this paper, we propose a multimodal deep learning approach for metadata extraction from scientific papers in the German language. We consider multiple types of input data by combining natural language processing and image vision processing. This model aims to increase the overall accuracy of metadata extraction compared to other state-of-the-art approaches. It enables the utilization of both spatial and contextual features in order to achieve a more reliable extraction. Our model for this approach was trained on a dataset consisting of around 8800 documents and is able to obtain an overall F1-score of 0.923.