# Automated pipeline for herbarium label digitization

arXiv: https://arxiv.org/abs/2608.28676 · HTML: https://mlsanity.com/paper/2608.28676/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Builds on (most cited first, 11 of 11)

- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition](https://mlsanity.com/paper/2203.10209/) (2022)
- [HTR-VT: Handwritten Text Recognition with Vision Transformer](https://mlsanity.com/paper/2409.08573/) (2024)
- [Text Detection Forgot About Document OCR](https://mlsanity.com/paper/2210.07903/) (2022)
- [SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds from RGB Images for 2D Classification](https://mlsanity.com/paper/2506.18683/) (2025)
- [A Unified Multilingual Handwriting Recognition System using multigrams sub-lexical units](https://mlsanity.com/paper/1808.09183/) (2018)
- [Application of Computer Vision and Machine Learning for Digitized Herbarium Specimens: A Systematic Literature Review](https://mlsanity.com/paper/2104.08732/) (2021)
- [Hespi: A pipeline for automatically detecting information from hebarium specimen sheets](https://mlsanity.com/paper/2410.08740/) (2024)
- [AT-ViT: Area-Targeted Multi-View Vision Transformer with Cross-Attention and Multi-Scale Patching for Plant Trait Recognition in Herbarium Images](https://mlsanity.com/paper/2608.21067/) (2026)

## Abstract (arXiv)

Digitized herbarium collections, now comprising over 100 million freely accessible specimen images, have become a critical resource for addressing fundamental questions in ecology and evolutionary biology. Yet the rich metadata encoded in herbarium labels (collector identities, geographic localities, collection dates, and ecological observations) remains largely inaccessible at scale, constraining both biodiversity informatics and the construction of specimen-specific image-text corpora for multimodal AI. We present HERBIOME, a modular end-to-end pipeline for automated herbarium label digitization, integrating YOLOv8-based component detection, CRAFT Hezar word-level text localization, fine-tuned TrOCR for recognition of mixed handwritten and printed text, and GPT-4o Mini for semantic metadata structuring into standardized fields. TrOCR was trained on a multi-source dataset combining general transcription corpora (CREMMA-AN, PictoCatalogs) with herbarium-specific data (R'eColNat), achieving a Character Error Rate of 4.05-4.10%. End-to-end evaluation on 450 French herbarium specimens, using a dual-metric framework of Maximum Window Similarity (MWS: 0.614-0.618) and Semantic Metadata Accuracy (SMA: 0.440-0.445), reveals that hybrid training strategies improve semantic fidelity while random sampling maximizes surface similarity, with taxonomic fields remaining the principal bottleneck. By automating the extraction of structured metadata from complex, heterogeneous labels, HERBIOME reduces transcription burden, enables the construction of paired image-text datasets that faithfully capture specimen individuality, which is a prerequisite for next-generation multimodal biodiversity AI systems.