# DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms

arXiv: https://arxiv.org/abs/2503.03644 · HTML: https://mlsanity.com/paper/2503.03644/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Builds on (most cited first, 18 of 18)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding](https://mlsanity.com/paper/2502.06020/) (2025)
- [OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?](https://mlsanity.com/paper/2412.01175/) (2024)
- [Neural Machine Translation: A Review of Methods, Resources, and Tools](https://mlsanity.com/paper/2012.15515/) (2020)
- [How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Language](https://mlsanity.com/paper/2204.11909/) (2022)
- [UMIE: Unified Multimodal Information Extraction with Instruction Tuning](https://mlsanity.com/paper/2401.03082/) (2024)
- [Efficient OCR for Building a Diverse Digital History](https://mlsanity.com/paper/2304.02737/) (2023)
- [Deciphering Undersegmented Ancient Scripts Using Phonetic Prior](https://mlsanity.com/paper/2010.11054/) (2020)
- [Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation](https://mlsanity.com/paper/2104.05964/) (2021)
- [HistRED: A Historical Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/2307.04285/) (2023)
- [LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP](https://mlsanity.com/paper/2408.04628/) (2024)
- [HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea](https://mlsanity.com/paper/2210.05112/) (2022)
- [Towards Boosting the Accuracy of Non-Latin Scene Text Recognition](https://mlsanity.com/paper/2201.03185/) (2022)

## Built on by (most cited first, 3 of 3)

- [ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering](https://mlsanity.com/paper/2509.16680/) (2025)
- [Enabling Stroke-Level Structural Analysis of Hieroglyphic Scripts without Language-Specific Priors](https://mlsanity.com/paper/2601.05508/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Dongba pictographic is the only pictographic script still in use in the world. Its pictorial ideographic features carry rich cultural and contextual information. However, due to the lack of relevant datasets, research on semantic understanding of Dongba hieroglyphs has progressed slowly. To this end, we constructed DongbaMIE - the first dataset focusing on multimodal information extraction of Dongba pictographs. The dataset consists of images of Dongba hieroglyphic characters and their corresponding semantic annotations in Chinese. It contains 23,530 sentence-level and 2,539 paragraph-level high-quality text-image pairs. The annotations cover four semantic dimensions: object, action, relation and attribute. Systematic evaluation of mainstream multimodal large language models shows that the models are difficult to perform information extraction of Dongba hieroglyphs efficiently under zero-shot and few-shot learning. Although supervised fine-tuning can improve the performance, accurate extraction of complex semantics is still a great challenge at present.