# OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?

arXiv: https://arxiv.org/abs/2412.01175 · HTML: https://mlsanity.com/paper/2412.01175/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 68 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 34)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [xGen-MM (BLIP-3): A Family of Open Large Multimodal Models](https://mlsanity.com/paper/2408.08872/) (2024)
- [Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception](https://mlsanity.com/paper/2403.02969/) (2024)
- [A-Bench: Are LMMs Masters at Evaluating AI-generated Images?](https://mlsanity.com/paper/2406.03070/) (2024)
- [GAIA: Rethinking Action Quality Assessment for AI-Generated Videos](https://mlsanity.com/paper/2406.06087/) (2024)
- [Deciphering Oracle Bone Language with Diffusion Models](https://mlsanity.com/paper/2406.00684/) (2024)

## Built on by (most cited first, 22 of 22)

- [PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving](https://mlsanity.com/paper/2504.10885/) (2025)
- [Just Noticeable Difference for Large Multimodal Models](https://mlsanity.com/paper/2507.00490/) (2025)
- [OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typography](https://mlsanity.com/paper/2506.21101/) (2025)
- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [Can Large Models Fool the Eye? A New Turing Test for Biological Animation](https://mlsanity.com/paper/2508.06072/) (2025)
- [Mitigating Long-tail Distribution in Oracle Bone Inscriptions: Dataset, Model, and Benchmark](https://mlsanity.com/paper/2504.09555/) (2025)
- [Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters](https://mlsanity.com/paper/2605.11960/) (2026)
- [PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters](https://mlsanity.com/paper/2509.05773/) (2025)
- [DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms](https://mlsanity.com/paper/2503.03644/) (2025)
- [Interpretable Oracle Bone Script Decipherment through Radical and Pictographic Analysis with LVLMs](https://mlsanity.com/paper/2508.10113/) (2025)
- [OracleAgent: A Multimodal Reasoning Agent for Oracle Bone Script Research](https://mlsanity.com/paper/2510.26114/) (2025)
- [MACEval: A Multi-Agent Continual Evaluation Network for Large Models](https://mlsanity.com/paper/2511.09139/) (2025)
- [Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models](https://mlsanity.com/paper/2511.10691/) (2025)
- [PriceSeer: Evaluating Large Language Models in Real-Time Stock Prediction](https://mlsanity.com/paper/2601.06088/) (2025)
- [Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation](https://mlsanity.com/paper/2604.06711/) (2026)
- [Enhancing Multimodal Large Language Models for Ancient Chinese Character Evolution Analysis via Glyph-Driven Fine-Tuning](https://mlsanity.com/paper/2604.11299/) (2026)
- [Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions](https://mlsanity.com/paper/2507.04377/) (2025)
- [Towards Computational Chinese Paleography](https://mlsanity.com/paper/2601.06753/) (2026)
- [OracleAnalyser: Analysing Implicit Semantics of Oracle Bone Scripts through MLLMs with Post-training](https://mlsanity.com/paper/2606.25906/) (2026)
- [Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understanding](https://mlsanity.com/paper/2606.31169/) (2026)
- [JieZi: A Large-Scale Expert-Audited Dataset and Benchmark for Ancient Chinese Character Exegesis](https://mlsanity.com/paper/2608.11741/) (2026)
- [Ancient-Bench: A Comprehensive Multi-millennial, Multi-medium, and Multi-script Benchmark for Ancient Chinese Artifact Text Recognition](https://mlsanity.com/paper/2608.27169/) (2026)

## Abstract (arXiv)

We introduce OBI-Bench, a holistic benchmark crafted to systematically evaluate large multi-modal models (LMMs) on whole-process oracle bone inscriptions (OBI) processing tasks demanding expert-level domain knowledge and deliberate cognition. OBI-Bench includes 5,523 meticulously collected diverse-sourced images, covering five key domain problems: recognition, rejoining, classification, retrieval, and deciphering. These images span centuries of archaeological findings and years of research by front-line scholars, comprising multi-stage font appearances from excavation to synthesis, such as original oracle bone, inked rubbings, oracle bone fragments, cropped single characters, and handprinted characters. Unlike existing benchmarks, OBI-Bench focuses on advanced visual perception and reasoning with OBI-specific knowledge, challenging LMMs to perform tasks akin to those faced by experts. The evaluation of 6 proprietary LMMs as well as 17 open-source LMMs highlights the substantial challenges and demands posed by OBI-Bench. Even the latest versions of GPT-4o, Gemini 1.5 Pro, and Qwen-VL-Max are still far from public-level humans in some fine-grained perception tasks. However, they perform at a level comparable to untrained humans in deciphering tasks, indicating remarkable capabilities in offering new interpretative perspectives and generating creative guesses. We hope OBI-Bench can facilitate the community to develop domain-specific multi-modal foundation models towards ancient language research and delve deeper to discover and enhance these untapped potentials of LMMs.