mlsanityData to 7 Oct 2026

OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?

arXiv 2412.01175 ↗First posted 2 Dec 2024cs.CV cs.AIcited by 22

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce OBI-Bench, a holistic benchmark crafted to systematically evaluate large multi-modal models (LMMs) on whole-process oracle bone inscriptions (OBI) processing tasks demanding expert-level domain knowledge and deliberate cognition. OBI-Bench includes 5,523 meticulously collected diverse-sourced images, covering five key domain problems: recognition, rejoining, classification, retrieval, and deciphering. These images span centuries of archaeological findings and years of research by front-line scholars, comprising multi-stage font appearances from excavation to synthesis, such as original oracle bone, inked rubbings, oracle bone fragments, cropped single characters, and handprinted characters. Unlike existing benchmarks, OBI-Bench focuses on advanced visual perception and reasoning with OBI-specific knowledge, challenging LMMs to perform tasks akin to those faced by experts. The evaluation of 6 proprietary LMMs as well as 17 open-source LMMs highlights the substantial challenges and demands posed by OBI-Bench. Even the latest versions of GPT-4o, Gemini 1.5 Pro, and Qwen-VL-Max are still far from public-level humans in some fine-grained perception tasks. However, they perform at a level comparable to untrained humans in deciphering tasks, indicating remarkable capabilities in offering new interpretative perspectives and generating creative guesses. We hope OBI-Bench can facilitate the community to develop domain-specific multi-modal foundation models towards ancient language research and delve deeper to discover and enhance these untapped potentials of LMMs.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 68 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

34 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024What matters when building vision-language models?3 May
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
20 more
  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  5. 2024InternLM2 Technical Reportcited by 391
  6. 2017NIMA: Neural Image Assessmentcited by 387
  7. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  9. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  10. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  11. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  12. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  13. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  14. 2024A-Bench: Are LMMs Masters at Evaluating AI-generated Images?cited by 39
  15. 2024GAIA: Rethinking Action Quality Assessment for AI-Generated Videoscited by 30
  16. 2024Deciphering Oracle Bone Language with Diffusion Modelscited by 18
  17. 2022Unsupervised Structure-Texture Separation Network for Oracle Character Recognitioncited by 15
  18. 2024An open dataset for the evolution of oracle bone characters: EVOBCcited by 15
  19. 2023Toward Zero-shot Character Recognition: A Gold Standard Dataset with Radical-level Annotationscited by 10
  20. 2021Recognition of Oracle Bone Inscriptions by using Two Deep Learning Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solvingcited by 15
  2. 2025Just Noticeable Difference for Large Multimodal Modelscited by 9
  3. 2025OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typographycited by 8
  4. 2025Can Large Models Fool the Eye? A New Turing Test for Biological Animationcited by 6
  5. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  6. 2025Mitigating Long-tail Distribution in Oracle Bone Inscriptions: Dataset, Model, and Benchmarkcited by 5
  7. 2026Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characterscited by 5
  8. 2025PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characterscited by 4
  9. 2025DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictogramscited by 3
  10. 2025Interpretable Oracle Bone Script Decipherment through Radical and Pictographic Analysis with LVLMscited by 3
  11. 2025OracleAgent: A Multimodal Reasoning Agent for Oracle Bone Script Researchcited by 3
  12. 2025MACEval: A Multi-Agent Continual Evaluation Network for Large Modelscited by 2
10 more
  1. 2025Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Modelscited by 2
  2. 2025PriceSeer: Evaluating Large Language Models in Real-Time Stock Predictioncited by 1
  3. 2026Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentationcited by 1
  4. 2026Enhancing Multimodal Large Language Models for Ancient Chinese Character Evolution Analysis via Glyph-Driven Fine-Tuningcited by 1
  5. 2025Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptionscited by 0
  6. 2026Towards Computational Chinese Paleographycited by 0
  7. 2026OracleAnalyser: Analysing Implicit Semantics of Oracle Bone Scripts through MLLMs with Post-trainingcited by 0
  8. 2026Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understandingcited by 0
  9. 2026JieZi: A Large-Scale Expert-Audited Dataset and Benchmark for Ancient Chinese Character Exegesiscited by 0
  10. 2026Ancient-Bench: A Comprehensive Multi-millennial, Multi-medium, and Multi-script Benchmark for Ancient Chinese Artifact Text Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OracleSage: Towards Unified Visual-Linguistic Understanding of Oracle Bone Scripts through Cross-Modal Knowledge Fusion
  2. 2026AlphaOracle: Oracle bone script decipherment via human-workflow-inspired deep learning
  3. 2023Diff-Oracle: Deciphering Oracle Bone Scripts with Controllable Diffusion Model
  4. 2026The Devil is in the Details -- From OCR for Old Church Slavonic to Purely Visual Stemma Reconstruction
  5. 2026Zero-shot large vision-language model prompting for automated bone identification in paleoradiology x-ray archives
  6. 2026AmalthAI: An Open-Source Computer Vision Platform for Cultural Heritage
  7. 2025Provenance Analysis of Archaeological Artifacts via Multimodal RAG Systems
  8. 2025LadderMoE: Ladder-Side Mixture of Experts Adapters for Bronze Inscription Recognition