Self-Verification is All You Need To Pass The Japanese Bar Examination
Not on the map. Why?
Abstract (arXiv)
Despite rapid advances in large language models (LLMs), achieving reliable performance on highly professional and structured examinations remains a significant challenge. The Japanese bar examination is a particularly demanding benchmark, requiring not only advanced legal reasoning but also strict adherence to complex answer formats that involve joint evaluation of multiple propositions. While recent studies have reported improvements by decomposing such questions into simpler true--false judgments, these approaches have not been systematically evaluated under the original exam format and scoring scheme, leaving open the question of whether they truly capture exam-level competence. In this paper, we present a self-verification model trained on a newly constructed dataset that faithfully replicates the authentic format and evaluation scale of the exam. Our model is able to exceed the official passing score when evaluated on the actual exam scale, marking the first demonstration, to our knowledge, of an LLM passing the Japanese bar examination without altering its original question structure or scoring rules. We further conduct extensive comparisons with alternative strategies, including multi-agent inference and decomposition-based supervision, and find that these methods fail to achieve comparable performance. Our results highlight the importance of format-faithful supervision and consistency verification, and suggest that carefully designed single-model approaches can outperform more complex systems in high-stakes professional reasoning tasks. Our dataset and codes are publicly available.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 19 references that are in the map. First posted on arXiv 6 Jan 2026.
Builds on
10 of 19 references.
References · 10
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
- 2024A Survey on Large Language Models for Code Generationcited by 658
- 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
- 2023On Learning to Summarize with Large Language Models as Referencescited by 80
- 2020JNLP Team: Deep Learning for Legal Processing in COLIEE 2020cited by 14
- 2024LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultationcited by 14
- 2021JNLP Team: Deep Learning Approaches for Legal Processing Tasks in COLIEE 2021cited by 8
- 2025Mitigating Manipulation and Enhancing Persuasion: A Reflective Multi-Agent Approach for Legal Argument Generationcited by 8
- 2025NOWJ@COLIEE 2025: A Multi-stage Framework Integrating Embedding Models and Large Language Models for Legal Retrieval and Entailmentcited by 1
Built on by
Not cited yet.