# Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation

arXiv: https://arxiv.org/abs/2410.20811 · HTML: https://mlsanity.com/paper/2410.20811/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations](https://mlsanity.com/paper/2307.08678/) (2023)
- [ChessGPT: Bridging Policy Learning and Language Modeling](https://mlsanity.com/paper/2306.09200/) (2023)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding](https://mlsanity.com/paper/2309.12482/) (2023)
- [Improving Chess Commentaries by Combining Language Models with Symbolic Reasoning Engines](https://mlsanity.com/paper/2212.08195/) (2022)
- [Automated Chess Commentator Powered by Neural Chess Engine](https://mlsanity.com/paper/1909.10413/) (2019)

## Built on by (most cited first, 6 of 6)

- [ChessQA: Evaluating Large Language Models for Chess Understanding](https://mlsanity.com/paper/2510.23948/) (2025)
- [From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentary](https://mlsanity.com/paper/2506.17294/) (2025)
- [Exploring Collaboration between a language and a non-language agent](https://mlsanity.com/paper/2609.00474/) (2026)
- [Abstract Concept Modelling in Conceptual Spaces: A Study on Chess Strategies](https://mlsanity.com/paper/2601.21771/) (2026)
- [Language Models that Play Chess and Explain Their Moves](https://mlsanity.com/paper/2610.03695/) (2026)
- [Representation-Aligned Auxiliary Supervision for Language Model Adaptation](https://mlsanity.com/paper/2610.04098/) (2026)

## Abstract (arXiv)

Deep learning-based expert models have reached superhuman performance in decision-making domains such as chess and Go. However, it is under-explored to explain or comment on given decisions although it is important for model explainability and human education. The outputs of expert models are accurate, but yet difficult to interpret for humans. On the other hand, large language models (LLMs) can produce fluent commentary but are prone to hallucinations due to their limited decision-making capabilities. To bridge this gap between expert models and LLMs, we focus on chess commentary as a representative task of explaining complex decision-making processes through language and address both the generation and evaluation of commentary. We introduce Concept-guided Chess Commentary generation (CCC) for producing commentary and GPT-based Chess Commentary Evaluation (GCC-Eval) for assessing it. CCC integrates the decision-making strengths of expert models with the linguistic fluency of LLMs through prioritized, concept-based explanations. GCC-Eval leverages expert knowledge to evaluate chess commentary based on informativeness and linguistic quality. Experimental results, validated by both human judges and GCC-Eval, demonstrate that CCC generates commentary which is accurate, informative, and fluent.