# Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System

arXiv: https://arxiv.org/abs/2601.05267 · HTML: https://mlsanity.com/paper/2601.05267/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 17 of 17)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Joint Deep Modeling of Users and Items Using Reviews for Recommendation](https://mlsanity.com/paper/1701.04783/) (2017)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition](https://mlsanity.com/paper/2402.15754/) (2024)
- [Content Moderation by LLM: From Accuracy to Legitimacy](https://mlsanity.com/paper/2409.03219/) (2024)
- [Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation](https://mlsanity.com/paper/2303.03608/) (2023)
- [Scaling Up LLM Reviews for Google Ads Content Moderation](https://mlsanity.com/paper/2402.14590/) (2024)
- [AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses](https://mlsanity.com/paper/2410.01246/) (2024)
- [Solving AI Foundational Model Latency with Telco Infrastructure](https://mlsanity.com/paper/2504.03708/) (2025)
- [Extracting Keywords from Open-Ended Business Survey Questions](https://mlsanity.com/paper/1808.10685/) (2018)

## Built on by (most cited first, 1 of 1)

- [Doc2AHP: Inferring Structured Multi-Criteria Decision Models via Semantic Trees with LLMs](https://mlsanity.com/paper/2601.16479/) (2026)

## Abstract (arXiv)

Evaluating complex texts across domains requires converting user defined criteria into quantitative, explainable indicators, which is a persistent challenge in search and recommendation systems. Single prompt LLM evaluations suffer from complexity and latency issues, while criterion specific decomposition approaches rely on naive averaging or opaque black-box aggregation methods. We present an interpretable aggregation framework combining LLM scoring with the Analytic Hierarchy Process. Our method generates criterion specific scores via LLM as judge, measures discriminative power using Jensen Shannon distance, and derives statistically grounded weights through AHP pairwise comparison matrices. Experiments on Amazon review quality assessment and depression related text scoring demonstrate that our approach achieves high explainability and operational efficiency while maintaining comparable predictive power, making it suitable for real time latency sensitive web services.