# NUBIA: NeUral Based Interchangeability Assessor for Text Generation

arXiv: https://arxiv.org/abs/2004.14667 · HTML: https://mlsanity.com/paper/2004.14667/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 34)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models](https://mlsanity.com/paper/2310.16340/) (2023)
- [CLAIR: Evaluating Image Captions with Large Language Models](https://mlsanity.com/paper/2310.12971/) (2023)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models](https://mlsanity.com/paper/2301.03797/) (2023)
- [Mutual Information Divergence: A Unified Metric for Multimodal Generative Models](https://mlsanity.com/paper/2205.13445/) (2022)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4](https://mlsanity.com/paper/2401.13810/) (2024)
- [RuCoLA: Russian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2210.12814/) (2022)
- [Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speech](https://mlsanity.com/paper/2209.04889/) (2022)
- [Neptune: The Long Orbit to Benchmarking Long Video Understanding](https://mlsanity.com/paper/2412.09582/) (2024)
- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [X-lifecycle Learning for Cloud Incident Management using LLMs](https://mlsanity.com/paper/2404.03662/) (2024)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking](https://mlsanity.com/paper/2411.05375/) (2024)
- [Test-Time Distribution Normalization for Contrastively Learned Vision-language Models](https://mlsanity.com/paper/2302.11084/) (2023)
- [Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2411.05706/) (2024)
- [Better Quality Estimation for Low Resource Corpus Mining](https://mlsanity.com/paper/2203.08259/) (2022)
- [BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation](https://mlsanity.com/paper/2110.09147/) (2021)
- [Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models](https://mlsanity.com/paper/2210.07373/) (2022)
- [Tackling Hallucinations in Neural Chart Summarization](https://mlsanity.com/paper/2308.00399/) (2023)
- [Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review](https://mlsanity.com/paper/2409.18170/) (2024)
- [LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi](https://mlsanity.com/paper/2504.02118/) (2025)
- [Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code](https://mlsanity.com/paper/2204.13848/) (2022)
- [Towards Explainable and Safe Conversational Agents for Mental Health: A Survey](https://mlsanity.com/paper/2304.13191/) (2023)
- [ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generation](https://mlsanity.com/paper/2310.17877/) (2023)

## Abstract (arXiv)

We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules: a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA which outperforms metrics currently used to evaluate machine translation, summaries and slightly exceeds/matches state of the art metrics on correlation with human judgement on the WMT segment-level Direct Assessment task, sentence-level ranking and image captioning evaluation. The model implemented is modular, explainable and set to continuously improve over time.