# A Re-ranker Scheme for Integrating Large Scale NLU models

arXiv: https://arxiv.org/abs/1809.09605 · HTML: https://mlsanity.com/paper/1809.09605/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2018.

## Built on by (most cited first, 9 of 9)

- [LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging](https://mlsanity.com/paper/2209.09900/) (2022)
- [An Efficient DP-SGD Mechanism for Large Scale NLP Models](https://mlsanity.com/paper/2107.14586/) (2021)
- [Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification](https://mlsanity.com/paper/2106.10826/) (2021)
- [Federated Learning of Models Pre-Trained on Different Features with Consensus Graphs](https://mlsanity.com/paper/2306.01240/) (2023)
- [Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks](https://mlsanity.com/paper/2210.04834/) (2022)
- [Securing RAG: A Risk Assessment and Mitigation Framework](https://mlsanity.com/paper/2505.08728/) (2025)
- [Design Considerations For Hypothesis Rejection Modules In Spoken Language Understanding Systems](https://mlsanity.com/paper/2211.09711/) (2022)
- [Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models](https://mlsanity.com/paper/2109.01754/) (2021)
- [Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space](https://mlsanity.com/paper/2607.01689/) (2026)

## Abstract (arXiv)

Large scale Natural Language Understanding (NLU) systems are typically trained on large quantities of data, requiring a fast and scalable training strategy. A typical design for NLU systems consists of domain-level NLU modules (domain classifier, intent classifier and named entity recognizer). Hypotheses (NLU interpretations consisting of various intent+slot combinations) from these domain specific modules are typically aggregated with another downstream component. The re-ranker integrates outputs from domain-level recognizers, returning a scored list of cross domain hypotheses. An ideal re-ranker will exhibit the following two properties: (a) it should prefer the most relevant hypothesis for the given input as the top hypothesis and, (b) the interpretation scores corresponding to each hypothesis produced by the re-ranker should be calibrated. Calibration allows the final NLU interpretation score to be comparable across domains. We propose a novel re-ranker strategy that addresses these aspects, while also maintaining domain specific modularity. We design optimization loss functions for such a modularized re-ranker and present results on decreasing the top hypothesis error rate as well as maintaining the model calibration. We also experiment with an extension involving training the domain specific re-rankers on datasets curated independently by each domain to allow further asynchronization. %The proposed re-ranker design showcases the following: (i) improved NLU performance over an unweighted aggregation strategy, (ii) cross-domain calibrated performance and, (iii) support for use cases involving training each re-ranker on datasets curated by each domain independently.