# Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking

arXiv: https://arxiv.org/abs/2605.12028 · HTML: https://mlsanity.com/paper/2605.12028/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 9 of 9)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Question Rewriting for Conversational Question Answering](https://mlsanity.com/paper/2004.14652/) (2020)
- [BM25S: Orders of magnitude faster lexical search via eager sparse scoring](https://mlsanity.com/paper/2407.03618/) (2024)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [Pytrec_eval: An Extremely Fast Python Interface to trec_eval](https://mlsanity.com/paper/1805.01597/) (2018)
- [MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations](https://mlsanity.com/paper/2602.23184/) (2026)

## Abstract (arXiv)

We describe our system for SemEval-2026 Task 8 (MTRAGEval), participating in Task A (Retrieval) across four English-language domains. Our approach employs a three-stage pipeline: (1) query rewriting via a LoRA-fine-tuned Qwen 2.5 7B model that transforms context-dependent follow-up questions into standalone queries, (2) hybrid BM25 and dense retrieval combined through Reciprocal Rank Fusion, and (3) cross-encoder reranking with BGE-reranker-v2-m3. On the official test set, the system achieves nDCG@5 of 0.531, ranking 8th out of 38 participating systems and 10.7% above the organizer baseline. Development comparisons reveal that domain-specific temperature tuning for query generation, where technical domains benefit from deterministic decoding and general domains from controlled randomness, provides consistent gains, while more complex strategies such as domain-aware prompting and multi-query expansion degrade performance.