mlsanityData to 7 Oct 2026

MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish

arXiv 2608.26344 ↗First posted 26 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 60 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

46 of 60 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
34 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  6. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2019Passage Re-ranking with BERTcited by 629
  9. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  10. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  11. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  12. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  13. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  14. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  15. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  16. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  17. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  18. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  19. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  20. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  21. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  22. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  23. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  24. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  25. 2017Learning Spread-out Local Feature Descriptorscited by 34
  26. 2024MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodingscited by 30
  27. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
  28. 2024Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retrievercited by 25
  29. 2025PyLate: Flexible Training and Retrieval for Late Interaction Modelscited by 17
  30. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  31. 2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
  32. 2024ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrievalcited by 7
  33. 2026DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Searchcited by 3
  34. 2025TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrievalcited by 1

Built on by

Not cited yet.