mlsanityData to 7 Oct 2026

Exploring Classic and Neural Lexical Translation Models for Information Retrieval: Interpretability, Effectiveness, and Efficiency Benefits

arXiv 2102.06815 ↗First posted 12 Feb 2021cs.CL cs.IRcited by 3

Not on the map. Why?

Abstract (arXiv)

We study the utility of the lexical translation model (IBM Model 1) for English text retrieval, in particular, its neural variants that are trained end-to-end. We use the neural Model1 as an aggregator layer applied to context-free or contextualized query/document embeddings. This new approach to design a neural ranking system has benefits for effectiveness, efficiency, and interpretability. Specifically, we show that adding an interpretable neural Model 1 layer on top of BERT-based contextualized embeddings (1) does not decrease accuracy and/or efficiency; and (2) may overcome the limitation on the maximum sequence length of existing BERT models. The context-free neural Model 1 is less effective than a BERT-based ranking model, but it can run efficiently on a CPU (without expensive index-time precomputation or query-time operations on large tensors). Using Model 1 we produced best neural and non-neural runs on the MS MARCO document ranking leaderboard in late 2020.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 35 of its 74 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

35 of 74 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Layer Normalizationcited by 5,579
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2019Attention is not Explanationcited by 916
  12. 2020Efficient Transformers: A Surveycited by 784
23 more
  1. 2016The Mythos of Model Interpretabilitycited by 716
  2. 2019Passage Re-ranking with BERTcited by 629
  3. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  4. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
  5. 2019Is Attention Interpretable?cited by 387
  6. 2020Overview of the TREC 2019 deep learning trackcited by 282
  7. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  8. 2019Document Expansion by Query Predictioncited by 225
  9. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  10. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 111
  11. 2017End-to-End Neural Ad-hoc Ranking with Kernel Poolingcited by 85
  12. 2019A Deep Look into Neural Ranking Models for Information Retrievalcited by 79
  13. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  14. 2015A Primer on Neural Network Models for Natural Language Processingcited by 75
  15. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  16. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  17. 2020Expansion via Prediction of Importance with Contextualizationcited by 26
  18. 2020Interpretable & Time-Budget-Constrained Contextualization for Re-Rankingcited by 21
  19. 2020Flexible retrieval with NMSLIB and FlexNeuARTcited by 14
  20. 2019Let's measure run time! Extending the IR replicability infrastructure to include performance aspectscited by 10
  21. 2020Conformer-Kernel with Query Term Independence for Document Retrievalcited by 3
  22. 2020Modularized Transfomer-based Ranking Frameworkcited by 2
  23. 2020Traditional IR rivals neural models on the MS MARCO Document Ranking Leaderboardcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

3 papers.

  1. 2021A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Modelscited by 8
  2. 2022Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigationcited by 2
  3. 2026Improving Neural Retrieval with Attribution-Guided Query Rewritingcited by 0