Revisiting Bi-Encoder Neural Search: An Encoding--Searching Separation Perspective

This paper reviews, analyzes, and proposes a new perspective on the bi-encoder architecture for neural search. While the bi-encoder architecture is widely used due to its simplicity and scalability at test time, it has some notable issues such as low performance on seen datasets and weak zero-shot performance on new datasets. In this paper, we analyze these issues and summarize two main critiques: the encoding information bottleneck problem and limitations of the basic assumption of embedding search. We then construct a thought experiment to logically analyze the encoding and searching operations and challenge the basic assumptions of embedding search. Building on these observations, we propose a new perspective on the bi-encoder architecture called the textitencoding--searching separation perspective, which conceptually and practically separates the encoding and searching operations. This framework is applied to explain the root cause of existing issues and suggest mitigation strategies, potentially lowering training costs and improving retrieval performance. Finally, we discuss the broader implications of the ideas underlying this perspective, the new design surface it exposes, and potential research directions arising from it.