Semantic Recall for Vector Search

We introduce semantic recall, a novel metric that evaluates the quality of embedding-based retrieval by considering only semantically relevant objects. We empirically show the advantages of semantic recall as a proxy for search quality, especially for queries with only a few relevant results among their nearest neighbors. Unlike traditional recall, semantic recall does not penalize algorithms for failing to retrieve objects that are semantically irrelevant to the query. Furthermore, it adapts to any dataset without manual tuning or parametrization.

Additionally, we introduce tolerant recall, a metric that approximates semantic recall when semantic neighbors cannot be identified. Finally, we show that using these recall metrics as an objective function for tuning search algorithms yields higher quality results at a lower cost. We hope that this study encourages the information retrieval community to adopt more
meaningful quality metrics to evaluate semantic search.