questions about sparse index

[Gladiator566]

Hi, I'd like use milvus 2.4.6 to perform a hybrid search using BGE-M3 model, I want to use both sparse vectors and dense vectors, my questions are focus on sparse vectors index type.
I've seen this tutorial about sparse index , I just wonder the difference between SPARSE_INVERTED_INDEX and SPARSE_WAND, if I want to index datasets whose number over millions, which one is better? Is SPARSE_INVERTED_INDEX performs more like ES and SPARSE_WAND better speed with low sacrifice in recall ? And is there some recommend empirical value for param drop_ratio_build and drop_ratio_search ?
Thank you~

[xiaofan-luan]

SPARSE_WAND is usually better at speed.

[Gladiator566]

what is the default value for index and search params? Is there some recommend empirical value for param drop_ratio_build and drop_ratio_search ?

[zhengbuqian]

Hi @Gladiator566, thanks for the question!

the difference between SPARSE_INVERTED_INDEX and SPARSE_WAND

• SPARSE_INVERTED_INDEX is the plain old inverted index: for each dimension, maintain a list of doc ids that is not zero in this dimension; at search time, only compute scores of docs in merged lists of the query's non zero dimensions.
• SPARSE_WAND is similar to SPARSE_INVERTED_INDEX in that they both have the same inverted index, but SPARSE_WAND maintains more metadata in the index to allow the utilization of the WAND algorithm to greatly reduce the number of docs that needs to be evaluated. If you are interested in the detail, you may refer to the original paper or this slide.

The choice between SPARSE_WAND and SPARSE_INVERTED_INDEX mostly depends on the density(average number of non zeros / total number of possible dimensions) of the dataset and the queries. In most cases SPARSE_WAND should be much faster than SPARSE_INVERTED_INDEX, especially for doc embeddings with higher density and datasets with tons of docs. But SPARSE_WAND's performance downgrades fast as the dimensions of the query increases: every time when we see slower WAND, we see long queries with small drop_ratio_search.

recommend empirical value for param drop_ratio_build and drop_ratio_search

The default value for drop_ratio_build/search is 0. With this setting both indexes yield 100% recall rate. And the same drop_ratio_build/search yields the same recall rate on SPARSE_WAND/SPARSE_INVERTED_INDEX.

Choice of drop_ratio_build/search mainly depends on the datasets density and the requirement for recall rate and we don't really have an empirical value as both are unknown.

I would suggest to start from drop_ratio_build=0.3 and drop_ratio_search=0.6 and see if that meets your recall requirement, and adjust as needed. Also I suggest to adjust drop_ratio_search more aggressively if it's ok to sacrifice recall a little for a huge performance improvement: short query can greatly speed up SPARSE_WAND with little accuracy loss, since small values contributes little to IP score.