Reducing randomness in milvus vector query

[sickerin]

Hi, I want to ask a question about sometimes not getting a hit. Suppose I have the same milvus DB with no changes, no reloads, and no new vectors added.

I sometimes find that when I search vector A, sometimes I will get vector A as a hit, but sometimes it won't hit the vector (maybe 1 out of 10 times, despite the Vector DB not changing, and no loads called or new vector inserted). I understand that milvus uses ANN and that's it is an approximation, but wondering whether there's a way to seed it such that if the Vector DB is the same, with no changes, searching the same vector will yield the same results.

This is my index config for example

    "VECTOR_DIM": 512,
    "NUM_TOP_K": 10,
    "METRIC_TYPE": "IP",
    "SEARCH_PARAM": {"ef": 30},
    "INDEX_TYPE": "HNSW",
    "INDEX_PARAM": {
          "M": 48,
          "efConstruction": 300
      },

or

index_params = {
    "metric_type": "IP",
    "index_type": "IVF_FLAT",
    "params": {
        "nlist":1024
    },
}
params_for_search = {
   "nprobe": 16
}

[yhmo]

Theoretically, if all the data is persisted into seal segments, compaction is finished, index is already finished, with the same search parameters, the search result should be the same.

How many entities in the collection? It would be helpful if you could reproduce it with a python script.

You can also try FLAT index to observe.

[xiaofan-luan]

that's a interesting question, forward to @liliu-z

[xiaofan-luan]

@sickerin
when you see randomess, can you share the two different result? is that possible that they are with same distance?
any script we can reproduce this problem?

[sickerin]

Basically, I would have image A's embedding in DB of about 10 million with vector 512 dim. When I query the DB with image A's embedding, most of the time, I would match Image A with >0.99 IP distance (this is as expected), but sometimes, image A wouldn't be there in the top embeddings, and I don't have vectors with IP score >0.5.

I'll try to create a simplified script to reproduce this. But we can try to reproduce this, by something like querying the DB 100+ times with the same vector that exists in DB to see if it will always give the same result, or sometimes won't hit the vector.

[sickerin]

My configs are usually like this

          "VECTOR_DIM": 512,
          "NUM_TOP_K": 10,
          "METRIC_TYPE": "IP",
          "SEARCH_PARAM": {"ef": 30},
          "INDEX_TYPE": "HNSW",
          "INDEX_PARAM": {
                "M": 48,
                "efConstruction": 300
            },

        "VECTOR_DIM": 512,
        "NUM_TOP_K": 10,
        "METRIC_TYPE": "IP",
        "SEARCH_PARAM": {"nprobe": 16},
        "INDEX_TYPE": "IVF_FLAT",
        "INDEX_PARAM": {"nlist":1024},

This is the type of code I used simplified

 def search(
        cls,
        collection_name: str,
        embedding_vec: List,
        image_type: str,
        query_types: List[str],
        timings,
    ):

        found_distance_search = []
        found_image_path = []
        found_image_type = []
        found_product_type = []

        # Step 1: collect searching parameters
        search_collection_config = Milvus.get_collection_config(collection_name)
        metric_type = search_collection_config["METRIC_TYPE"]
        if image_type == "type_a":
            cosine_threshold = search_collection_config["COSINE_THRESHOLD"]["type_a"]
        elif image_type == "type_b":
            cosine_threshold = search_collection_config["COSINE_THRESHOLD"]["type_b"]
        else:
            raise ValueError("not supported image_type for cosine threshold")
        num_top_k = search_collection_config["NUM_TOP_K"]
        params = search_collection_config["SEARCH_PARAM"]
        embedding_vec = np.array(embedding_vec)
        # Form query expression
        expr = f'{SchemaName.IMAGE_TYPE}=="{image_type}"'
        search_param = {
            "anns_field": SchemaName.EMBED_VECTOR,
            "param": {"metric_type": metric_type, "params": params},
            "limit": num_top_k,
            "expr": expr,
            "partition_names": query_types,
            "output_fields": [
                SchemaName.IMAGE_PATH,
                SchemaName.IMAGE_TYPE,
                SchemaName.PRODUCT_TYPE,
            ],
        }


        # # Step 2: Skip to load collection, use lazy load instead

        # Step 3: Transform embedding vector to float32
        embedding_vec = embedding_vec.reshape(1, -1).astype(np.float32)

        # Step 4: Search the vector in the collection
        step_name = "milvus_search"
        with DictKeyTimer(timings, step_name):
            collection = Collection(collection_name)
            results = collection.search(embedding_vec, **search_param)
        
        # Step 5: Collect searching results and do filtering
        step_name = "filter_result"
        with DictKeyTimer(timings, step_name):
            for hit in results[0]:  # For single embedding vector
                dis = hit.distance
                dis = float(dis)
                if dis < cosine_threshold:
                    continue
                found_distance_search.append(dis)
                found_image_path.append(hit.entity.get(SchemaName.IMAGE_PATH))
                found_image_type.append(hit.entity.get(SchemaName.IMAGE_TYPE))
                found_product_type.append(hit.entity.get(SchemaName.PRODUCT_TYPE))

    
        if len(found_image_path) > 0:
            return (
                found_distance_search,
                found_image_path,
                found_image_type,
                found_product_type,
            )
        else:
            return [], [], [], []

[xiaofan-luan]

oh, usually I filter the NUM_TOP_K=10 results with a threshold of example let's say 0.5. Then most of the the time, I get searching with image_A

filtered_match_results : {
"image_A": 0.999,
"image_B": 0.6
}

but sometimes, I get

filtered_match_results : {
"image_B": 0.6
}

or

filtered_match_results : {
}

The numbers (0.999, 0.6) are IP between the image vectors.

So you are saying with different search, sometimes you can not get result?
I think this is as expected, sometimes with the filtering condition you can not find any data with range > certain ratio.
Unless you find that for same pic and same expr you can randomly see or not see result, then it's a problem.

[liliu-z]

@sickerin Did you repeat search with same params and filters but get different result?

[sickerin]

Yes, I repeat the same search with the same image and settings, sometimes I can't find it, so I was wondering whether there's some quantization or randomness that affects this. I understand that if I search with a different image sometimes due to ANN, it may not find it's matching pair despite the high score. But suppose image_A can be found, does it mean it can always be found?

[xiaofan-luan]

There is one exceptional case, convert growing segment in to sealed or compaction. Under those case, the index actaully changed so you might or might not get the same search result