Reducing randomness in milvus vector query #34645
Replies: 4 comments 8 replies
-
Theoretically, if all the data is persisted into seal segments, compaction is finished, index is already finished, with the same search parameters, the search result should be the same. How many entities in the collection? It would be helpful if you could reproduce it with a python script. You can also try FLAT index to observe. |
Beta Was this translation helpful? Give feedback.
-
that's a interesting question, forward to @liliu-z |
Beta Was this translation helpful? Give feedback.
-
@sickerin |
Beta Was this translation helpful? Give feedback.
-
Basically, I would have image A's embedding in DB of about 10 million with vector 512 dim. When I query the DB with image A's embedding, most of the time, I would match Image A with >0.99 IP distance (this is as expected), but sometimes, image A wouldn't be there in the top embeddings, and I don't have vectors with IP score >0.5. I'll try to create a simplified script to reproduce this. But we can try to reproduce this, by something like querying the DB 100+ times with the same vector that exists in DB to see if it will always give the same result, or sometimes won't hit the vector. |
Beta Was this translation helpful? Give feedback.
-
Hi, I want to ask a question about sometimes not getting a hit. Suppose I have the same milvus DB with no changes, no reloads, and no new vectors added.
I sometimes find that when I search vector A, sometimes I will get vector A as a hit, but sometimes it won't hit the vector (maybe 1 out of 10 times, despite the Vector DB not changing, and no loads called or new vector inserted). I understand that milvus uses ANN and that's it is an approximation, but wondering whether there's a way to seed it such that if the Vector DB is the same, with no changes, searching the same vector will yield the same results.
This is my index config for example
or
Beta Was this translation helpful? Give feedback.
All reactions