Filtering in a large collection

[aping]

I have a large collection with 9B rows, dim is 768, searching is ok with 3TB of memory with ivf_sq8 and mmap enabled.
there're also ~10 dynamic fields in the collection, question is, is it possible to do searching + filtering with any one of the dynamic fields?
I mean like

client.search(
        collection_name=collection_name,
        data=vectors,
        filter="field_a == something",
        search_params=search_params,
        ...
    )

I tried but it didn't work, I saw query nodes oom, or restarted by k8s because it failed to respond liveness probe (this is a little bit bizzare, why would it fail liveness probe).
so just wondering, in general is it still possible to filter within a collection of this size?

[yhmo]

Filtering search on a collection with huge dataset is a normal operation.
The OOM you encountered might be an unknown memory issue of mmap.

[aping]

Filtering search on a collection with huge dataset is a normal operation. The OOM you encountered might be an unknown memory issue of mmap.

yeah oom, then I increased the query nodes number, to a total of 5TB memory, then I got exception like query node was gone after ~30s, and when I checked the pods I see some were restarted by k8s because they failed liveness probe. so is there any guidelines here, for example for me 9B embs collection, dim 768, with ivf_sq8 index, how much memory is required, for searching + filtering work?

[xiaofan-luan]

might be related with #34442?

Did you see any OOM kill signal or it might be a SIGBUS cased by this bug

[aping]

not sure, I increased memory a little bit then I don't see oom anymore, now what I see is some query nodes get restarted by k8s because they failed liveness probe, so curious why.
plus I'm also curious how searching+filtering works, I mean does milvus applies filtering first, get a list of ids, then search only with the ids? or something else?
and it's hard to imagine how searching+filtering can work fast with a large collection, I mean I have a 9B collection, if my filter condition matches 100 million of them, can it be possibly fast enough?

[xiaofan-luan]

it kindly based on how did you define fast and how many resources you have.
With enough resources you can achieve 100+QPS and 100ms latency. You can also build inverted index to accelerate filtering

[aping]

whoa that's kind of unexpected, can you describe in general how searching+filtering work?

plus I'm also curious how searching+filtering works, I mean does milvus applies filtering first, get a list of ids, then search only with the ids? or something else?

I'm curious how milvus is able to filter 100M rows from 9B rows within 1 second. for partition key I understand, but for any other random dynamic column that sounds crazy

[xiaofan-luan]

query nodes get restarted by k8s -> can you verify error message or Signal why milvus killed it self?

[aping]

when I do kubectl describe pod it says that it failed liveness probe so restarted by k8s, the liveness check is default and is like

        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: metrics
            scheme: HTTP
          initialDelaySeconds: 90
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5

[xiaofan-luan]

you need to check on milvus log.
there is a reason log why milvus stopped.
To maintain milvus in production environemnt, you need logs && monitoring