After milvus migration a search shows dimension as None although it is 128

[rongilaad]

After milvus migration completes successfully, when I try to perform a search there is an error that the dimension is None although it is 128 when I check it with collection.describe().

Here are the details:

1. The migration completes successfully:

./milvus-migration dump
[2024/07/17 09:48:26.640 +03:00] [INFO] [starter/starter.go:37] ["[Dumper] Dump Success!"] [Cost=8.625285483999999]
Dump Success! Job dumpfggQHSqnxc13ASlTJIf0C cost=[8.626198]
Dump JobInfo! {"jobId":"dumpfggQHSqnxc13ASlTJIf0C","jobStatus":"success","jobProcess":0,"msg":"","totalTasks":0,"finishTasks":1}

./milvus-migration load
[rootPath=output/migration]
[2024/07/17 10:07:35.068 +03:00] [INFO] [cmd/load.go:43] ["[Cleaner] clean file success!"]
Load Success! Job loadP0erG6XYgYrLAlcXtFwaE cost=[40.037896]

2. However when I try to perform a search, it shows that the dimension is None:
   python3 main.py --host=127.0.0.1 --port=19530 --collection=sift1m --query_file=/home/qa/sift1M/sift_query.fvecs --k=1 --dim=128 search
   Collection dimension (None) and query vectors dimension (128) must equal

3. Checking the collection created shows that the dimension is 128:
   python3 main.py --host=127.0.0.1 --port=19530 --collection=sift1m schema_info
   {'collection_name': 'sift1m', 'auto_id': False, 'num_shards': 1, 'description': 'Migration from Milvus1.x', 'fields': [{'field_id': 100, 'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'params': {}, 'is_primary': True}, {'field_id': 101, 'name': 'data', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}], 'aliases': [], 'collection_id': 451160002643446615, 'consistency_level': 2, 'properties': {}, 'num_partitions': 1, 'enable_dynamic_field': False}

[Thomas-HuWei]

After milvus migration completes successfully, when I try to perform a search there is an error that the dimension is None although it is 128 when I check it with collection.describe().

Here are the details:

1. The migration completes successfully:

./milvus-migration dump [2024/07/17 09:48:26.640 +03:00] [INFO] [starter/starter.go:37] ["[Dumper] Dump Success!"] [Cost=8.625285483999999] Dump Success! Job dumpfggQHSqnxc13ASlTJIf0C cost=[8.626198] Dump JobInfo! {"jobId":"dumpfggQHSqnxc13ASlTJIf0C","jobStatus":"success","jobProcess":0,"msg":"","totalTasks":0,"finishTasks":1}

./milvus-migration load [rootPath=output/migration] [2024/07/17 10:07:35.068 +03:00] [INFO] [cmd/load.go:43] ["[Cleaner] clean file success!"] Load Success! Job loadP0erG6XYgYrLAlcXtFwaE cost=[40.037896]

2. However when I try to perform a search, it shows that the dimension is None:
   python3 main.py --host=127.0.0.1 --port=19530 --collection=sift1m --query_file=/home/qa/sift1M/sift_query.fvecs --k=1 --dim=128 search
   Collection dimension (None) and query vectors dimension (128) must equal
3. Checking the collection created shows that the dimension is 128:
   python3 main.py --host=127.0.0.1 --port=19530 --collection=sift1m schema_info
   {'collection_name': 'sift1m', 'auto_id': False, 'num_shards': 1, 'description': 'Migration from Milvus1.x', 'fields': [{'field_id': 100, 'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'params': {}, 'is_primary': True}, {'field_id': 101, 'name': 'data', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}], 'aliases': [], 'collection_id': 451160002643446615, 'consistency_level': 2, 'properties': {}, 'num_partitions': 1, 'enable_dynamic_field': False}

Can you provider a small faiss file to let me do a test. Indeed, the faiss migration function is on beta.

[rongilaad]

@Thomas-HuWei I am using sift1M dataset, and I did the migration with the IVFFLAT index file of it. It's too big to send here even after zip. You can find this dataset on the internet. You can also try it with siftSmall dataset (I haven't tried this one' only the sift1M (It's 1 milion vectors)

[Thomas-HuWei]

@Thomas-HuWei I am using sift1M dataset, and I did the migration with the IVFFLAT index file of it. It's too big to send here even after zip. You can find this dataset on the internet. You can also try it with siftSmall dataset (I haven't tried this one' only the sift1M (It's 1 milion vectors)

I just need a small dataset, and it data struct same with yours enough. So that I can debug the code, I don't know how to reproduce your dataset.