-
Notifications
You must be signed in to change notification settings - Fork 567
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vector Embedding Database not persisting when using YAML configuration #600
Comments
I'm using this search workflow as an intermediate step in another workflow in a RAG solution. The search workflow is not able to find the data in the vector database to retrieve relevant texts. |
Sorry for the delayed response. I will try your config and let you know. |
I ran the above configuration and it saves content to the Can you share more on search workflow? Once you load an embeddings database it doesn't automatically refresh.You would need a way to reload the read-only search index when data is loaded. Could this be the issue? |
Sorry for the delay. Yes, that seems to be the issue. Looks like it doesn't automatically load in read-only search index. |
Ok, thank you for confirming. I can think about a method to autodetect index changes and force a refresh. I think that would be a good addition. |
One idea in the meantime to think about is if you can trigger anything with the read-only process when you update your index. |
Not sure if its the same thing, but with the same configuration above, i'm seeing situations where the index directories are not always created. I also noticed when I try to call index, it eventually errors. Should the index directories be added when txtai starts up or not until an index is called? |
I have the following workflow configuration with subindices for two different datasets.
workflow.yaml
After adding the data,
app.count()
returns 1. However, when this data doesn't persist when the session is refreshed.app.count()
returns 0 when run in another separate environment.@davidmezzetti Could you please confirm if I am missing anything in the code and suggest what we could do to persist data?
The text was updated successfully, but these errors were encountered: