Avoid of creating any local/in-memory document storage

[omriel1]

Hi,
I'm trying to build a simple RAG application using llama index and milvus as the vector storage.
The issue is, I don't want to allocate any local or in-memory storage for indices or document storage: I want to store the document texts in Milvus (alongside with its embeddings), and to use this text in retrieval stage.

When I use the following simple code, it seems VectorStoreIndex.from_vector_store creates behind the scene in-memory BaseIndexStore and BaseDocumentStore (while initializing default StorageContext). Can I avoid this behaviour?

custom_text_splitter = RecursiveCharacterTextSplitter(
  chunk_size=200,
  chunk_overlap=20,
  separators=["\n\n", "(?<=\. )", "\n", " ", ""],
  length_function=len
)
splitter = LangchainNodeParser(lc_splitter=custom_text_splitter)

# Setup LLM
llm: LLM = get_llm(creds=creds)

# Setup embedding model
embed_model = FakeEmbeddings(size=384)
service_context = ServiceContext.from_defaults(
  node_parser=splitter,
  llm=llm,
  embed_model=embed_model,
  chunk_size=1024,
  query_wrapper_prompt=Prompt("<s>[INST] {query_str} [/INST]"),
  system_prompt="<<SYS>> Answer just what you are asked, without being polite. Answer straight to the point, "
                "do not use formalities nor greetings, don't start with Here. don't start with Sure<</SYS>>",
)

vector_store = MilvusVectorStore(
    host = "localhost",
    port = "19530",
    collection_name = "webscrape",
    dim=384,
    overwrite=False,
)
index = VectorStoreIndex.from_vector_store(
    vector_store=vector_store,
    service_context=service_context
)

[dosubot[bot]]

🤖

Hey @OmriLevyTau! I'm Dosu, a friendly bot here to help you navigate through LlamaIndex while you wait for a human maintainer to assist you. I'm here to help with bugs, answer your questions, and guide you on becoming a contributor. Let's dive into your issue together!

Based on the information provided, it seems that the VectorStoreIndex.from_vector_store method in LlamaIndex does create in-memory BaseIndexStore and BaseDocumentStore by default. This is because the from_vector_store method creates a StorageContext using StorageContext.from_defaults(vector_store=vector_store), which by default creates these in-memory stores.

However, there are a couple of ways you might be able to work around this:

1. Modify the StorageContext.from_defaults method: You could potentially modify this method to accept additional parameters that specify whether to create in-memory stores or not. For example, you could add a new parameter in_memory: bool = False to the method. If in_memory is True, you could create instances of the stores without passing the persist_dir parameter. Here is an example of how you could modify the method:

@classmethod
def from_defaults(
    cls,
    docstore: Optional[BaseDocumentStore] = None,
    index_store: Optional[BaseIndexStore] = None,
    vector_store: Optional[Union[VectorStore, BasePydanticVectorStore]] = None,
    image_store: Optional[VectorStore] = None,
    vector_stores: Optional[
        Dict[str, Union[VectorStore, BasePydanticVectorStore]]
    ] = None,
    graph_store: Optional[GraphStore] = None,
    persist_dir: Optional[str] = None,
    fs: Optional[fsspec.AbstractFileSystem] = None,
    in_memory: bool = False,  # new parameter
) -> "StorageContext":
    """Create a StorageContext from defaults.

    Args:
        docstore (Optional[BaseDocumentStore]): document store
        index_store (Optional[BaseIndexStore]): index store
        vector_store (Optional[VectorStore]): vector store
        graph_store (Optional[GraphStore]): graph store
        image_store (Optional[VectorStore]): image store
        in_memory (bool): whether to create in-memory stores or not

    """
    if in_memory or persist_dir is None:
        docstore = docstore or SimpleDocumentStore()
        index_store = index_store or SimpleIndexStore()
        graph_store = graph_store or SimpleGraphStore()
        image_store = image_store or SimpleVectorStore()
        if vector_store:
            vector_stores = {DEFAULT_VECTOR_STORE: vector_store}
        else:
            vector_stores = vector_stores or {
                DEFAULT_VECTOR_STORE: SimpleVectorStore()
            }
        if image_store:
            # append image store to vector stores
            vector_stores[IMAGE_VECTOR_STORE_NAMESPACE] = image_store
    else:
        # existing code
        ...

In this modification, if in_memory is True, the method will create instances of the stores without passing the persist_dir parameter, which means the stores will be in-memory stores. If in_memory is False, the method will behave as before.

2. Directly query data without loading documents: If you already have an existing vector store with data already inside, you can directly query data without loading documents. Here is an example of how you can do this:

client = OpensearchVectorClient(endpoint, idx, 1536, embedding_field=embedding_field, text_field=text_field)
# initialize vector store
vector_store = OpensearchVectorStore(client)

And start using that. Note that we changed our API in the latest version (0.6.0), so instead of GPTOpensearchIndex, you can just create the vector store and pass it to the GPTVectorStoreIndex.

Also, you may need to pass the embedding like this:

from llama_index.vector_stores import OpensearchVectorStore
from llama_index.vector_stores.types import VectorStoreQuery
from gpt_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores import OpensearchVectorClient

embed_model = OpenAIEmbedding()

auth = {"user": "admin", "pass": "admin"}
client = OpensearchVectorClient(endpoint, idx, 1536, embedding_field=embedding_field, text_field=text_field, auth=auth)
vector_store = OpensearchVectorStore(client)
res = vector_store.query(VectorStoreQuery(query_str="which projects work on supermarkets?" ,query_embedding=embed_model.get_text_embedding("which projects work on supermarkets?")))
print(res.nodes[0].extra_info["content"])

Please note that such modifications might have other implications on the overall functionality of the LlamaIndex, so it's recommended to thoroughly test the system after such changes.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• [Question]:
• Is there a way to not load documents in memory in order to be able to query an index?
• llama_index/indices/vector_store/base.py
• llama_index/storage/storage_context.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[omriel1]

So any time I use llama_index, all documents are stored in-meory?

[logan-markewich]

See the answer I gave below. That is untrue

[dheerajiiitv]

This is critical, why do the library need to create SimpleKVStore? This uses extra memory which is not required at all.
CC: @logan-markewich @hatianzhang

[dheerajiiitv]

Thanks for the reply,
here is the code

• I am initialising vector index class for indexing and retrieval.

self.vector_index = VectorStoreIndex(
            nodes=[],
            storage_context=self.storage_context,
        )
self.vector_index.insert() <<<

retrieval = vector_index.as_retriever(
                    similarity_top_k=top_k,
                    filters=query_filter,
                )

retrieval.retrieve(query)<<< For Retrieval

[logan-markewich]

So in this case, the tokenizer is need when calling insert, because that's the step where documents are parses into nodes, and we need to count tokens to do that

[dheerajiiitv]

Iam creating nodes at my end l, I am just passing nodes instead of the document

[logan-markewich]

Should be using index.insert_nodes(nodes) then, which skips the chunking process

[dheerajiiitv]

Yes, exactly.

[fernando0315]

OmriLevyTau How did you fix this?

[omriel1]

Didn't fix (but to be honest didn't dive into the issue too deep). I prefer working with LangChain