-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: RAPTOR failed due to duplicated Document ID, chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: 17ab7f1d-*********** #14711
Comments
Chroma doesn't allow duplicate ids, which kind of needs to be handled for this process to work. Try a different vector db |
There are duplicates because chunks can belong to more than one cluster |
Could we utilize ChromaDB for this use case? |
Can suggest an better vector store for this use case, which wont end up with duplicate issue. |
I think every other vector store I've tried handles this fine 😅 Maybe try qdrant |
When querying the index (Qdrant Vector DB), I encounter the following issue. Code: from llama_index.packs.raptor import RaptorRetriever from llama_index.core.query_engine import RetrieverQueryEngine query_engine = RetrieverQueryEngine.from_args( response = query_engine.query("Explain about catalogues") Error Logs: Cell In[28], line 23 File c:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\core\instrumentation\dispatcher.py:230, in Dispatcher.span..wrapper(func, instance, args, kwargs) File c:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\core\base\base_query_engine.py:52, in BaseQueryEngine.query(self, str_or_query_bundle) AttributeError: 'NoneType' object has no attribute 'search' |
|
I'm having a use case where i need to input the complete content (End to End with Images and tables) from multiple documents into an LLM/Generative AI model. The goal is to generate a single cohesive piece of content in a specified schema, such as a course or similar output. This means the LLM/Generative AI model needs to have access to all the details from each document to produce content from them. Are there any alternatives to RAG or model fine-tuning? Can you recommend an approach or way for implementing this use case? |
Bug Description
I'm following https://github.com/run-llama/llama_index/blob/main/llama-index-packs/llama-index-packs-raptor/examples/raptor.ipynb to kick off a few RAPTOR-type experiments by myself. However, Im getting an issue when creating the index due to duplicated Document ID. I've provided codes below, where simple directory reader reads is a list of 12 documents.
Error Logs:
Loading files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:49<00:00, 4.11s/file]
Current time: 06:45:08.876467
Generating embeddings for level 0.
Performing clustering for level 0.
Generating summaries for level 0 with 142 clusters.
Level 0 created summaries/clusters: 142
Traceback (most recent call last):
File "c:\Users\017912\Desktop\Dell ChatBot Workflow\ingestion.py", line 43, in
RaptorPack(
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\packs\raptor\base.py", line 348, in init
self.retriever = RaptorRetriever(
^^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\packs\raptor\base.py", line 139, in init
asyncio.run(self.insert(documents))
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\asyncio\base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\packs\raptor\base.py", line 225, File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\core\indices\vector_store\base.py", line 330, in insert_nodes
self._insert(nodes, **insert_kwargs)
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\core\indices\vector_store\base.py", line 312, in _insert
self._add_nodes_to_index(self._index_struct, nodes, **insert_kwargs)
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\core\indices\vector_store\base.py", line 234, in _add_nodes_to_index
new_ids = self._vector_store.add(nodes_batch, **insert_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\llama_index\vector_stores\chroma\base.py", line 265, in add
self._collection.add(
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\chromadb\api\models\Collection.py", line 146, in add
) = self._validate_embedding_set(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\chromadb\api\models\Collection.py", line 545, in _validate_embedding_set valid_ids = validate_ids(maybe_cast_one_to_many_ids(ids))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\017912\AppData\Local\anaconda3\envs\dell_chatbot\Lib\site-packages\chromadb\api\types.py", line 255, in validate_ids
raise errors.DuplicateIDError(message)
chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: 17ab7f1d-b145-42c1-8dc3-8e2e29541823
Version
pip install llama-index-packs-raptor==0.1.3
Steps to Reproduce
Code:
from llama_index.core.settings import Settings
Settings.llm = ingest_llm
Settings.embed_model = embed_model
Settings.context_window = 8192
Settings.num_output = 1024
documents = SimpleDirectoryReader(input_dir="./docs").load_data(show_progress=True)
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("dump")
vector_store = ChromaVectorStore(chroma_collection=collection)
RaptorPack(
documents,
embed_model=embed_model,
llm=ingest_llm,
vector_store=vector_store,
mode="tree_traversal",
transformations=[SentenceSplitter(chunk_size=1024, chunk_overlap=128)],
tree_depth=3,
verbose=True,
)
Relevant Logs/Tracbacks
No response
The text was updated successfully, but these errors were encountered: