How I bring my 'RAGStringQueryEngine' to Agentic RAG?

[mraguth]

My objective here is to integrate my 'RAGStringQueryEngine' and OpenAIAgentTools (which I developed using Python functions) into an agentic RAG. I was inspired by the article linked here (specifically the Agentic RAG section). Both my RAGString and OpenAIAgents codes work fine individually. However, my vision is to combine them within the agentic RAG framework. Please help!!!

My Code which is working fine individually:

query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    pandas_prompt=pandas_prompt,
    pandas_output_parser=pandas_output_parser,
    llm=llm,
    qa_prompt=qa_prompt,
    synthesize_response=True,
    response_synthesis_prompt=response_synthesis_prompt,
    callback_manager=Settings.callback_manager,  # Important for Langfuse.
)
agent = OpenAIAgent.from_tools(
    [multiply_tool, add_tool, prq_tool, prq_save_tool], llm=llm, verbose=True
)

The code I am inspired and I like to build like this....

from llama_index.core.tools import ToolMetadata
from llama_index.core.tools import QueryEngineTool
!mkdir vector_data
!wget "https://vectorinstitute.ai/wp-content/uploads/2024/02/Vector-Annual-Report-2022-23_accessible_rev0224-1.pdf" -O "./vector_data/Vector-Annual-Report-2022-23_accessible_rev0224-1.pdf"


# Build basic RAG over Vector
vector_loader = SimpleDirectoryReader(input_dir="./vector_data")
vector_documents = vector_loader.load_data()
vector_index = VectorStoreIndex.from_documents(vector_documents)
vector_query_engine = vector_index.as_query_engine(llm=mistral_llm)

query_engine_tools = [
    QueryEngineTool(
        query_engine=query_engine,
        metadata=ToolMetadata(
            name="rbc_annual_report_2023",
            description=("Provides information about RBC in the year 2023."),
        ),
    ),
    QueryEngineTool(
        query_engine=vector_query_engine,
        metadata=ToolMetadata(
            name="vector_annual_report_2023",
            description=(
                "Provides information about Vector in the year 2023."
            ),
        ),
    ),
]

agent = OpenAIAgent.from_tools(query_engine_tools, verbose=True)

response = agent.chat(
    "According to Vector Institute's Annual Report 2022-2023, "
    "how many AI jobs were created in Ontario?"
)

[dosubot[bot]]

To integrate your RAGStringQueryEngine and OpenAIAgentTools into the agentic RAG framework, you can follow the example provided below. This example demonstrates how to combine both components within the agentic RAG framework:

from llama_index.core.tools import ToolMetadata, QueryEngineTool
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, Settings
from llama_index.core.agents import OpenAIAgent
from llama_index.agent.llm_compiler import LLMCompilerAgentWorker
from llama_index.core.agent import AgentRunner
from llama_index.llms.openai import OpenAI

# Set up the LLM in global settings
Settings.llm = OpenAI(model="gpt-4", temperature=0)

# Load and index documents for RAGStringQueryEngine
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
rag_query_engine = index.as_query_engine()

# Initialize your OpenAIAgentTools
openai_agent_tools = OpenAIAgentTools(...)

# Create QueryEngineTool instances
query_engine_tools = [
    QueryEngineTool(
        query_engine=rag_query_engine,
        metadata=ToolMetadata(
            name="rag_string_query_engine",
            description="Provides information using RAGStringQueryEngine."
        ),
    ),
    QueryEngineTool(
        query_engine=openai_agent_tools,
        metadata=ToolMetadata(
            name="openai_agent_tools",
            description="Provides information using OpenAIAgentTools."
        ),
    ),
]

# Combine the tools into an LLMCompilerAgentWorker
agent_worker = LLMCompilerAgentWorker.from_tools(
    query_engine_tools,
    llm=Settings.llm,
    verbose=True,
    callback_manager=Settings.llm.callback_manager,
)

# Create an AgentRunner
agent = AgentRunner(agent_worker, callback_manager=Settings.llm.callback_manager)

# Use the agent to handle queries
response = agent.chat("Your query here")
print(response)

This code sets up the QueryEngineTool instances for both RAGStringQueryEngine and OpenAIAgentTools, and then combines them into an LLMCompilerAgentWorker. The AgentRunner is then used to handle queries ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu I am using my query_engine with below code. My question here is how I incorporate below query_engine with your code? Your code is saying that rag_query_engine = index.as_query_engine() that I need to index my query engine. But I did already indexed in 'recursive_retriever' like shown below. Please hlep.

construct top-level vector index + query engine

vector_index = VectorStoreIndex(doc_nodes + df_nodes)
vector_retriever = vector_index.as_retriever(similarity_top_k=1)

recursive_retriever = RecursiveRetriever(
    "vector",
    retriever_dict={"vector": vector_retriever},
    query_engine_dict=df_id_query_engine_mapping,
    verbose=False,                                                  #### Change to see the data
)

query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    pandas_prompt=pandas_prompt,
    pandas_output_parser=pandas_output_parser,
    llm=llm,
    qa_prompt=qa_prompt,
    synthesize_response=True,
    response_synthesis_prompt=response_synthesis_prompt,
    callback_manager=Settings.callback_manager,                                  ###### Important for Langfuse.
)

[dosubot[bot]]

To incorporate your already indexed RAGStringQueryEngine into the agentic RAG framework, you can directly use your query_engine in place of the rag_query_engine in the provided code. Here is how you can do it:

1. Ensure your query_engine is properly initialized and indexed.
2. Replace the rag_query_engine assignment with your query_engine.
3. Use this query_engine to create the QueryEngineTool.

Here is the modified code snippet:

from llama_index.core.tools import ToolMetadata, QueryEngineTool
from llama_index.core.agents import OpenAIAgent
from llama_index.core import Settings

# Assuming your RAGStringQueryEngine is already initialized and indexed
query_engine = RAGStringQueryEngine(
    retriever=recursive_retriever,
    response_synthesizer=response_synthesizer,
    pandas_prompt=pandas_prompt,
    pandas_output_parser=pandas_output_parser,
    llm=llm,
    qa_prompt=qa_prompt,
    synthesize_response=True,
    response_synthesis_prompt=response_synthesis_prompt,
    callback_manager=Settings.callback_manager,
)

# Create QueryEngineTool instances
query_engine_tools = [
    QueryEngineTool(
        query_engine=query_engine,
        metadata=ToolMetadata(
            name="rag_string_query_engine",
            description="Provides information using RAGStringQueryEngine."
        ),
    ),
    # Add other tools if needed
]

# Initialize the agent with the tools
agent = OpenAIAgent.from_tools(query_engine_tools, verbose=True)

# Use the agent to handle queries
response = agent.chat("Your query here")
print(response)

This code integrates your custom RAGStringQueryEngine into the agentic RAG framework by creating a QueryEngineTool with your query_engine and then initializing the OpenAIAgent with this tool ^[1][2][3].

To continue talking to Dosu, mention @dosu.