Configure and customize queries and response generation

Follow the console steps at Query a knowledge base and generate AI responses. When you open the Configurations pane, you'll see the following options for Search type:

When you make a Retrieve or RetrieveAndGenerate request, include a retrievalConfiguration field, mapped to a KnowledgeBaseRetrievalConfiguration object. To see the location of this field, refer to the Retrieve and RetrieveAndGenerate request bodies in the API reference.

The following JSON object shows the minimal fields required in the KnowledgeBaseRetrievalConfiguration object to set search type configurations:

"retrievalConfiguration": {
    "vectorSearchConfiguration": {
        "overrideSearchType": "HYBRID | SEMANTIC"
    }
}

Specify the search type in the overrideSearchType field. You have the following options:

POST /retrieveAndGenerate HTTP/1.1
Content-type: application/json
{
   "input": {
      "text": "string"
   },
   "retrieveAndGenerateConfiguration": {
      "knowledgeBaseConfiguration": {
         "orchestrationConfiguration": { // Query decomposition
           "queryTransformationConfiguration": {
                "type": "string" // enum of QUERY_DECOMPOSITION
           }
         },
...}
}
     

To modify inference parameters when querying a knowledge base – Follow the console steps at Query a knowledge base and generate AI responses. When you open the Configurations pane, you'll see an Inference parameters section. Modify the parameters as necessary.

To modify inference parameters when chatting with your document – Follow the steps at Chat with your document without a knowledge base configured. In the Configurations pane, expand the Inference parameters section and modify the parameters as necessary.

You provide the model parameters in the call to the RetrieveAndGenerate API. You can customize the model by providing inference parameters in the inferenceConfig field of either the knowledgeBaseConfiguration (if you query a knowledge base) or the externalSourcesConfiguration (if you chat with your document).

Within the inferenceConfig field is a textInferenceConfig field that contains the following parameters that you can:

You can customize the model by using the following parameters in the inferenceConfig field of both externalSourcesConfiguration and knowledgeBaseConfiguration:

For a detailed explanation of the function of each of these parameters, see Influence response generation with inference parameters.

Additionally, you can provide custom parameters not supported by textInferenceConfig via the additionalModelRequestFields map. You can provide parameters unique to specific models with this argument, for the unique paramaters see Inference parameters for foundation models.

If a parameter is omitted from textInferenceConfig, a default value will be used. Any parameters not recognized in textInferneceConfig will be ignored, while any parameters not recognized in AdditionalModelRequestFields will cause an exception.

A validation exception is thrown if there is the same parameter in both additionalModelRequestFields and TextInferenceConfig.

Using model parameters in RetrieveAndGenerate

The following is an example of the structure for inferenceConfig and additionalModelRequestFields under the generationConfiguration in the RetrieveAndGenerate request body:

"inferenceConfig": {
    "textInferenceConfig": {
        "temperature": 0.5,  
        "topP": 0.5,
        "maxTokens": 2048,
        "stopSequences": ["\nObservation"]
    }
},
"additionalModelRequestFields": {
    "top_k": 50
}

The proceeding example sets a temperature of 0.5, top_p of 0.5, maxTokens of 2048, stops generation if it encounters the string "\nObservation" in the generated response, and passes a custom top_k value of 50.

Follow the console steps at Query a knowledge base and generate AI responses. In the Configurations pane, expand the Maximum number of retrieved results.

When you make a Retrieve or RetrieveAndGenerate request, include a retrievalConfiguration field, mapped to a KnowledgeBaseRetrievalConfiguration object. To see the location of this field, refer to the Retrieve and RetrieveAndGenerate request bodies in the API reference.

The following JSON object shows the minimal fields required in the KnowledgeBaseRetrievalConfiguration object to set the maximum number of results to return:

"retrievalConfiguration": {
    "vectorSearchConfiguration": {
        "numberOfResults": number
    }
}

Specify the maximum number of retrieved results (see the numberOfResults field in KnowledgeBaseRetrievalConfiguration for the range of accepted values) to return in the numberOfResults field.

Follow the console steps at Query a knowledge base and generate AI responses. When you open the Configurations pane, you'll see a Filters section. The following procedures describe different use cases:

The following image shows an example filter configuration that returns all documents written after 2018 whose genre is "entertainment", in addition to documents whose genre is "cooking" or "sports" and whose author starts with "C".

When you make a Retrieve or RetrieveAndGenerate request, include a retrievalConfiguration field, mapped to a KnowledgeBaseRetrievalConfiguration object. To see the location of this field, refer to the Retrieve and RetrieveAndGenerate request bodies in the API reference.

The following JSON objects show the minimal fields required in the KnowledgeBaseRetrievalConfiguration object to set filters for different use cases:

The following table describes the filter types that you can use:

To combine filter types, you can use one of the following logical operators:

For examples, see Send a query and include filters (Retrieve) and Send a query and include filters (RetrieveAndGenerate).

Follow the console steps at Query a knowledge base and generate AI responses. In the test window, turn on Generate responses. Then, in the Configurations pane, expand the Knowledge base prompt template section.

When you make a RetrieveAndGenerate request, include a generationConfiguration field, mapped to a GenerationConfiguration object. To see the location of this field, refer to the RetrieveAndGenerate request body in the API reference.

The following JSON object shows the minimal fields required in the GenerationConfiguration object to set the maximum number of retrieved results to return:

"generationConfiguration": {
    "promptTemplate": {
        "textPromptTemplate": "string"
    }
}

Enter your custom prompt template in the textPromptTemplate field, including prompt placeholders and XML tags as necessary. For the maximum number of characters allowed in the system prompt, see the textPromptTemplate field in GenerationConfiguration.

Follow the console steps at Query a knowledge base and generate AI responses. In the test window, turn on Generate responses. Then, in the Configurations pane, expand the Guardrails section.

When you make a RetrieveAndGenerate request, include the guardrailsConfiguration field within the generationConfiguration to use your guardrail with the request. To see the location of this field, refer to the RetrieveAndGenerate request body in the API reference.

The following JSON object shows the minimal fields required in the GenerationConfiguration to set the guardrailsConfiguration:

"generationConfiguration": {
    "guardrailsConfiguration": {
        "guardrailsId": "string",
        "guardrailsVersion": "string"
    }
}

Specify the guardrailsId and guardrailsVersion of your chosen guardrails.