Basic RAG Example

Learn how to implement Retrieval-Augmented Generation with NeMo Guardrails

This example demonstrates how to build a RAG (Retrieval-Augmented Generation) system with fact-checking and hallucination detection using NeMo Guardrails.

Overview

This example showcases:

• Custom RAG implementation with knowledge base integration
• Fact-checking using self-check mechanisms
• Hallucination detection for bot responses
• Integration with LangChain for retrieval

Configuration

1. Create the configuration file

   Define your models and enable output rails for fact-checking and hallucination detection.

   config.yml

   models:
     - type: main
       engine: openai
       model: gpt-3.5-turbo
   
   rails:
     output:
       flows:
         - self check facts
         - self check hallucination
   
   prompts:
     - task: self_check_facts
       content: |-
         You are given a task to identify if the hypothesis is grounded and entailed to the evidence.
         You will only use the contents of the evidence and not rely on external knowledge.
         Answer with yes/no. "evidence": {{ evidence }} "hypothesis": {{ response }} "entails":
   
     - task: self_check_hallucinations
       content: |-
         You are given a task to identify if the hypothesis is in agreement with the context below.
         You will only use the contents of the context and not rely on external knowledge.
         Answer with yes/no. "context": {{ paragraph }} "hypothesis": {{ statement }} "agreement":
   
2. Implement the custom RAG action

   Create a custom action that performs retrieval and prepares context for fact-checking.

   config.py

   from langchain_core.language_models import BaseLLM
   from langchain_core.output_parsers import StrOutputParser
   from langchain_core.prompts import PromptTemplate
   
   from nemoguardrails import LLMRails
   from nemoguardrails.actions.actions import ActionResult
   from nemoguardrails.kb.kb import KnowledgeBase
   
   TEMPLATE = """Use the following pieces of context to answer the question at the end.
   If you don't know the answer, just say that you don't know, don't try to make up an answer.
   Use three sentences maximum and keep the answer as concise as possible.
   Always say "thanks for asking!" at the end of the answer.
   
   {context}
   
   Question: {question}
   
   Helpful Answer:"""
   
   
   async def rag(context: dict, llm: BaseLLM, kb: KnowledgeBase) -> ActionResult:
       user_message = context.get("last_user_message")
       context_updates = {}
   
       # For our custom RAG, we re-use the built-in retrieval
       chunks = await kb.search_relevant_chunks(user_message)
       relevant_chunks = "\n".join([chunk["body"] for chunk in chunks])
       # Store the chunks for fact-checking
       context_updates["relevant_chunks"] = relevant_chunks
   
       # Use a custom prompt template
       prompt_template = PromptTemplate.from_template(TEMPLATE)
       input_variables = {"question": user_message, "context": relevant_chunks}
       # Store the template for hallucination-checking
       context_updates["_last_bot_prompt"] = prompt_template.format(**input_variables)
   
       print(f"RAG :: prompt_template: {context_updates['_last_bot_prompt']}")
   
       # Put together a simple LangChain chain
       output_parser = StrOutputParser()
       chain = prompt_template | llm | output_parser
       answer = await chain.ainvoke(input_variables)
   
       return ActionResult(return_value=answer, context_updates=context_updates)
   
   
   def init(app: LLMRails):
       app.register_action(rag, "rag")
   
3. Define the output rails flow

   Create flows that trigger RAG and enable fact-checking.

   rails/output.co

   define user ask about report
     "What was last month's unemployment rate?"
     "Which industry added the most jobs?"
     "How many jobs were added in the transportation industry?"
   
   define flow answer report question
     user ...
     $answer = execute rag()
     $check_facts = True
     $check_hallucination = True
     bot $answer
   
4. Add your knowledge base

   Create a kb/ folder and add your documents. For example:

   config/
   ├── config.yml
   ├── config.py
   ├── rails/
   │   └── output.co
   └── kb/
       └── report.md
   

Usage

Python API

from nemoguardrails import LLMRails, RailsConfig

config = RailsConfig.from_path("./config")
rails = LLMRails(config)

response = rails.generate(
    messages=[{
        "role": "user",
        "content": "What was last month's unemployment rate?"
    }]
)
print(response["content"])

Expected Behavior

1. Query with accurate context

   When asking a question that can be answered from the knowledge base:

   User: What was last month's unemployment rate?
   Bot: The unemployment rate was 3.5 percent. Thanks for asking!
   

   The fact-checking rail verifies the answer against retrieved chunks.

2. Query without sufficient context

   When the answer cannot be verified:

   User: What will next month's unemployment rate be?
   Bot: I don't know that information. Thanks for asking!
   

   The hallucination detection prevents making up answers.

Key Features

• Custom RAG Integration: Implements custom retrieval logic using LangChain
• Fact-Checking: Validates responses against retrieved context
• Hallucination Detection: Prevents the model from generating unsupported claims
• Context Storage: Stores relevant chunks and prompts for verification

Advanced: AlignScore Fact-Checking

For more advanced fact-checking, you can use AlignScore:

config.yml

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo-instruct

rails:
  config:
    fact_checking:
      parameters:
        endpoint: "http://localhost:5123/alignscore_base"

  output:
    flows:
      - alignscore check facts

Related Examples

• QA System - Question-answering with knowledge bases
• Custom Guardrails - Building custom rail logic
• Customer Support - Enterprise chatbot with RAG