The NVIDIA Retriever and RAG Evaluation Toolkit provides a standardized way to run evaluations for Retrieval Augmented Generation (RAG) and Retriever pipelines. This toolkit allows users to assess the performance of document retrieval systems and end-to-end RAG systems, including retrieval, context augmentation, and language model generation. Evaluations are configured using YAML files, specifying datasets, models, metrics, and other parameters.

Features

• Retriever Pipelines Evaluation: Assess the performance of document retrieval systems.
• RAG Pipelines Evaluation: Evaluate end-to-end RAG systems, including retrieval, context augmentation, and language model generation.
• YAML Configuration: Evaluations are configured using YAML files, specifying datasets, models, metrics, and other parameters.
• Docker and Local Python Support: Run evaluations inside a benchmark container using Docker or in a local Python environment.
• Metrics Calculation: Supports various metrics for evaluating retriever and RAG pipelines, including recall, nDCG, faithfulness, and relevancy.

Benefits

• Standardized Evaluation: Provides a standardized way to evaluate RAG and Retriever pipelines, ensuring consistency and comparability.
• Flexibility: Supports both Docker and local Python environments, offering flexibility in how evaluations are run.
• Comprehensive Metrics: Offers a wide range of metrics for thorough evaluation of retrieval and generation performance.
• Ease of Use: Simplifies the evaluation process with YAML configuration and detailed documentation.
• Integration with Existing Tools: Compatible with existing tools and workflows, making it easy to integrate into current systems.