patronus-mcp-server
Patronus MCP Server is a standardized server designed to optimize and evaluate LLM systems using the Patronus SDK. It offers features like configurable evaluators for single and batch evaluations, and the capability to run experiments with datasets.
Patronus MCP Server
An MCP server implementation for the Patronus SDK, providing a standardized interface for running powerful LLM system optimizations, evaluations, and experiments.
Features
- Initialize Patronus with API key and project settings
- Run single and batch evaluations with configurable evaluators
- Run experiments with datasets
Installation
- Clone the repository and navigate into it.
- Create and activate a virtual environment.
- Install main and dev dependencies.
Usage
Running the Server
Run the server using an API key via command line or environment variable.
Interactive Testing
The test script provides interactive testing for evaluation endpoints. Can be run using command line or environment settings.
API Usage
- Initialize
- Run Single and Batch Evaluations
- Run Experiments
Development
Project Structure
Outlines the directory structure of the project.
Adding New Features
Detailed guide on defining new request models, implementing tool functions, adding tests, and updating the README.
Running Tests
Test script supports interactive test selection and JSON response formatting.
Running the Server
Basic command to run the server application.
License
This project is licensed under the Apache License 2.0.
Contributing
Instructions on how to contribute to the project.