Crawl4AI-RAG-MCP-Server
The Crawl4AI RAG MCP Server is designed to enhance the capabilities of AI agents and AI coding assistants by providing powerful web crawling and RAG functionalities. It allows users to scrape, store, and utilize web content efficiently through a Model Context Protocol implementation. Notable features include intelligent URL handling and parallel page processing.
Overview
The Crawl4AI RAG MCP Server is a robust implementation of the Model Context Protocol (MCP) that integrates web crawling and Retrieval-Augmented Generation (RAG) capabilities, specifically designed for AI agents and AI coding assistants. It leverages Crawl4AI and Supabase to provide advanced web crawling tools, enabling the scraping and storage of web content in a vector database, and performing RAG over this content. Key features include smart URL detection, recursive crawling, parallel processing, content chunking, and vector search. The server supports various setup methods for Docker or direct Python execution, requiring basic prerequisites like Docker, Python 3.12+, Supabase, and an OpenAI API key.