MCPDocSearch

MCPDocSearch

12
research_and_datadeveloper_tools

The Documentation Crawler & MCP Server project provides a comprehensive toolset to crawl websites, generate Markdown documentation, and enable semantic search using a Model Context Protocol (MCP) server. It integrates seamlessly with tools like Cursor, offering robust management and query capabilities for documentation content.

Documentation Crawler & MCP Server

The project provides a toolset for crawling websites, generating Markdown documentation from the crawled content, and making it searchable via a Model Context Protocol (MCP) server. Key features include:

  • Web Crawler: Crawls and generates a single Markdown file from website content.
  • MCP Server: Loads Markdown files, parses them into semantic chunks, generates embeddings, and performs semantic searches.
  • Integration: Designed to run with clients like Cursor via stdio transport.

Workflow

  1. Crawl a website and generate a .md file.
  2. Run the mcp_server using an MCP client like Cursor.
  3. Load, chunk, and embed content from .md files.
  4. Interact with the server using tools like list_documents and search_documentation.

Setup

Install dependencies with uv, clone the repository, and run the crawler or MCP server as described.

License

Licensed under the MIT License.