WEB-SCRAPING-MCP

WEB-SCRAPING-MCP

6
browser_automation

Crawl4AI Web Scraper MCP Server is a tool for web scraping and intelligent content extraction using a Model Context Protocol server. It allows AI agents to interact with web pages and extracts information based on natural language instructions.

Crawl4AI Web Scraper MCP Server

This project provides an MCP server for web scraping and intelligent content extraction using the Crawl4AI library. It interacts with web pages, retrieves content, and uses language models for data extraction.

Features

  • Provides MCP tools like scrape_url, extract_text_by_query, and smart_extract.
  • Configurable via environment variables.
  • Docker support for deployment.
  • Communicates on port 8002 using SSE.

Setup and Running

You can run the server using Docker or locally with Python.

Docker Setup

  1. Install Docker
  2. Clone the repository
  3. Create a .env file with API keys
  4. Build and run the Docker image

Local Setup

  1. Install Python >= 3.9
  2. Clone the repository
  3. Create a virtual environment
  4. Install dependencies
  5. Create .env file with API keys
  6. Run the server script