mcp-server-spider
If you are the rightful owner of mcp-server-spider and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to henry@mcpreview.com.
A Model Context Protocol server for Spider crawler interaction and automation.
mcp-server-spider is a Model Context Protocol server designed to facilitate interaction and automation for web crawling and scraping tasks. It provides essential tools to crawl and scrape web pages, making it easier to gather and process web data. The server is currently in early development, which means that users might encounter bugs, and new features may be added over time. The primary functionalities include crawling URLs to discover links and scraping web pages to extract content and links in a structured JSON format. This server is particularly useful for developers and data scientists who need to automate web data collection processes.
Features
- Crawl URLs to discover links
- Scrape web pages to extract content
- Supports custom headers and user agents
- Respects robots.txt and handles invalid certificates
- Whitelist and blacklist URL patterns
Tools
Related MCP Servers
View all browser_automation servers →Fetch
by modelcontextprotocol
A Model Context Protocol server that provides web content fetching capabilities, enabling LLMs to retrieve and process content from web pages.
markdownify-mcp
by zcaceres
Markdownify is a Model Context Protocol (MCP) server that converts various file types and web content to Markdown format.
deepwiki-mcp
by regenrek
This is an unofficial Deepwiki MCP Server that processes Deepwiki URLs, crawls pages, converts them to Markdown, and returns documents or lists by page.
mcp-playwright
by executeautomation
A Model Context Protocol server that provides browser automation capabilities using Playwright.
fetch-mcp
by zcaceres
This MCP server provides functionality to fetch web content in various formats, including HTML, JSON, plain text, and Markdown.
web-eval-agent
by Operative-Sh
operative.sh's MCP Server is a tool for autonomous debugging of web applications directly from your code editor.
cursor-talk-to-figma-mcp
by sonnylazuardi
This project implements a Model Context Protocol (MCP) integration between Cursor AI and Figma, allowing Cursor to communicate with Figma for reading designs and modifying them programmatically.