Web Crawler
Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.
1 stars
0 tools
Created Aug 2025
@jitsmaster/jitsmaster-web-crawler-webscrapemcpserverAbout
Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.
Categories
general
Installation
npm
npm install @jitsmaster/jitsmaster-web-crawler-webscrapemcpserver
Configuration
{
"mcpServers": {
"jitsmaster-web-crawler-webscrapemcpserver": {
"command": "npx",
"args": [
"-y",
"@jitsmaster/jitsmaster-web-crawler-webscrapemcpserver"
]
}
}
}