johannschifferAnti-bot web scraping, clean Markdown, proxy rotation and TLS evasion
Anti-bot web scraping, clean Markdown, proxy rotation and TLS evasion
CrawlEngine MCP (Lite Edition)
⚡ Enterprise Infrastructure & Tooling
Package Access Features Starter Kit Free ($0+) 25+ .cursorrules,CLAUDE.mdstack rules & MCP config templates. Download FreeEnterprise Suite Pro $19 USD 6 Pro MCP Servers, Zero-Downtime DB migrations, Residential Proxies, Docker Compose Stack & Commercial License. Get Pro License Config Generator Free Web Tool Interactive visual config generator for Claude Desktop & Cursor. Open Generator
Official Indexing: Verified and listed in TensorBlock/awesome-mcp-servers under Browser Automation & Web Scraping.
CrawlEngine MCP is a resilient Model Context Protocol server that gives AI agents in Claude Desktop, Cursor, Windsurf, and Antigravity clean web-reading superpowers without choking context windows on raw HTML or getting instantly blocked by basic anti-bot firewalls.
Run without cloning
uvx crawlengine-mcpClaude Desktop / Cursor config entry:
{
"mcpServers": {
"crawlengine-mcp": {
"command": "uvx",
"args": [
"crawlengine-mcp"
]
}
}
}🕷️ Why CrawlEngine MCP?
Most default web-fetching tools used by AI models suffer from severe drawbacks:
- Raw HTML Token Bloat: A 150KB web page contains ~90% useless boilerplate (navbars, tracking pixels, CSS bundles, cookie banners) that burns LLM tokens and induces hallucinations.
- Instant HTTP 403 Blocks: Default Python
urlliborrequestsUser-Agents trigger anti-bot WAFs immediately. - Broken Relative Links: Naive scrapers return unnavigable relative paths (
/docs/quickstart) instead of canonical URLs.
CrawlEngine solves this by rotating realistic modern browser fingerprints and transforming messy DOM trees into pristine, semantic Markdown.
⚡ Features (Lite Edition)
- 🧹 Semantic Markdown Conversion: Strips scripts, styles, popups, cookie consent overlays, and ads, keeping only the core article or documentation.
- 🎭 Realistic Browser Fingerprinting: Automatically rotates
User-Agent,Sec-Ch-Ua, and platform headers across Chrome, Safari, and Firefox. - 🔗 Deep Link Discovery: Extracts all outbound and internal links with anchor text.
- 🏷️ Metadata Extraction: Gathers OpenGraph, Twitter Cards, meta descriptions, and canonical URLs.
- 🪶 Zero Complex Dependencies: Pure Python using
httpxandbeautifulsoup4.
🚀 Quick Setup in 2 Minutes
1. Installation
git clone https://github.com/johannschiffer/crawlengine-mcp.git
cd crawlengine-mcp
pip install -r requirements.txt2. Configure Claude Desktop (claude_desktop_config.json)
On Linux: ~/.config/Claude/claude_desktop_config.json
On macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
On Windows: %APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"crawlengine": {
"command": "python",
"args": ["-m", "crawlengine.server"],
"cwd": "/path/to/crawlengine-mcp"
}
}
}3. Verify via CLI
python -m crawlengine.cli scrape https://news.ycombinator.com🛠️ MCP Tools Exposed to AI Agents
| Tool | Parameters | Description |
|---|---|---|
scrape_url | url, extract_markdown, timeout | Fetches web page and extracts clean, token-efficient Markdown |
extract_links | url, internal_only | Returns structured array of hyperlinks with anchor texts |
extract_metadata | url | Retrieves page title, meta description, and OpenGraph data |
💎 Need Enterprise Anti-Bot Scraping? Upgrade to CrawlEngine Pro
Dealing with complex JavaScript SPAs, Cloudflare Turnstile, or scraping behind proxy pools?
| Feature | Lite (Open Source) | Enterprise Suite ($19) |
|---|---|---|
| Semantic Markdown Extraction | ✅ | ✅ |
| Browser Fingerprint Rotation | ✅ | ✅ |
| OpenGraph & Link Extraction | ✅ | ✅ |
| Automatic Headless Browser Fallback (Playwright Chromium) | ❌ | ✅ |
| JavaScript SPA Hydration (React, Next.js, Vue) | ❌ | ✅ |
| Proxy Pool Rotation (HTTP & SOCKS5 with auto-ejection) | ❌ | ✅ |
| Structured Data Extractor (JSON-LD, Pricing Tables, Products) | ❌ | ✅ |
| Production Docker Container | ❌ | ✅ |
| Full Pytest Suite & Anti-Detection Guide | ❌ | ✅ |
👉 Get the Genesis Enterprise Suite ($19)
📄 License
MIT License. See LICENSE for details.
Recommended Infrastructure
Proxies that work well with CrawlEngine MCP. Some links are referral links.
Genesis MCP Ecosystem
| Module | What it does | Repo |
|---|---|---|
| CrawlEngine MCP (this repo) | Anti-bot web scraping, clean Markdown, proxy rotation and TLS evasion | GitHub |
| BrowserBridge MCP | Deterministic headless browser automation with cryptographic action receipts | GitHub |
| PgSafe MCP | Zero-trust PostgreSQL/SQLite guard, live PII masking, zero-downtime migration checks | GitHub |
| OmniBridge MCP | Multi-channel notification and webhook bus with a persistent SQLite WAL queue | GitHub |
| SysFlow MCP | Safe Linux administration plane: systemd, port audit, isolated execution | GitHub |
| FastMCP Forge | Production MCP server scaffolder: stdio, SSE and auth middleware | GitHub |
Live config builder | Starter Kit, free | Enterprise Suite, $19
Pick your client and paste the snippet. Each one is the same server, written the way that client expects it.
- Run the command below in your terminal.
- Restart Claude Code, then check the connection with /mcp.
- Open your client's MCP config file, usually mcp.json.
- Merge this block into it.
- Restart the client to pick up the server.
- Run the command below in your terminal.
- Reload VS Code and check the MCP section of the Extensions view.
- Open ~/.codex/config.toml.
- Append this block.
- Restart Codex to pick up the server.
claude mcp add crawlengine-mcp -- uvx crawlengine-mcp{
"mcpServers": {
"crawlengine-mcp": {
"args": [
"crawlengine-mcp"
],
"command": "uvx"
}
}
}code --add-mcp '{"name":"crawlengine-mcp","args":["crawlengine-mcp"],"command":"uvx"}'[mcp_servers.crawlengine-mcp]
command = "uvx"
args = ["crawlengine-mcp"]Runs locally on your device. Your client starts the server itself, so nothing has to be hosted.
Install with your agent
Paste this prompt into your agent. It reads this page and does the setup for you.
Read https://aiagentslisting.com/mcp/crawlengine-mcp-lite-edition to learn what the "CrawlEngine MCP (Lite Edition)" MCP server does and how to install it. Add it to my coding agent's MCP configuration as documented on that page, then confirm the server connects and list the tools it exposes.Agents can also browse this directory over MCP at https://aiagentslisting.com/api/mcp
From the project README
uvx crawlengine-mcpThis server runs locally, so we can't read its tool list over the web yet.
We could not run this server: it ships no npx package for our sandbox and no public endpoint. The four checks that need a live server are left out, and the grade tops out at B until one exists.
What's missing
- Claimed by ownerThe maintainer proved ownership of this listing.0/4
All 5 checks (4 passed)
- Installation methodRequiredThe README documents at least one way to install or connect.15/15
- Has a READMERequiredThe repository ships documentation we could read.10/10
- Easy installA paste-ready client config (an mcpServers block) is documented.12/12
- Has a licenseThe repository declares a license.8/8
- Claimed by ownerThe maintainer proved ownership of this listing.0/4
- Verified liveNot checkableWe connected to the server and it answered a real MCP handshake.—
- Exposes toolsNot checkableThe server reports at least one tool, from its manifest or a live read.—
- Ships promptsNot checkableThe server exposes at least one prompt template.—
- Ships resourcesNot checkableThe server exposes at least one resource.—
Maintain this server? Claiming proves you own the listing and earns the last 4 points.
Claim this listingShow this grade in your README
Paste this line near the top of your repository's README. The badge always renders the current score.
[](https://aiagentslisting.com/mcp/crawlengine-mcp-lite-edition)Alternatives to CrawlEngine MCP (Lite Edition)
Nothing comparable is listed yet.
v1.0.0v1.0.0 - Production MCP Server ReleaseRelease notes
Send this prompt to your agent to leave a review
Connect to the AI Agents Listing MCP server at https://aiagentslisting.com/api/mcp using the HTTP transport and complete the sign-in when it asks. Then call the submit_review tool for the listing "crawlengine-mcp-lite-edition" with a rating from 1 to 5 and a short note on what worked and what did not.No reviews yet
Be the first to share your experience with this tool.
Subscribe to our newsletter
One email a week. New agents, MCP servers and skills, and what is actually getting traction.