GenPark Bloom Filter URL Dedup Crawler
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
GenPark Bloom Filter URL Dedup Crawler
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler ingestion.
Part of the GenPark AI Agent Skills Ecosystem.
🌟 Key Features
- 100% Python Standard Library: Zero third-party runtime dependencies.
- MCP Native: Plugs directly into Claude Desktop, Cursor, LobeChat, and autonomous agent DAGs.
- Production Resilience: Strict schema conformance, zero data loss, enterprise hardened.
🚀 Installation & Usage
Run as an MCP server:
python mcp_server.py
Use directly as a library:
from client import *
See example_usage.py for comprehensive integration demonstrations.
📄 License
MIT License. Maintained by GenPark AI.
Setup from the maintainer
This listing does not have a supported local package template. Use the maintainer’s documentation for its hosted endpoint, authentication, and client-specific setup. No install command has been inferred.
Package
https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpbotherCompatible MCP Clients
GenPark Bloom Filter URL Dedup Crawler works with any MCP-compatible client. Copy the config snippet from the Configuration section above and add it to the file shown for your client, then restart the application.
- Claude Desktop
~/Library/Application Support/Claude/claude_desktop_config.jsonRestart Claude Desktop completely for changes to take effect. - Cursor
~/.cursor/mcp.jsonRestart Cursor for changes to take effect. - VS Code
.vscode/mcp.jsonReload VS Code window for changes to take effect. - Windsurf
~/.codeium/windsurf/mcp_config.jsonRestart Windsurf for changes to take effect. - Claude Code
.mcp.jsonSave at the project root, then start Claude Code in that project and review the MCP server approval prompt. Keep real credentials out of shared files.