Scrape, crawl, and map websites to Markdown or JSON via local CLI.
Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.
You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.
It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.
Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.
| Feature | Description |
|---|---|
| Structured extraction | Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, response headers |
| Markdown conversion | Clean Markdown with citations, document structure, and fit-content mode |
| Concurrent crawling | Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency |
| 14 language bindings | Rust, Python, Node.js, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly |
| Smart filtering | BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, sitemap discovery |
| Browser rendering | Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass |
| Batch & streaming | Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams |
| SSRF-safe by default | Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or CrawlConfig |
| Auth & rate limiting | HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling |
| MCP server & REST API | Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec |
Precompiled binaries for glibc Linux (x86_64/aarch64), macOS, and Windows across every binding. On musl Linux (Alpine) coverage is deliberately partial — the CLI, Docker image, Rust, Ruby, Java, and Elixir work there; the Node, Python, Go, PHP, and C# bindings do not ship musl artifacts. See the platform support matrix for the full table and the Alpine workarounds.
Available on Maven Central as io.xberg.crawlberg:crawlberg. See Java README for the dependency snippet and current version.
Add {:crawlberg, "~> 0.3"} to your mix.exs dependencies. See Elixir README for full documentation.
Available on Maven Central as io.xberg.crawlberg.android:crawlberg-android. See Kotlin README for the dependency snippet and current version.
Add via Swift Package Manager. See Swift README for full documentation.
See Zig README for installation and usage.
C header + shared library from GitHub Releases. See FFI crate for full documentation.
cargo install crawlberg-cli
# Or install the prebuilt binary via cargo-binstall:
cargo binstall crawlberg-cli
brew install xberg-io/tap/crawlberg
Windows users can install the same binary through Scoop:
scoop bucket add xberg https://github.com/xberg-io/scoop-bucket
scoop install crawlberg
See CLI README for full documentation.
Install the Crawlberg plugin from xberg-io/crawlberg. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, — select your runtime below.
The bundle targets Claude Code, Codex, Cursor, Gemini CLI, Kimi, OpenCode, Factory Droid, Hermes, Pi, Agent Plugins and GitHub Copilot CLI. Read the plugin guide for runtime resources, CLI prerequisites and configuration. MCP-capable adapters expose the product's tools; Hermes uses a local stdio bridge and explicitly selected skills. Pi loads skills and prompt templates and requires the product CLI for executable operations.
Install only the public Agent Skills, without a plugin or MCP registration, using the skills CLI:
npx skills add xberg-io/crawlberg --list
npx skills add xberg-io/crawlberg
Choose the skills and target agents interactively. Installing instructions does not install the product CLI or configure provider credentials.
/plugin marketplace add xberg-io/crawlberg
/plugin install crawlberg@crawlberg
/plugins add https://github.com/xberg-io/crawlberg
Then search for crawlberg and select Install Plugin.
Settings → Plugins → Add from URL → https://github.com/xberg-io/crawlberg, then select crawlberg.
From a checkout of this repository:
gemini extensions install ./plugin
The manifest is inside plugin/. See the Gemini extension reference
for local installation and updates.
droid plugin marketplace add https://github.com/xberg-io/crawlberg
droid plugin install crawlberg@crawlberg
copilot plugin marketplace add https://github.com/xberg-io/crawlberg
copilot plugin install crawlberg@crawlberg
From a checkout of this repository:
pi install ./plugin
After the next plugin release, install the same package used by OpenCode:
pi install npm:@xberg-io/opencode-crawlberg
Pi loads the bundled skills and prompt templates. Use /crawlberg-scrape or /crawlberg-crawl with the input as arguments. Install the crawlberg CLI separately for shell access. This package does not register MCP servers, hooks, or native Pi extensions.
Add the package to opencode.json:
{
"$schema": "https://opencode.ai/config.json",
"plugin": ["@xberg-io/opencode-crawlberg"]
}
Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at docs.crawlberg.xberg.io.
Contributions are welcome! See our Contributing Guide.
Source-derived launch command. Check the maintainer’s required arguments and credentials before running:
docker run -i --rm ghcr.io/kreuzberg-dev/kreuzcrawl:0.3.0Merge this template into ~/Library/Application Support/Claude/claude_desktop_config.json. Keep existing servers. Add any arguments, credentials, and permissions required by the maintainer; this template has not been install-tested.
{
"mcpServers": {
"io-github-kreuzberg-dev-kreuzcrawl": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/kreuzberg-dev/kreuzcrawl:0.3.0"
]
}
}
}Restart Claude Desktop completely for changes to take effect. Confirm the server appears connected in the client’s tool list, then try a read-only example from its documentation.
Claude Desktop setup referenceghcr.io/kreuzberg-dev/kreuzcrawl:0.3.0dockerKreuzcrawl works with any MCP-compatible client. Copy the config snippet from the Configuration section above and add it to the file shown for your client, then restart the application.
~/Library/Application Support/Claude/claude_desktop_config.jsonRestart Claude Desktop completely for changes to take effect.~/.cursor/mcp.jsonRestart Cursor for changes to take effect..vscode/mcp.jsonReload VS Code window for changes to take effect.~/.codeium/windsurf/mcp_config.jsonRestart Windsurf for changes to take effect..mcp.jsonSave at the project root, then start Claude Code in that project and review the MCP server approval prompt. Keep real credentials out of shared files.