io.github.kubently/kubently

Kubently

Kubently - Troubleshooting Kubernetes Agentically

Overview

Kubently (Kubernetes + Agentically) is a free, self-hosted, vendor-neutral multi-cluster Kubernetes troubleshooter. Ask one question, get AI-diagnosed answers from every cluster in your fleet in parallel — including clusters you can't reach directly: executors dial outbound to the central API, so there's no inbound ingress, no shared kubeconfig, and no per-cluster credentials to distribute.

Agents collaborate over the A2A (Agent-to-Agent) protocol, and any MCP client (Claude Code, Cursor, Claude Desktop) can use Kubently as a tool out of the box.

Key Features

• Multi-Cluster Fleet Troubleshooting: One question fans out across all registered clusters in parallel
• Outbound-Dial Executors: Reach clusters behind firewalls/NAT — no inbound ingress, no shared kubeconfig
• Natural Language Interface: Conversational Kubernetes troubleshooting and debugging
• Comprehensive Analysis: Automated issue detection, root cause analysis, and solution recommendations
• Multi-LLM Support: Compatible with Google Gemini, OpenAI, Anthropic, and other providers
• A2A Protocol: Industry-standard agent-to-agent communication for complex workflows
• MCP Server: Optional Model Context Protocol endpoint so MCP clients (Claude Desktop, Cursor, custom agents) get direct tool access
• Security-First: API key authentication, OAuth/OIDC support, and TLS with cert-manager
• Persistent Sessions: Redis-backed conversation history and context management
• Extensive Tool Suite: kubectl integration, log analysis, resource inspection, and more

Quick Start

For Users: Get Started in 5 Minutes

Point kubectl at any cluster (kind, minikube, or real) and run:

npm install -g @kubently/cli
kubently install

That's it. The CLI installs Kubently via Helm, wires up secrets and the executor, port-forwards the API, and drops you into a debug chat:

kubently> why is my nginx pod crashlooping?

You'll need an LLM API key (Anthropic, OpenAI, or Google) — the installer prompts for it, or reads ANTHROPIC_API_KEY / OPENAI_API_KEY / GOOGLE_API_KEY from your environment. Use --provider to pick the LLM, --chart ./deployment/helm/kubently to install from a local checkout, and kubently install --help for everything else.

Use from Claude Code / Cursor (MCP)

Already ran kubently install? Add Kubently to Claude Code:

claude mcp add kubently -- kubently mcp

Or connect directly over HTTP (no bridge process):

claude mcp add --transport http kubently http://localhost:8080/mcp/ \
  --header "X-API-Key: <your-api-key>"

Then ask Claude things like "use kubently to figure out why payments pods are crashlooping". Any MCP client works — see docs/MCP.md for Cursor and generic configuration.

Proactive diagnosis (Alertmanager → Slack)

Set api.env.SLACK_WEBHOOK_URL to a Slack incoming-webhook URL and point Alertmanager at Kubently:

receivers:
  - name: kubently
    webhook_configs:
      - url: https://<your-kubently-host>/webhooks/alertmanager
        http_config:
          http_headers:            # Alertmanager >= 0.28
            X-API-Key:
              secrets: ["<your-api-key>"]

Each firing alert is diagnosed by the agent and the result is posted to Slack — the bot often explains the root cause before you've opened your laptop.

Scheduled fleet health digest

Alerts are reactive. A digest sweeps every registered cluster on a schedule and posts one summary to the same Slack webhook — healthy clusters collapse to a single line, so what's left is what needs you.

fleetReport:
  enabled: true
  schedule: "0 13 * * 1-5"   # weekday mornings

Preview it before you schedule it — dry_run returns the digest and posts nothing:

curl -X POST https://<your-kubently-host>/webhooks/fleet-report \
  -H "X-API-Key: <your-api-key>" -H 'Content-Type: application/json' \
  -d '{"dry_run": true}'

The digest question is yours to change. Pass query in that request to try one immediately, then keep the wording you like via fleetReport.query in values:

fleetReport:
  query: |-
    Check every cluster for pods restarting more than 5 times and for PVCs above
    85% usage. One line per healthy cluster. No preamble.

To run the real scheduled path once — image, secrets, in-cluster URL and all:

kubectl create job --from=cronjob/kubently-fleet-report fleet-report-test -n kubently

Deployment verification (did that deploy actually work?)

Tell Kubently what you just deployed and it watches the rollout settle, then runs a real investigation — pods ready? events clean? errors in the new logs? metrics regressed vs the pre-deploy window (when Prometheus is configured)? — and posts a PASS/FAIL verdict with the evidence to Slack. Wire it into the last step of your CI pipeline:

curl -X POST https://<your-kubently-host>/webhooks/verify-deployment \
  -H "X-API-Key: <your-api-key>" -H 'Content-Type: application/json' \
  -d '{"cluster": "prod-east", "namespace": "shop",
       "workload": "deploy/checkout-api", "context": "v1.42.0"}'

Add "dry_run": true to get the verdict back synchronously without posting. No CI access? Label the workload instead — kubently.io/verify=enabled — and enable verifyDeployment.watch in values: Kubently notices every generation change and verifies the rollout unprompted.

Scheduled checks (your recurring questions, on cron)

The digest asks one broad question. Scheduled checks let you ask your questions on their schedules — each check is a named prompt with a cron schedule and optional target clusters, run by the agent and posted to Slack:

scheduledChecks:
  enabled: true
  checks:
    - name: cert-expiry
      schedule: "0 8 * * 1"        # Monday mornings
      prompt: |-
        Check TLS secrets for certificates expiring within 21 days.
        List each as namespace/name with days remaining.
    - name: pvc-pressure
      schedule: "0 */6 * * *"
      clusters: [prod-east]
      prompt: Find PersistentVolumeClaims above 85% usage.

A passing check posts nothing — silence means green (set notifyOnPass: true to hear about passes too). Failures always post, evidence included. Iterate on a check without waiting for cron:

curl -X POST https://<your-kubently-host>/webhooks/scheduled-check \
  -H "X-API-Key: <your-api-key>" -H 'Content-Type: application/json' \
  -d '{"check": "cert-expiry", "dry_run": true}'

📖 See QUICK_START.md for full quick-start guide

📚 See GETTING_STARTED.md for production deployment

For Developers: Local Testing

# Deploy to a local kind cluster (builds images from HEAD)
ANTHROPIC_API_KEY=sk-... ./deployment/scripts/kind-e2e.sh

# Run comprehensive test suite
./test-automation/run_tests.sh test-and-analyze --api-key test-api-key

📖 See CLAUDE.md for development guidelines

Configuration

LLM Providers

Pick a provider with LLM_PROVIDER and supply that provider's key. There is no default provider — the agent refuses to start without LLM_PROVIDER. For local development with deployment/docker-compose.yaml, put both in .env (see deployment/.env.example):

# Anthropic
LLM_PROVIDER=anthropic-claude
ANTHROPIC_API_KEY=your-anthropic-api-key

# OpenAI (also matches Azure and OpenAI-compatible endpoints)
LLM_PROVIDER=openai
OPENAI_API_KEY=your-openai-api-key

# Google Gemini
LLM_PROVIDER=google-gemini
GOOGLE_API_KEY=your-gemini-api-key

In Kubernetes the keys come from the kubently-llm-secrets secret and LLM_PROVIDER goes under api.env.

Helm Deployment

Customize deployment using Helm values:

Kubently ships as a single chart. Its components are switched on and off with api.enabled, redis.enabled and executor.enabled — an executor-only install on a remote cluster is the same chart with the first two disabled.

# From a checkout
helm install kubently ./deployment/helm/kubently -n kubently \
  -f deployment/helm/test-values.yaml

# Or from the published chart repository
helm repo add kubently https://kubently.github.io/kubently
helm install kubently kubently/kubently -n kubently -f my-values.yaml

LLM_PROVIDER is required and has no chart default — set it under api.env (anthropic-claude, openai, or google-gemini). See ENVIRONMENT_VARIABLES.md for the full configuration surface, and GETTING_STARTED.md for the production walkthrough.

Operator Runbooks

Feed your organization's tribal knowledge into investigations. Runbooks are hand-written markdown files with lightweight frontmatter; when an investigation (a chat question, an Alertmanager alert, or an A2A call) matches a runbook's criteria, the agent receives it as "the operator's runbook for this situation", follows it where applicable, notes deviations, and cites it by name in the diagnosis.

A worked example:

---
name: Payments CrashLoopBackOff
match:
  alerts: ["KubePodCrashLooping", "PaymentsPod*"]   # alert-name globs
  namespaces: ["payments", "payments-*"]            # namespace selectors
  workloads: ["payment-api*"]                       # matches derived pod names too
  topics: ["crashloop", "OOMKilled", "payment service"]  # free-text tags
---
1. Check recent deploys first: payment-api ships through ArgoCD, and 90% of
   crashloops here follow a bad config sync.
2. OOMKilled almost always means the JVM heap flag drifted from the container
   memory limit — compare `-Xmx` against `resources.limits.memory` before
   blaming traffic.
3. If the DB connection pool is exhausted, do NOT restart the pods; escalate
   to #payments-oncall (restarts thundering-herd the database).

Deploy runbooks as Helm values (they become a ConfigMap mounted into the API pod; edits go live without a pod restart):

# production-values.yaml
runbooks:
  payments-crashloop.md: |
    ---
    name: Payments CrashLoopBackOff
    match:
      alerts: ["KubePodCrashLooping"]
      namespaces: ["payments"]
    ---
    1. Check recent deploys first ...

Matching is scored: an alert-name hit outranks namespace/workload selector hits, which outrank topic hits. The best match is injected first, and the total injected size is capped (KUBENTLY_RUNBOOKS_MAX_CHARS, default 8000 characters) — one complete, best-matching runbook beats fragments of many. Outside Helm, point KUBENTLY_RUNBOOKS_DIR at any directory of .md files.

Incident History

Past diagnoses become searchable institutional memory. Whenever an investigation concludes with a root cause, Kubently stores a compact record — date, cluster, resources involved, symptom keywords, the root-cause one-liner, and the resolution when one was stated — in Redis, isolated per authenticated caller (the same namespace boundary as conversation memory, so in multi-tenant deployments one tenant's incidents are never visible to another).

The history is used two ways:

• The agent's search_past_incidents tool answers "have we seen this before?" — keyword search over resources, clusters, symptoms and root-cause text, newest first.
• Auto-surface: when a new investigation strongly matches a past incident (same resources/symptoms/cluster), a one-line SIMILAR PAST INCIDENT (date): <root cause> note is injected into context — framed as something to verify against fresh evidence, never to assume. When a past incident materially informs the diagnosis, the RCA cites it ("same root cause as the 2026-07-03 incident").

This is retrieval over stored summaries, not a learning system: records are plain data with a TTL (default 90 days, KUBENTLY_INCIDENT_TTL_SECONDS) and a per-tenant cap (default 200, KUBENTLY_INCIDENT_MAX_PER_NAMESPACE, oldest evicted). The feature is on by default; set KUBENTLY_INCIDENT_HISTORY=false (Helm: under api.env) to disable both recording and retrieval.

Architecture

• API Server: FastAPI-based REST API for cluster management and authentication
• A2A Server: Implements A2A protocol with LangGraph for workflow orchestration
• Test Automation: Comprehensive testing framework with 20+ Kubernetes scenarios
• CLI Tools: Modern Node.js CLI for interactive debugging

Agent Toolset

The diagnostic agent investigates with a small set of read-only tools:

• list_clusters — enumerate registered clusters
• execute_kubectl — read-only kubectl against one cluster (whitelist-enforced on the executor)
• execute_kubectl_multi — one read-only kubectl command fanned out across many clusters
• get_recent_changes — "what changed?" timeline for a workload or namespace: rollouts (ReplicaSet revisions + change-causes), Helm release history (opt-in: changeCorrelation.helmHistory.enabled), ArgoCD sync history (optional: changeCorrelation.argocd.url), and Normal+Warning events — correlated against first-error timestamps in the RCA
• get_events_for_resource — chronological events for a resource and its children (deployment → replicasets → pods)
• search_pod_logs — structured log search across every pod/container matching a label selector (substring or regex, time bounds, previous-container support). Logs are filtered on the cluster's executor so only matching lines — capped, with explicit truncation notes — come back
• query_loki (optional) — LogQL range queries against a cluster's Loki for aggregated/historical log search, including logs from pods that have restarted or been deleted. Enabled by setting loki.url in Helm values (unset by default); queries execute on each cluster's executor through the same outbound channel as kubectl commands
• query_prometheus (optional) — instant and range PromQL queries for latency, saturation, OOM-trend and restarts-over-time evidence. Enabled by setting prometheus.url in Helm values (unset by default); queries execute on each cluster's executor through the same outbound channel as kubectl commands
• search_past_incidents — keyword search over this deployment's incident history (see below). On by default when Redis is available; disable with KUBENTLY_INCIDENT_HISTORY=false
• get_manifest_file (optional) — read-only fetch of a file from the configured GitOps manifests repo, so proposed fixes are diffed against the real manifest instead of a hallucinated one. Enabled with gitRemediation in Helm values (off by default)
• propose_fix_pr (optional) — proposes a high-confidence manifest fix as a pull request against the configured GitOps manifests repo (GitHub or GitLab): branch → commit → PR with the investigation evidence in a body clearly marked machine-proposed. The agent never merges — a human reviews and merges, and your GitOps controller applies. Size-capped (files/changed lines), token never enters model context, cluster access stays read-only. See GitOps PR Remediation
• query_cloud_logs, query_cloud_metrics, get_recent_cloud_changes (optional) — read-only cloud telemetry for a cluster: CloudWatch Logs Insights / CloudWatch metrics / CloudTrail on AWS, Cloud Logging / Cloud Monitoring / GKE audit logs on GCP. The executor answers them using the workload identity you grant its ServiceAccount (EKS Pod Identity, IRSA, or GKE Workload Identity) — no cloud key is ever stored by Kubently, and revoking the IAM role kills the capability instantly. Enabled per cluster with executor.cloud.enabled in Helm values (off by default); each call re-checks that the target cluster's executor actually reports an identity. Operations are additionally limited by a code-level allowlist. See Cloud Telemetry
• mcp_<server>_* (optional) — tools from external MCP servers (streamable HTTP, e.g. Grafana Cloud's or Datadog's remote MCP) configured via mcpServers in Helm values (unset by default). Tool names are prefixed with the server name to avoid collisions; results are treated as untrusted input (framed and size-capped) and credentials stay in Kubernetes secrets. Connect read-scoped servers/credentials only — Kubently cannot enforce read-only semantics on a remote server's tools. See docs/MCP_CLIENT_TOOLS.md

Documentation

Getting Started

• Quick Start Guide - Get running in 5 minutes
• Getting Started - Complete setup for production use
• Deployment Guide - Detailed deployment options and configuration

Usage & Operations

• CLI Admin Guide - Managing clusters and executors
• Test Queries - Example API requests and A2A protocol usage
• MCP Connect Guide - Connect MCP clients (Claude Desktop, Cursor, custom agents)
• Environment Variables - Configuration reference
• GitOps PR Remediation - Agent-proposed fix PRs (human-merged, default off)
• Cloud Telemetry - Read-only CloudWatch / Cloud Logging access via workload identity (default off)
• External MCP Tools - Mounting third-party MCP servers into the agent, including per-request injection

Architecture & Development

• Architecture - System design and components
• A2A Protocol Spec - Official protocol documentation
• Development Guide - Guidelines for contributors

Troubleshooting

• Getting Started - Common Issues - Troubleshooting guide

Contributing

See CLAUDE.md for development guidelines and contribution instructions.

Maintainer

Kubently Team - hello@kubently.io

License

Apache 2.0 License - See LICENSE file for details