Kubernetes Incident Response

Kestrel monitors your Kubernetes clusters 24/7, automatically detecting incidents as they occur. When something goes wrong — a pod enters CrashLoopBackOff, a deployment rollout stalls, a node becomes unresponsive — Kestrel’s AI runs root cause analysis, generates YAML fixes, identifies causal PRs, and creates postmortems.

​

Prerequisites

​

How Detection Works

The Kestrel operator runs inside your cluster and continuously monitors:

• Pod status — CrashLoopBackOff, OOMKilled, ImagePullBackOff, evictions, probe failures
• Deployment rollouts — stalled rollouts, failed replica scaling, rollback events
• Node health — NotReady nodes, resource pressure, taint/toleration mismatches
• Kubernetes events — scheduling failures, volume errors, RBAC denials, container runtime errors
• Application signals — segfaults, panics, uncaught exceptions, null pointer errors, deadlocks

When the operator detects an abnormal condition, it creates an incident and immediately begins AI-powered investigation.

​

Root Cause Analysis

When an incident is detected, Kestrel’s AI analyzes multiple data sources to determine the root cause:

1. Container logs — Recent logs from the affected pod and its previous restarts
2. Kubernetes events — Events attached to the pod, deployment, node, and namespace
3. Resource specifications — CPU/memory requests and limits, volume mounts, environment variables, probes
4. Metrics (if Datadog or OpenTelemetry is connected) — CPU usage, memory consumption, request latency, error rates
5. Knowledge sources (if connected) — Historical incidents, runbooks, and tribal knowledge from Confluence, Jira, Slack, Glean, or Linear

The AI produces:

• Investigation summary — A step-by-step account of what was analyzed and what was found
• Root cause — The identified cause of the incident with supporting evidence
• Timeline — Sequence of events leading to the incident
• Affected resources — All Kubernetes resources involved (pods, deployments, services, nodes)
• Service dependencies — Upstream and downstream services that may be impacted

​

Fix Generation

After identifying the root cause, Kestrel generates YAML fixes that address the issue. Fixes are generated as Kubernetes resource patches — for example, increasing memory limits for an OOMKilled pod or correcting an image tag for an ImagePullBackOff. Each fix includes:

• Diff view — Side-by-side comparison of the current and proposed resource spec
• Explanation — Why this change addresses the root cause
• Apply options:

For application-level incidents (segfaults, exceptions, panics), Kestrel also generates code fixes — suggested source code changes with file paths and diffs, ready to be applied as a PR.

​

Causal PR Detection

When a GitHub or GitLab integration is connected, Kestrel searches for recently merged pull requests or merge requests that may have caused the incident. The AI correlates:

• The affected resource (deployment, namespace, image tag)
• The incident timeline (when the issue first appeared)
• Recent merges in connected repositories (code changes, config changes, Helm value updates)

If a likely causal PR is found, the incident detail page links directly to it with an explanation of why Kestrel believes it contributed to the issue.

​

Incident Types

​

Infrastructure Incidents

​

Application Incidents

​

Auto-Remediation

By default, the Kestrel operator runs in read-only mode — it can detect incidents and generate fixes, but cannot apply changes to your cluster. To enable automatic or one-click fix application:

1. Navigate to Clusters in the Kestrel dashboard
2. Select the cluster you want to configure
3. Enable Safe Apply to allow one-click fix application from the dashboard
4. Optionally enable Auto-Remediation to let Kestrel automatically apply approved fix types without manual intervention

​

Custom Workflows

Build custom incident response workflows that trigger when Kubernetes incidents are detected. For example:

• Notify a specific Slack channel when a critical OOMKilled incident occurs in the production namespace
• Create a Jira ticket for every incident in a specific cluster
• Automatically restart a deployment after a transient CrashLoopBackOff
• Escalate to PagerDuty only if the incident persists for more than 5 minutes

See Workflows to build custom Kubernetes incident response automation.

​

Next Steps

• Connect a Kubernetes cluster to start detecting incidents
• Set up Slack notifications for real-time alerts
• Connect GitHub or GitLab for causal PR detection and GitOps remediation
• Build custom workflows for automated incident response
• Configure PagerDuty for on-call routing with AI context