Prerequisites
Kubernetes Cluster
Connect at least one Kubernetes cluster with the Kestrel operator installed.
Slack (optional)
Receive incident notifications and ask follow-up questions in threads.
Observability (optional)
Connect Datadog or OpenTelemetry for metrics context during root cause analysis.
PagerDuty (optional)
Route incidents to your on-call team with AI-enriched context.
GitHub / GitLab (optional)
Enable causal PR detection and GitOps remediation workflows.
Knowledge Sources (optional)
Connect Confluence, Jira, Slack history, Glean, or Linear for historical incident context.
How Detection Works
The Kestrel operator runs inside your cluster and continuously monitors:- Pod status — CrashLoopBackOff, OOMKilled, ImagePullBackOff, evictions, probe failures
- Deployment rollouts — stalled rollouts, failed replica scaling, rollback events
- Node health — NotReady nodes, resource pressure, taint/toleration mismatches
- Kubernetes events — scheduling failures, volume errors, RBAC denials, container runtime errors
- Application signals — segfaults, panics, uncaught exceptions, null pointer errors, deadlocks
Root Cause Analysis
When an incident is detected, Kestrel’s AI analyzes multiple data sources to determine the root cause:- Container logs — Recent logs from the affected pod and its previous restarts
- Kubernetes events — Events attached to the pod, deployment, node, and namespace
- Resource specifications — CPU/memory requests and limits, volume mounts, environment variables, probes
- Metrics (if Datadog or OpenTelemetry is connected) — CPU usage, memory consumption, request latency, error rates
- Knowledge sources (if connected) — Historical incidents, runbooks, and tribal knowledge from Confluence, Jira, Slack, Glean, or Linear
- Investigation summary — A step-by-step account of what was analyzed and what was found
- Root cause — The identified cause of the incident with supporting evidence
- Timeline — Sequence of events leading to the incident
- Affected resources — All Kubernetes resources involved (pods, deployments, services, nodes)
- Service dependencies — Upstream and downstream services that may be impacted
Fix Generation
After identifying the root cause, Kestrel generates YAML fixes that address the issue. Fixes are generated as Kubernetes resource patches — for example, increasing memory limits for an OOMKilled pod or correcting an image tag for an ImagePullBackOff. Each fix includes:- Diff view — Side-by-side comparison of the current and proposed resource spec
- Explanation — Why this change addresses the root cause
- Apply options:
For application-level incidents (segfaults, exceptions, panics), Kestrel also generates code fixes — suggested source code changes with file paths and diffs, ready to be applied as a PR.
Causal PR Detection
When a GitHub or GitLab integration is connected, Kestrel searches for recently merged pull requests or merge requests that may have caused the incident. The AI correlates:- The affected resource (deployment, namespace, image tag)
- The incident timeline (when the issue first appeared)
- Recent merges in connected repositories (code changes, config changes, Helm value updates)
Incident Types
Infrastructure Incidents
Application Incidents
Auto-Remediation
By default, the Kestrel operator runs in read-only mode — it can detect incidents and generate fixes, but cannot apply changes to your cluster. To enable automatic or one-click fix application:- Navigate to Clusters in the Kestrel dashboard
- Select the cluster you want to configure
- Enable Safe Apply to allow one-click fix application from the dashboard
- Optionally enable Auto-Remediation to let Kestrel automatically apply approved fix types without manual intervention
Auto-remediation policies are configured per cluster. You can define which incident types and severities are eligible for automatic remediation.
Custom Workflows
Build custom incident response workflows that trigger when Kubernetes incidents are detected. For example:- Notify a specific Slack channel when a critical OOMKilled incident occurs in the
productionnamespace - Create a Jira ticket for every incident in a specific cluster
- Automatically restart a deployment after a transient CrashLoopBackOff
- Escalate to PagerDuty only if the incident persists for more than 5 minutes
Next Steps
- Connect a Kubernetes cluster to start detecting incidents
- Set up Slack notifications for real-time alerts
- Connect GitHub or GitLab for causal PR detection and GitOps remediation
- Build custom workflows for automated incident response
- Configure PagerDuty for on-call routing with AI context