# Cloud Investigation Source: https://docs.usekestrel.ai/copilot/cloud Query AWS and OCI resources, security configurations, CloudTrail events, and network traffic The copilot can answer questions about your connected AWS and OCI accounts — resource inventory, security configurations, CloudTrail audit events, CloudWatch metrics, VPC Flow Logs, and more. The examples on this page illustrate common query patterns, but you can ask anything about your cloud accounts in natural language. ## Prerequisites Connect one or more AWS accounts for EC2, S3, RDS, IAM, VPC, Lambda, CloudTrail, CloudWatch, and VPC Flow Log queries. Connect Oracle Cloud accounts for Compute, Object Storage, VCN, IAM, and Audit log queries. ## Targeting Cloud Accounts Use `@cloud:connection-name` to direct your query to a specific AWS or OCI account. You can target multiple accounts in a single query for cross-account comparisons. ``` @cloud:prod-aws show me all publicly accessible S3 buckets ``` ``` @cloud:prod-aws @cloud:staging-aws compare the IAM policies attached to the EKS node group roles ``` If you have only one cloud account connected, the copilot targets it automatically. Find your connection names on the **Cloud Connections** page in the Kestrel dashboard. The name matches whatever you set when connecting the account. ## Resource Inspection Query any cloud resource — instances, storage, databases, serverless functions, networking, and more. ``` @cloud:prod-aws Show me all EC2 instances in us-east-1 and us-west-2, grouped by instance type. Include running state and launch time. ``` ``` @cloud:prod-aws Which S3 buckets don't have versioning enabled? Flag any that also lack server-side encryption. ``` ``` @cloud:prod-aws List all RDS instances that aren't using Multi-AZ deployment. Include their storage size, engine version, and whether automated backups are enabled. ``` ``` @cloud:prod-aws Show me all Lambda functions that haven't been invoked in the last 30 days. What's the total allocated memory across them? ``` ``` @cloud:prod-aws What ECS services are running in the production cluster? Show me the task definition, desired count, and running count for each. ``` ``` @cloud:prod-aws Which ELBs have no healthy targets registered? Include the target group health check configuration. ``` ## Security Configuration Inspect IAM policies, security groups, encryption settings, and access controls. ``` @cloud:prod-aws Which security groups allow inbound SSH (port 22) from 0.0.0.0/0? Show me the group name, VPC, and what resources are using each group. ``` ``` @cloud:prod-aws List all IAM users with console access that don't have MFA enabled. ``` ``` @cloud:prod-aws Show me all IAM roles that have AdministratorAccess or full S3 access policies attached. Which services or users assume these roles? ``` ``` @cloud:prod-aws Which KMS keys are scheduled for deletion? Are any of them still being used by S3 buckets or RDS instances? ``` ``` @cloud:prod-aws Are there any S3 bucket policies that grant cross-account access? Show me the principal ARNs and what permissions they have. ``` ``` @cloud:prod-aws Which EC2 instances have public IP addresses and security groups that allow inbound traffic on non-standard ports? ``` ## CloudTrail and Audit Events Query who did what, when, and from where. ``` @cloud:prod-aws Who modified the IAM policy for the deploy-bot role in the last 48 hours? Show the full event details including source IP and user agent. ``` ``` @cloud:prod-aws Show me all console logins in the last 7 days. Flag any that came from unusual IP addresses or happened outside business hours. ``` ``` @cloud:prod-aws What API calls did user david@acme.com make yesterday? Focus on any destructive actions like Delete* or Terminate*. ``` ``` @cloud:prod-aws Were there any security group changes in the last 24 hours? Show me the before and after rules for each change. ``` ``` @cloud:prod-aws Which IAM roles were created or modified in the last week? Show me who made the changes and what permissions were granted. ``` ## CloudWatch Metrics Query performance and health metrics from CloudWatch. ``` @cloud:prod-aws What's the CPU utilization for the prod-analytics RDS instance over the last 6 hours? Is it approaching any performance thresholds? ``` ``` @cloud:prod-aws Show me the request count and error rate for the prod-api ALB over the last 24 hours. Were there any spikes? ``` ``` @cloud:prod-aws Which EC2 instances in us-east-1 have averaged less than 10% CPU utilization over the last 30 days? ``` ``` @cloud:prod-aws What's the current queue depth for the order-processing SQS queue? How many messages are in the dead letter queue? ``` ## Network Traffic Query VPC Flow Logs for network traffic analysis. ``` @cloud:prod-aws Show me all rejected traffic in the production VPC over the last hour. Group by source and destination. ``` ``` @cloud:prod-aws Is there any traffic from the application subnet to the internet that isn't going through the NAT gateway? ``` ``` @cloud:prod-aws What's the traffic volume between the production VPC and the staging VPC over the last 24 hours? Break it down by port. ``` ``` @cloud:prod-aws Are any EC2 instances sending traffic to known bad IP ranges or unusual external destinations? ``` ## OCI-Specific Queries Query Oracle Cloud Infrastructure resources and audit logs. ``` @cloud:prod-oci List all compute instances in the production compartment. Include shape, state, and VNIC details. ``` ``` @cloud:prod-oci Which object storage buckets in the production tenancy are publicly accessible? ``` ``` @cloud:prod-oci Show me the audit events for the networking compartment in the last 48 hours. Were there any VCN or security list changes? ``` ``` @cloud:prod-oci What IAM policies apply to the deployment group? Are there any overly permissive statements? ``` ## Cross-Account Comparison Compare configurations and resources across multiple cloud accounts. ``` @cloud:prod-aws @cloud:staging-aws Compare the security group rules for the EKS node groups. Are there any rules in staging that aren't in production? ``` ``` @cloud:prod-aws @cloud:staging-aws Which RDS instances exist in production but not in staging? Are there any configuration differences for the ones that exist in both? ``` ``` @cloud:prod-aws @cloud:staging-aws Compare the Lambda function configurations for the payment-processor functions across both accounts. Flag any differences in memory, timeout, or environment variables. ``` ``` @cloud:prod-aws @cloud:dev-aws Are the same IAM policies attached to the CI/CD role in both accounts? ``` ## Example Conversation **You:** > `@cloud:prod-aws` We got an alert about unauthorized access. Show me all CloudTrail events from unusual source IPs in the last 12 hours. **Kestrel:** > I found 3 API calls from IP addresses not in your organization's known CIDR ranges: > > 1. `AssumeRole` on `arn:aws:iam::123456789012:role/deploy-bot` from `198.51.100.42` at 03:14 UTC — user agent `aws-cli/2.15.0` > 2. `ListBuckets` from the same IP at 03:16 UTC using the assumed `deploy-bot` session > 3. `GetObject` on `s3://prod-config-bucket/database.yml` at 03:17 UTC from the same session > > The `deploy-bot` role was assumed using access key `AKIA...3F7Q` which belongs to IAM user `ci-pipeline`. This key hasn't been rotated in 214 days. **You:** > What permissions does that deploy-bot role have? And is the access key compromised? **Kestrel:** > The `deploy-bot` role has the following attached policies: > > * `AmazonS3FullAccess` (AWS managed) > * `custom-deploy-policy` (inline) — grants `ec2:*`, `ecs:*`, and `lambda:*` on all resources > > This is significantly over-permissioned for a CI/CD role. Regarding the access key — the source IP `198.51.100.42` geolocates to a datacenter in Ashburn, VA (AWS us-east-1 region), which could be a legitimate CI runner. However, previous API calls from `ci-pipeline` all originated from `10.0.x.x` (your VPC). I'd recommend rotating the key immediately and scoping down the role permissions. **You:** > Generate a Terraform change to scope down that role to just the S3 and ECS permissions it actually needs. **Kestrel:** > Based on the CloudTrail history for this role over the last 90 days, the `deploy-bot` role actually only uses these actions: `s3:GetObject`, `s3:PutObject` on the `prod-config-bucket` and `prod-artifacts` buckets, and `ecs:UpdateService`, `ecs:DescribeServices` on the `production` cluster. Here's a least-privilege policy: > > *(displays Terraform diff replacing the over-permissioned policies with a scoped-down custom policy)* ## Next Steps * [Kubernetes Investigation](/copilot/kubernetes) — query pods, deployments, events, and metrics * [Code Investigation](/copilot/code) — search and trace code changes across repositories * [Cost Analysis](/copilot/costs) — analyze AWS spend patterns and anomalies * [Configuration Changes](/copilot/config-changes) — generate and apply Terraform and YAML changes # Code Investigation Source: https://docs.usekestrel.ai/copilot/code Search, read, and analyze code across connected GitHub and GitLab repositories The copilot can search files, read source code, trace change history, and look up infrastructure-as-code definitions across your connected GitHub and GitLab repositories. Use it to find where things are configured, who changed what, and how infrastructure definitions map to running resources. ## Prerequisites Connect GitHub to search repositories, read files, and query PR history. Connect GitLab to search repositories, read files, and query MR history. Code investigation queries don't require the `@` targeting syntax. The copilot searches across all connected repositories automatically. Mention a specific repository name in your query to narrow the search. ## File Search and Reading Find and read files across your connected repositories. ``` Find the Dockerfile for the payments-api service ``` ``` Show me the Kubernetes deployment manifest for the auth-service in the infrastructure repo ``` ``` Where is the database connection string configured in acme/checkout-service? ``` ``` Find all files named values.yaml across our connected repos ``` ``` Show me the contents of acme/infrastructure/terraform/modules/eks/main.tf ``` ## Code Search Search for specific functions, configurations, patterns, and definitions across repositories. ``` Search for all usages of the STRIPE_SECRET_KEY environment variable across our repos ``` ``` Where is the rate limiting middleware defined in the API gateway service? ``` ``` Find all Golang functions that call the database.Query method in acme/payments-service ``` ``` Search for any hardcoded AWS account IDs across all connected repositories ``` ``` Where is the MAX_CONNECTIONS environment variable set? Check both application code and Kubernetes manifests. ``` ## Change History Query recent PRs, commits, blame information, and diffs to understand what changed and why. ``` What PRs were merged to the main branch of acme/infrastructure in the last 48 hours? ``` ``` Show me the git blame for the resource limits in the auth-service deployment manifest. When were they last changed and what was the reasoning? ``` ``` What changed in the Dockerfile for acme/payments-service in the last week? Show me the diff. ``` ``` Who last modified the Terraform module for the production RDS instance? What was the PR description? ``` ``` Show me all commits to acme/checkout-service/src/config/ in the last 7 days. Did any of them change database or cache settings? ``` ``` What open PRs are there in acme/infrastructure that touch IAM policies or security group rules? ``` ## Infrastructure-as-Code Lookup Find and read Terraform, Helm, CloudFormation, and Pulumi definitions that manage your running infrastructure. ``` Find the Terraform code that manages our production RDS instances. What module is it in? ``` ``` Show me the Helm values file for the payments-service chart. What are the current replica count and resource limits? ``` ``` Where is the S3 bucket policy for prod-data-lake defined in Terraform? Show me the access rules. ``` ``` Find the CloudFormation template that creates our VPC and subnets. What CIDR ranges are configured? ``` ``` Search across all repos for Terraform code that references security group sg-0a1b2c3d4e5f6g7h8 ``` ``` Show me the Helm chart templates for the auth-service. What environment variables are templated in? ``` ## Cross-Repository Search Search across multiple repositories to trace configurations and dependencies. ``` Search across all connected repos for any code that references the prod-vpc VPC ID. I need to understand all resources deployed into it. ``` ``` Find all repositories that contain a reference to the payments-db.internal hostname. Which services depend on this database? ``` ``` Search for any Terraform, Kubernetes, or application code that references the elasticache-prod Redis cluster endpoint ``` ``` Which repos have GitHub Actions workflows that deploy to the production EKS cluster? ``` ## Example Conversation **You:** > The payments-api is failing with `connection refused` errors when trying to reach the orders database. Can you find how the database host is configured? **Kestrel:** > I searched across your connected repositories and found the database configuration in two places: > > 1. **Application code** — `acme/payments-service/src/config/database.ts` reads the `ORDERS_DB_HOST` environment variable, falling back to `localhost:5432` if unset. > 2. **Kubernetes manifest** — `acme/infrastructure/k8s/payments/deployment.yaml` sets `ORDERS_DB_HOST` to `orders-db.payments.svc.cluster.local:5432` via a ConfigMap reference (`payments-db-config`). > > The ConfigMap `payments-db-config` is defined in `acme/infrastructure/k8s/payments/configmap.yaml`. The current value on the `main` branch is `orders-db.payments.svc.cluster.local:5432`. **You:** > Was this ConfigMap changed recently? **Kestrel:** > Yes. PR #312 in `acme/infrastructure` was merged 3 hours ago by @alex.kim. The PR changed the `ORDERS_DB_HOST` value from `orders-db.payments.svc.cluster.local:5432` to `orders-db-v2.payments.svc.cluster.local:5432` as part of a database migration. The PR description mentions: "Point payments-api to the new PostgreSQL 16 instance." > > However, I don't see a corresponding Kubernetes Service named `orders-db-v2` in the `payments` namespace — the service may not have been created yet, which would explain the `connection refused` errors. **You:** > Show me the Terraform code for the new database instance. Is it fully provisioned? **Kestrel:** > The new RDS instance is defined in `acme/infrastructure/terraform/modules/rds/orders-db-v2.tf`. It was added in PR #308, merged last week. The Terraform config provisions a `db.r6g.large` PostgreSQL 16.2 instance in `us-east-1a` with Multi-AZ enabled. > > The Terraform output `orders_db_v2_endpoint` is referenced in the Kubernetes Service definition in `acme/infrastructure/k8s/payments/service-orders-db-v2.yaml` — but this file was added in PR #313, which is still open and hasn't been merged yet. That's why the Service doesn't exist in the cluster. ## Next Steps * [Kubernetes Investigation](/copilot/kubernetes) — query pods, deployments, events, and metrics * [Cloud Investigation](/copilot/cloud) — inspect AWS and OCI resources and security configurations * [Cost Analysis](/copilot/costs) — analyze AWS spend patterns and anomalies * [Configuration Changes](/copilot/config-changes) — generate and apply YAML and Terraform changes # Configuration Changes Source: https://docs.usekestrel.ai/copilot/config-changes Generate and apply YAML and Terraform changes through the copilot The copilot can generate Kubernetes YAML, Terraform, Helm values, and CloudFormation changes when you ask it to modify infrastructure. Every change goes through a review and approval workflow — the copilot never applies anything without your explicit consent. ## How It Works Ask the copilot to make a change: "Increase the memory limit for the payments-api to 1Gi" or "Add a network policy to block egress from the sandbox namespace." The copilot determines the target resource, fetches its current configuration, and generates the modification. The copilot presents a side-by-side diff showing exactly what will change. Each diff includes an explanation of why the change addresses your request and what the impact will be. Select one of three options: * **Safe Apply** — Apply the change directly to the cluster or cloud account through the Kestrel operator * **Create PR** — Open a pull request in your connected GitHub or GitLab repository (GitOps or IaC workflow) * **Copy** — Copy the generated configuration to your clipboard for manual application ## What You Can Change The copilot generates changes for multiple infrastructure-as-code formats depending on the target. ### Kubernetes Resources Generate YAML patches for any Kubernetes resource — deployments, services, configmaps, network policies, RBAC, HPA, and more. ``` @prod-eks-cluster Scale the checkout-api deployment to 5 replicas ``` ``` @prod-eks-cluster Increase the memory limit for the payments-api to 1Gi and add a CPU request of 250m ``` ``` @prod-eks-cluster Create a network policy that blocks all egress from the sandbox namespace except DNS ``` ``` @prod-eks-cluster Add a readiness probe to the auth-service deployment that checks /healthz on port 8080 every 10 seconds ``` ``` @prod-eks-cluster Update the HPA for order-service to scale between 3 and 20 replicas based on CPU at 70% ``` ### Terraform Generate Terraform changes for AWS, OCI, and other cloud resources. ``` @cloud:prod-aws Change the instance type for the analytics EC2 instances from m5.2xlarge to m5.large ``` ``` @cloud:prod-aws Add encryption at rest to the prod-data S3 bucket using the default KMS key ``` ``` @cloud:prod-aws Create a new security group rule that allows the application subnet to access RDS on port 5432 ``` ``` @cloud:prod-aws Scope down the deploy-bot IAM role to only the permissions it has actually used in the last 90 days ``` ### Helm Values Generate value overrides for Helm-managed releases. ``` @prod-eks-cluster Update the Helm values for the auth-service to set replicas to 3 and enable autoscaling ``` ``` @prod-eks-cluster Change the ingress annotations for the frontend Helm release to use the internal ALB class ``` ### CloudFormation Generate CloudFormation template changes for AWS resources managed through stacks. ``` @cloud:prod-aws Update the CloudFormation stack for the production VPC to add a new private subnet in us-east-1c ``` ``` @cloud:prod-aws Modify the Lambda function in the data-pipeline stack to increase the timeout to 300 seconds and memory to 1024MB ``` ## Change Review Interface Every generated change is presented in a structured review interface: | Component | Description | | ------------------ | ----------------------------------------------------------------- | | **Diff view** | Side-by-side comparison of the current and proposed configuration | | **Explanation** | Why the change was generated and what it addresses | | **Impact summary** | What resources will be affected and any dependencies | | **Apply options** | Safe Apply, Create PR, and Copy buttons | The copilot generates changes based on the live state of your resources, not just the IaC definitions. If there's drift between your running configuration and your Git repository, the copilot will flag it. ## Safety and Permissions The copilot follows strict safety rules when generating and applying changes. * **Never auto-applies** — Every change requires your explicit approval through Safe Apply, Create PR, or Copy. The copilot will not apply changes on its own. * **Safe Apply requires write permissions** — The Kestrel operator must have Safe Apply or auto-remediation enabled on the target cluster. Without write permissions, only Create PR and Copy are available. * **Create PR requires Git integration** — Opening a pull request requires a connected [GitHub](/integrations/github) or [GitLab](/integrations/gitlab) repository with write access. * **Validation before apply** — The copilot validates generated YAML and Terraform against the target schema before presenting the change. Invalid configurations are flagged before you can apply them. * **Audit trail** — Every generated change, approval, and application is logged and visible on the Configuration Changes page. Safe Apply applies changes directly to the running cluster or cloud account. Use Create PR for production changes that should go through your standard GitOps review process. ## Chat Configuration Changes Page All AI-generated configuration changes are recorded on the **Chat Configuration Changes** page, accessible from the Kestrel dashboard sidebar. This page shows: * Every change generated by the copilot, including the original request and the generated diff * The approval status — whether it was applied, opened as a PR, copied, or dismissed * Who approved or dismissed each change and when * The target resource and cluster or cloud account Use this page to review pending changes, audit what was applied, and track configuration modifications made through the copilot across your team. ## Example Conversation **You:** > `@prod-eks-cluster` The checkout-api keeps getting OOMKilled. Current memory limit is 256Mi but Datadog shows it peaks at 400Mi. Fix it. **Kestrel:** > I'll increase the memory limit for the `checkout-api` deployment. Based on the Datadog metrics showing peaks at 400Mi, I recommend setting the limit to 512Mi to provide a 25% buffer above peak usage. Here's the proposed change: > > *(displays diff changing `resources.limits.memory` from `256Mi` to `512Mi`)* > > The memory request will also be updated from `128Mi` to `256Mi` to ensure the pod gets adequate guaranteed memory on the node. Would you like to Safe Apply, Create PR, or Copy? **You:** > Create a PR for this. Also add a CPU request of 250m while you're at it — it's missing. **Kestrel:** > Updated the change to include both modifications: > > 1. Memory limit `256Mi` → `512Mi`, memory request `128Mi` → `256Mi` > 2. Added CPU request of `250m` (was unset) > > *(displays updated diff with both changes)* > > I'll open this as a PR in `acme/infrastructure` targeting the `main` branch. The change applies to `k8s/checkout/deployment.yaml`. **You:** > Go ahead. **Kestrel:** > PR #892 created in `acme/infrastructure`: [Increase checkout-api memory limit and add CPU request](https://github.com/acme/infrastructure/pull/892). The PR includes the diff and a description explaining the OOMKill context and Datadog metrics that motivated the change. ## Next Steps * [Kubernetes Investigation](/copilot/kubernetes) — query pods, deployments, events, and metrics * [Cloud Investigation](/copilot/cloud) — inspect AWS and OCI resources and security configurations * [Code Investigation](/copilot/code) — search and trace code changes across repositories * [Cost Analysis](/copilot/costs) — analyze AWS spend patterns and anomalies # Cost Analysis Source: https://docs.usekestrel.ai/copilot/costs Query AWS costs, identify spend patterns, and surface anomalies The copilot can query AWS Cost Explorer data to break down spend by service, region, and account, identify cost trends, detect anomalies, drill into resource-level costs, and forecast future spend. Use it to answer billing questions, investigate cost spikes, and find optimization opportunities without leaving the chat interface. ## Prerequisites Connect an AWS account with Cost Explorer API access enabled. The IAM role used by Kestrel needs `ce:GetCostAndUsage`, `ce:GetCostForecast`, and related Cost Explorer permissions. AWS Cost Explorer data is typically delayed by 24–48 hours. The copilot returns the most recent data available, but today's costs may not be fully reflected yet. ## Targeting Accounts Use `@cloud:connection-name` to query costs for a specific AWS account. You can target multiple accounts for cross-account comparisons. ``` @cloud:prod-aws What are our top 10 services by cost this month? ``` ``` @cloud:prod-aws @cloud:staging-aws Compare total spend across both accounts for the last 3 months. ``` If you have only one AWS account connected, the copilot targets it automatically. ## Spend Breakdown Query costs by service, region, account, or any combination. ``` @cloud:prod-aws What are our top 10 most expensive AWS services this month? Compare to last month and flag any that increased by more than 20%. ``` ``` @cloud:prod-aws Break down our EC2 costs by region for the last 30 days. Which region is the most expensive? ``` ``` @cloud:prod-aws How much are we spending on data transfer this month? Break it down by service and transfer type (inter-region, internet, etc.). ``` ``` @cloud:prod-aws What's our total RDS spend? Break it down by instance, and include the instance type and engine for each. ``` ``` @cloud:prod-aws Show me a cost breakdown for the last month grouped by the "team" cost allocation tag. Which team is spending the most? ``` ## Cost Trends Over Time Track how costs change over days, weeks, and months. ``` @cloud:prod-aws How has our total AWS spend changed month-over-month for the last 6 months? What's the growth rate? ``` ``` @cloud:prod-aws Show me the daily EC2 spend for the last 30 days. Were there any days with unusual spikes? ``` ``` @cloud:prod-aws Compare our Lambda costs between Q4 last year and Q1 this year. Did the migration to ARM functions reduce costs? ``` ``` @cloud:prod-aws What's the trend for our S3 storage costs over the last 3 months? Is it growing linearly or accelerating? ``` ## Anomaly Detection Identify unusual cost patterns and unexpected charges. ``` @cloud:prod-aws Are there any unusual cost spikes in the last 7 days? What services are driving them? ``` ``` @cloud:prod-aws Which services had the biggest percentage increase in cost compared to the previous month? Flag anything over 30% growth. ``` ``` @cloud:prod-aws Did anything unexpected happen with our NAT Gateway costs last week? They seem higher than usual. ``` ``` @cloud:prod-aws Show me any new services or regions that appeared on our bill this month that weren't there last month. ``` ## Resource-Level Costs Drill into specific resources to understand cost drivers. ``` @cloud:prod-aws Which RDS instances cost the most? Include instance type, storage, and whether they're Multi-AZ. ``` ``` @cloud:prod-aws Estimate how much we'd save if we right-sized all EC2 instances in us-east-1 that have averaged less than 15% CPU utilization over the last 30 days. ``` ``` @cloud:prod-aws Which Elastic IPs are allocated but not associated with a running instance? How much are they costing us? ``` ``` @cloud:prod-aws What are the most expensive EBS volumes? Flag any that are unattached or over-provisioned relative to their utilization. ``` ``` @cloud:prod-aws How much are we spending on idle or underutilized load balancers? ``` ## Forecasting Project future costs based on current trends. ``` @cloud:prod-aws What's our projected total AWS spend for next month based on current usage patterns? ``` ``` @cloud:prod-aws At the current growth rate, what will our EC2 costs be in 3 months? What about 6 months? ``` ``` @cloud:prod-aws What's our projected EKS spend for the next quarter based on current trends? Break it down by cluster if possible. ``` ``` @cloud:prod-aws If we continue at the current data transfer rate, what will our CloudFront costs be next month? ``` ## Cross-Account Comparison Compare costs across multiple AWS accounts. ``` @cloud:prod-aws @cloud:staging-aws Compare total monthly spend across both accounts for the last 3 months. How much is staging costing relative to production? ``` ``` @cloud:prod-aws @cloud:staging-aws @cloud:dev-aws What's the cost per environment for EC2 and RDS? Are non-production environments disproportionately expensive? ``` ``` @cloud:prod-aws @cloud:staging-aws Are we running the same instance types in staging as production? If staging is using larger instances, how much would we save by right-sizing? ``` ## Example Conversation **You:** > `@cloud:prod-aws` Our AWS bill jumped 30% this month. What's going on? **Kestrel:** > Your total AWS spend for the current month is $47,200, up from $36,300 last month (+30%). The top 3 drivers of the increase: > > 1. **EC2** — up \$5,800 (+28%). Three new `m5.2xlarge` instances were launched in `us-west-2` on the 15th. They're running in the `analytics` account and aren't part of any Auto Scaling group. > 2. **RDS** — up \$3,100 (+41%). The `prod-analytics` instance had a storage auto-scaling event that grew it from 500GB to 1.2TB. There was also a class change from `db.r5.large` to `db.r5.2xlarge` on the 8th. > 3. **NAT Gateway** — up \$2,000 (+62%). Data processing through the NAT gateway in `us-east-1` doubled compared to last month, likely from the new batch processing jobs. **You:** > Can we save money on those EC2 instances? They might be over-provisioned. **Kestrel:** > Based on CloudWatch metrics for the last 2 weeks, the three `m5.2xlarge` instances in `us-west-2` average 12% CPU utilization and 4.2GB of memory usage (out of 32GB available). You have a few options: > > * **Right-size to `m5.large`**: Saves \~\$4,200/month. The instances would still have headroom at 48% average CPU and 8GB memory. > * **Switch to Spot**: These aren't in an ASG but could tolerate interruption if they're batch analytics jobs. Spot pricing for `m5.2xlarge` in `us-west-2` is currently 67% cheaper. > * **Reserved Instances**: If you need them long-term, a 1-year no-upfront RI for `m5.large` would save an additional 30% on top of right-sizing. **You:** > Let's right-size them to `m5.large`. Can you find the Terraform code? **Kestrel:** > The instances are defined in `acme/infrastructure/terraform/analytics/ec2.tf`. Here's the proposed change to update all three from `m5.2xlarge` to `m5.large`: > > *(displays Terraform diff with Create PR and Copy options)* ## Next Steps * [Kubernetes Investigation](/copilot/kubernetes) — query pods, deployments, events, and metrics * [Cloud Investigation](/copilot/cloud) — inspect AWS and OCI resources and security configurations * [Code Investigation](/copilot/code) — search and trace code changes across repositories * [Configuration Changes](/copilot/config-changes) — generate and apply YAML and Terraform changes # Kubernetes Investigation Source: https://docs.usekestrel.ai/copilot/kubernetes Query pods, deployments, events, logs, and metrics across your Kubernetes clusters The copilot can answer questions about any aspect of your connected Kubernetes clusters — pod status, container logs, events, resource specifications, metrics, network policies, deployments, and more. The examples on this page illustrate common query patterns, but you can ask anything about your clusters in natural language. ## Prerequisites At least one cluster connected with the Kestrel operator installed. Connect Datadog or OpenTelemetry to enable metrics queries (CPU, memory, latency, error rates). ## Targeting Clusters Use `@cluster-name` to direct your query to a specific Kubernetes cluster. You can target multiple clusters in a single query for cross-cluster comparisons. ``` @prod-eks-cluster show me pods in CrashLoopBackOff ``` ``` @prod-eks-cluster @staging-eks-cluster compare the resource limits for the auth-service deployment ``` If you have only one cluster connected, the copilot targets it automatically — no `@` prefix needed. Find your cluster names on the **Clusters** page in the Kestrel dashboard. The name matches whatever you set when connecting the cluster. ## Pod Status and Logs Ask about the health, status, and logs of any pod or group of pods. ``` @prod-eks-cluster Why is the payments-api pod crashing? Show me the last 50 lines of logs from the most recent restart. ``` ``` @prod-eks-cluster Which pods in the checkout namespace have restarted more than 3 times in the last hour? ``` ``` @prod-eks-cluster Show me all OOMKilled pods across all namespaces in the last 24 hours, grouped by deployment, with their current memory limits. ``` ``` @prod-eks-cluster What's the current status of all pods in the payments namespace? Flag any that aren't Running or have high restart counts. ``` ``` @prod-eks-cluster Get the logs from the init container of the migration-job pod — it failed on startup. ``` ## Events and Recent Changes Query Kubernetes events to understand what happened and when. ``` @prod-eks-cluster What events happened in the production namespace in the last 2 hours? Focus on warnings and errors. ``` ``` @prod-eks-cluster Were there any scheduling failures or evictions in the last 6 hours? ``` ``` @prod-eks-cluster What changed in the checkout namespace today? Include deployments, config maps, secrets, and any resource modifications. ``` ``` @prod-eks-cluster Show me all FailedScheduling events from the last hour and explain what's preventing the pods from being scheduled. ``` ## Resource Specs and Configuration Inspect the configuration of deployments, pods, services, and other Kubernetes resources. ``` @prod-eks-cluster What are the CPU and memory requests and limits for the order-service deployment? Are any containers missing resource requests? ``` ``` @prod-eks-cluster Show me the environment variables configured on the payments-api deployment, excluding secrets. ``` ``` @prod-eks-cluster Which deployments in the production namespace don't have liveness or readiness probes configured? ``` ``` @prod-eks-cluster What's the HPA configuration for the frontend deployment? What are the current vs desired replica counts? ``` ``` @prod-eks-cluster Show me all pods running with privileged security context across the cluster. ``` ## Metrics Requires [Datadog](/integrations/datadog) or [OpenTelemetry](/integrations/opentelemetry) to be connected. ``` @prod-eks-cluster What's the CPU and memory utilization for the checkout-service over the last 4 hours? Is it approaching its limits? ``` ``` @prod-eks-cluster Show me the P99 request latency for the payments-api service from Datadog. Has it degraded in the last hour? ``` ``` @prod-eks-cluster Which pods in the analytics namespace are consistently using more than 80% of their memory limit? ``` ``` @prod-eks-cluster Compare the error rate for the auth-service between today and yesterday. Did the latest deployment make it worse? ``` ## Network Policies and Service Mesh Query network policies, services, and connectivity. ``` @prod-eks-cluster What network policies apply to pods in the payments namespace? Can the payments-api reach the orders-db service? ``` ``` @prod-eks-cluster Show me all ingress and egress rules for the frontend namespace. Are there any policies that could be blocking traffic to the external payment gateway? ``` ``` @prod-eks-cluster List all services in the production namespace with their types, ports, and selectors. ``` ``` @prod-eks-cluster Trace the full request path from the frontend ingress to the database for the order-service — which services are involved and what network policies apply at each hop? ``` ## Deployment Status and Rollouts Inspect deployments, rollouts, and scaling operations. ``` @prod-eks-cluster Is the latest rollout for the order-service deployment healthy? Show me the rollout history. ``` ``` @prod-eks-cluster Which deployments were updated in the last 12 hours? Did any of them cause pod restart spikes? ``` ``` @prod-eks-cluster What Helm releases were upgraded in the last 24 hours across the production namespace? ``` ``` @prod-eks-cluster The checkout-api deployment is stuck at 2/5 ready replicas. Why aren't the remaining pods coming up? ``` ## Cross-Cluster Comparison Compare resources and configurations across multiple connected clusters. ``` @prod-eks-cluster @staging-eks-cluster Compare the resource limits for the auth-service deployment across both clusters. Are there any discrepancies in CPU or memory requests? ``` ``` @prod-eks-cluster @staging-gke-cluster Are the same network policies applied to the payments namespace in both clusters? ``` ``` @prod-eks-cluster @staging-eks-cluster Which deployments exist in production but not in staging? ``` ``` @prod-eks-cluster @staging-eks-cluster Compare the image tags for all deployments in the checkout namespace. Which services are running different versions? ``` ## Example Conversation **You:** > `@prod-eks-cluster` Pods in the payments namespace are throwing 5xx errors. What's going on? Cross-reference with Datadog latency and error rate metrics for the payments-api service. **Kestrel:** > I found 3 pods in the `payments` namespace showing issues: > > * `payments-api-7f8b9c-x4k2m` — Running but 12 container restarts in the last hour > * `payments-api-7f8b9c-n3j8p` — CrashLoopBackOff since 14:47 UTC > * `payments-worker-6d4a8b-j9n3p` — Running, 0 restarts > > Datadog metrics show the `payments-api` P99 latency spiked from 85ms to 3.8s at 14:32 UTC. Error rate jumped from 0.1% to 14.2% at the same time. The container logs from `payments-api-7f8b9c-n3j8p` show repeated `FATAL: connection pool exhausted, max_connections=10` errors. The deployment was updated at 14:30 UTC — image changed from `v2.13.2` to `v2.14.0`. **You:** > What changed in v2.14.0? Can you find the PR? **Kestrel:** > The image tag `v2.14.0` maps to PR #847 in `acme/payments-service`, merged at 14:28 UTC by @jane.doe. The PR description says "Optimize DB connection handling" — it reduced `DB_MAX_CONNECTIONS` from 50 to 10 in the deployment environment variables. This matches the `connection pool exhausted` error in the logs. The connection pool is now too small for the current request volume. **You:** > Roll back the deployment to v2.13.2 and increase the memory limit to 512Mi while you're at it. **Kestrel:** > Here's the proposed change for the `payments-api` deployment: > > 1. Revert image tag from `v2.14.0` to `v2.13.2` > 2. Increase memory limit from `256Mi` to `512Mi` > > *(displays diff with Safe Apply, Create PR, and Copy options)* ## Next Steps * [Cloud Investigation](/copilot/cloud) — query AWS and OCI resources and security configurations * [Code Investigation](/copilot/code) — search and trace code changes across repositories * [Cost Analysis](/copilot/costs) — analyze Kubernetes and cloud spend patterns * [Configuration Changes](/copilot/config-changes) — generate and apply YAML and Terraform changes # Cloud AI Copilot Source: https://docs.usekestrel.ai/copilot/overview Investigate your infrastructure and applications with conversational AI The Kestrel Cloud AI Copilot lets you ask questions about your Kubernetes clusters, cloud accounts, code repositories, metrics, and costs in plain English. Query resources, investigate incidents, analyze spend, and generate configuration changes — all through a conversational interface in the Kestrel dashboard or the `/kestrel` Slack command. ## What the Copilot Can Do Query pod logs, events, resource specs, deployments, metrics, and network policies across your clusters. Inspect AWS and OCI resources, security configurations, CloudTrail events, CloudWatch metrics, and VPC Flow Logs. Search files, read source code, trace change history, and look up IaC definitions across GitHub and GitLab repositories. Query AWS spend breakdowns, identify cost trends, detect anomalies, and forecast future costs. Generate Kubernetes YAML, Terraform, Helm, and CloudFormation changes with an approval workflow before applying. ## Getting Started Navigate to **Cloud AI Copilot → Chat** in the Kestrel dashboard sidebar. Alternatively, use the `/kestrel` slash command in any Slack channel if the [Slack integration](/integrations/slack) is connected. Use `@cluster-name` to target a specific Kubernetes cluster and `@cloud:connection-name` to target a specific AWS or OCI account. You can mention multiple targets in a single query. Type your question in plain English and press Enter. The copilot maintains conversation context, so follow-up questions work naturally. **Examples:** ``` @prod-eks-cluster Why are pods in the payments namespace restarting? ``` ``` @cloud:prod-aws Show me all S3 buckets that are publicly accessible ``` ``` @prod-eks-cluster @cloud:prod-aws What changed in the last 2 hours that could explain the spike in 5xx errors? ``` ## Targeting Syntax The copilot needs to know which infrastructure to query. Use the `@` prefix to target specific clusters and cloud accounts. | Syntax | Target | Example | | ------------------------ | ---------------------- | --------------------------------------------------------------- | | `@cluster-name` | Kubernetes cluster | `@prod-eks-cluster show me crashing pods` | | `@cloud:connection-name` | AWS or OCI account | `@cloud:prod-aws list EC2 instances in us-east-1` | | Multiple targets | Cross-resource queries | `@prod-eks @staging-eks compare memory limits for auth-service` | You can find your cluster and connection names on the **Clusters** and **Cloud Connections** pages in the Kestrel dashboard. ## Integration Prerequisites The copilot answers are only as good as the integrations connected. Each integration unlocks additional capabilities: | Integration | Copilot Capabilities | | ---------------------------------------------------- | ------------------------------------------------------------------------------------ | | [Kubernetes](/integrations/kubernetes) | Pod logs, events, resource specs, deployments, namespaces, network policies | | [AWS](/integrations/aws) | EC2, S3, RDS, IAM, VPC, Lambda, CloudTrail, CloudWatch, Cost Explorer, VPC Flow Logs | | [OCI](/integrations/oci) | Compute, Object Storage, VCN, IAM, Audit logs, VCN Flow Logs | | [Datadog](/integrations/datadog) | Metrics, monitors, dashboards, APM traces | | [OpenTelemetry](/integrations/opentelemetry) | Traces, metrics, and spans | | [GitHub](/integrations/github) | Code search, file reading, commit history, PRs | | [GitLab](/integrations/gitlab) | Code search, file reading, commit history, MRs | | [Knowledge Sources](/integrations/knowledge-sources) | Confluence, Jira, Slack history, Glean, Linear context | The more integrations you connect, the more context the copilot has. Connecting observability, code, and knowledge sources alongside your clusters and cloud accounts produces significantly richer answers. ## Using via Slack With the [Slack integration](/integrations/slack) connected, use the `/kestrel` slash command anywhere in your workspace: ``` /kestrel @prod-eks-cluster what pods are crashing in the checkout namespace? ``` ``` /kestrel @cloud:prod-aws who modified the IAM policy for the deploy role yesterday? ``` Kestrel responds in-thread so conversations stay organized. Other team members can see the answers and ask follow-ups in the same thread. The copilot also works inside Kestrel incident notification threads, where it automatically has the incident context. ## Example Conversation **You:** > `@prod-eks-cluster` Why are pods in the checkout namespace restarting? Cross-reference with Datadog metrics. **Kestrel:** > Two pods in the `checkout` namespace are in CrashLoopBackOff: `checkout-api-7f8b9c-x4k2m` and `checkout-worker-6d4a8b-j9n3p`. Both started crashing 23 minutes ago after deployment `checkout-api` was updated to image tag `v2.14.0`. Datadog shows P99 latency spiked from 120ms to 4.2s at 14:32 UTC, coinciding with the deployment. The container logs show `FATAL: too many connections for role "checkout_svc"` — the connection pool was exhausted. **You:** > What PR caused that deployment change? **Kestrel:** > The deployment was updated via ArgoCD sync at 14:32 UTC. The change originated from PR #847 in `acme/checkout-service`, authored by @jane.doe and approved by @john.smith. The PR modified the database connection pool settings, reducing `maxConnections` from 50 to 10. This matches the `too many connections` error in the logs. **You:** > Roll back the deployment to the previous image tag. **Kestrel:** > Here's the proposed change to roll back `checkout-api` to image tag `v2.13.2`: > > *(displays diff with Apply, Create PR, and Copy options)* ## Next Steps * [Kubernetes Investigation](/copilot/kubernetes) — query pods, deployments, events, logs, and metrics * [Cloud Investigation](/copilot/cloud) — inspect AWS and OCI resources and security configurations * [Code Investigation](/copilot/code) — search and read code across connected repositories * [Cost Analysis](/copilot/costs) — analyze AWS spend patterns and anomalies * [Configuration Changes](/copilot/config-changes) — generate and apply YAML and Terraform changes # Security & Compliance Source: https://docs.usekestrel.ai/enterprise/security Enterprise-grade security controls and data privacy Kestrel is built for organizations that demand the highest standards of data privacy, security, and compliance. Every design decision prioritizes your data sovereignty and operational security. ## SOC 2 Compliance Kestrel maintains SOC 2 Type II compliance, with security controls audited by independent third parties. Our compliance program covers: * **Access controls** — Role-based access with SSO enforcement * **Data encryption** — AES-256 at rest, TLS 1.2+ in transit * **Change management** — Audited deployment pipelines with automated testing * **Incident response** — Documented internal security incident procedures * **Monitoring** — Continuous infrastructure and application monitoring Contact [hello@usekestrel.ai](mailto:hello@usekestrel.ai) to request our SOC 2 report. ## Data Isolation Every customer's data is fully isolated at the application and database level: * **Tenant-scoped data** — All data (incidents, cluster inventory, flows, configurations) is scoped to your tenant. Queries are always filtered by tenant ID at the database layer. * **Separate processing** — AI analysis runs in tenant-isolated contexts. Your incident data, logs, and resource specs are never mixed with another customer's data. * **Integration credentials** — API keys, tokens, and OAuth credentials for your integrations (AWS, Slack, GitHub, etc.) are encrypted and stored per-tenant. ## No Cross-Customer Model Training Kestrel's AI models are **never trained on customer data**. Your Kubernetes logs, CloudTrail events, infrastructure configurations, and incident analysis are never used to train, fine-tune, or improve models for other customers. AI analysis uses foundation models (via API) with your data provided as context at inference time only. No customer data is retained by the model provider after inference. ## Read-Only by Default The Kestrel operator and cloud integrations are **read-only by default**: | Component | Default Access | Write Access | | ----------------------- | -------------------------------------------------- | ----------------------------------------------------------- | | **Kubernetes Operator** | Read pods, events, logs, resources | Requires explicit Safe Apply or auto-remediation enablement | | **AWS Integration** | Read CloudTrail, describe resources, Cost Explorer | Requires explicit IaC remediation configuration | | **OCI Integration** | Read Audit logs, describe resources | Requires explicit IaC remediation configuration | | **GitHub / GitLab** | Read repository contents | PR creation requires explicit enablement | Write operations only occur when you explicitly: 1. Click **Safe Apply** on a generated fix in the dashboard 2. Click **Create PR** to open a pull request with a fix 3. Enable **auto-remediation** for specific incident types on a specific cluster Every write operation is logged in the [audit trail](#full-audit-trail). ## No Sensitive Data Storage Kestrel does not store secrets, credentials, or sensitive environment variables from your clusters or cloud accounts: * **Environment variables** — Kestrel reads resource specs for analysis but does not persist Secret values or sensitive environment variable contents * **Cloud credentials** — AWS and OCI credentials are used for API calls and never logged or stored in plaintext * **Integration tokens** — OAuth tokens and API keys are encrypted at rest using AES-256 ## Full Audit Trail Every action in Kestrel is logged in an immutable audit trail: * **User actions** — Login, configuration changes, fix approvals, integration setup * **AI actions** — Incident detection, root cause analysis, fix generation, auto-remediation * **Write operations** — Every Safe Apply, PR creation, and auto-remediation event with before/after state * **Integration events** — Slack notifications sent, PagerDuty incidents created, Jira tickets filed Audit logs are accessible from the Kestrel dashboard and can be exported for compliance reporting. ## SSO Support Kestrel supports Single Sign-On via Google OAuth for enterprise customers. All team members authenticate through your organization's Google Workspace identity provider, with: * Centralized user provisioning * Enforced multi-factor authentication (via Google Workspace policies) * Automatic session management ## On-Premise Deployment For organizations with strict data residency requirements, regulatory compliance needs, or air-gapped environments, Kestrel supports full self-hosted deployment: * **Your infrastructure** — All components run in your own Kubernetes cluster (AWS, GCP, Azure, OCI) * **Your LLM provider** — Use Amazon Bedrock, Vertex AI, Azure OpenAI, or OCI Generative AI with VPC/private endpoints * **Air-gapped option** — No internet access required; all services accessed via private endpoints * **Native workload identity** — IRSA, GKE Workload Identity, Azure Managed Identity, or OKE Workload Identity See [On-Premise Overview](/on-premise/overview) and [On-Premise Setup](/on-premise/setup) for deployment details. ## Questions For security questionnaires, compliance documentation, or to discuss your organization's specific requirements, contact us at [hello@usekestrel.ai](mailto:hello@usekestrel.ai). # Cloud Incident Response Source: https://docs.usekestrel.ai/incident-response/cloud AI-powered detection and remediation for AWS and OCI cloud incidents Kestrel monitors your AWS and OCI cloud accounts for security events, misconfigurations, and anomalous activity. When an incident is detected — an IAM policy violation, an S3 bucket made public, a VPC route change, a database misconfiguration — Kestrel runs root cause analysis and generates Terraform, CloudFormation, Pulumi, or CLI fixes. ## Prerequisites Connect an AWS account with CloudTrail enabled. Connect an OCI tenancy with Audit logging enabled. Receive cloud incident notifications in your team's channels. Route cloud incidents to your on-call team. Enable IaC remediation workflows with PR-based fixes. ## How Detection Works ### AWS Kestrel analyzes your AWS environment through: * **CloudTrail events** — API call analysis for IAM changes, security group modifications, S3 policy changes, VPC route changes, KMS key operations, Secrets Manager events, and other security-relevant actions * **Resource monitoring** — Continuous assessment of EC2 instances, S3 buckets, RDS databases, DynamoDB tables, Lambda functions, ECS/EKS containers, and other AWS resources * **VPC Flow Logs** — Ingestion and analysis of network traffic patterns from VPC Flow Logs in CloudWatch for detecting suspicious traffic, unauthorized access attempts, and network anomalies * **CloudWatch Alarms** — Integration with CloudWatch alarms for metric-based incident detection * **Cost anomalies** — Detection of unusual spend patterns across services and accounts via AWS Cost Anomaly Detection * **AWS Config** — Monitoring for compliance rule violations across your resources * **Security Hub** — Integration with AWS Security Hub for aggregated security findings * **PagerDuty signals** — If PagerDuty is connected, Kestrel enriches PagerDuty alerts with cloud context and AI analysis ### OCI Kestrel analyzes your OCI environment through: * **Audit log events** — API call analysis for IAM policy changes, compartment modifications, network security changes, and other security-relevant actions * **Cloud Guard problems** — Integration with OCI Cloud Guard for threat detection * **VCN Flow Logs** — Ingestion and analysis of network traffic from VCN Flow Logs for detecting suspicious traffic patterns and network anomalies * **Resource monitoring** — Continuous assessment of compute instances, object storage buckets, databases, networking configurations, load balancers, and DNS records ## Root Cause Analysis When a cloud incident is detected, Kestrel's AI investigates: 1. **API event history** — The sequence of CloudTrail or OCI Audit events that led to the incident 2. **Resource configuration** — Current and previous state of the affected resource 3. **IAM context** — Who performed the action, what role they assumed, and whether it was authorized 4. **Network context** — VPC/VCN, subnet, security group, and route table configurations 5. **Network traffic** — VPC/VCN Flow Logs analysis for suspicious traffic patterns related to the incident 6. **Knowledge sources** (if connected) — Historical incidents, runbooks, and organizational context from Confluence, Jira, Slack, Glean, or Linear The AI produces: * **Investigation summary** — What happened, who was involved, and the blast radius * **Root cause** — The specific API call or configuration change that caused the incident * **Timeline** — Chronological sequence of events with timestamps and principals * **Affected resources** — All cloud resources impacted * **Severity assessment** — Severity and potential downstream impact ## Fix Generation Kestrel generates infrastructure-as-code fixes for cloud incidents. The fix format depends on your infrastructure tooling: | Fix Type | Description | | ------------------ | ------------------------------------------------- | | **Terraform** | HCL resource definitions ready to apply or commit | | **CloudFormation** | CloudFormation template changes | | **Pulumi** | Pulumi code changes | | **AWS CLI** | AWS CLI commands to remediate the issue directly | | **OCI CLI** | OCI CLI commands for OCI resource remediation | Each fix includes: * **Diff view** — Before and after comparison of the resource configuration * **Explanation** — Why this change resolves the issue * **Apply options** — Create a PR to your IaC repository, apply directly via CLI, or copy to apply manually ## IaC Remediation Workflow When a GitHub or GitLab integration is connected along with an IaC repository configuration, Kestrel can open pull requests with IaC fixes directly in your infrastructure repository: Kestrel detects a cloud security event (e.g., S3 bucket made public). Root cause analysis runs and a Terraform/CloudFormation/Pulumi fix is generated. Kestrel opens a pull request in your configured IaC repository with the fix, targeting the exact file managing the resource. Your team reviews the PR. Merging applies the fix through your existing GitOps pipeline. Configure IaC repositories from **Integrations → CI/CD** by selecting your connected GitHub or GitLab repo, the IaC type (Terraform, CloudFormation, or Pulumi), and the target directory. ## Cloud Incident Types ### AWS — Security & Identity | Type | Description | | --------------------------- | ------------------------------------------------------- | | IAM Compromise | Potential IAM credential compromise detected | | IAM Policy Violation | IAM policy change that violates security best practices | | IAM Role Abuse | Suspicious IAM role assumption patterns | | Unauthorized Access | API calls from unauthorized principals | | Credential Leak | Exposed credentials detected | | Privilege Escalation | Attempt to escalate IAM privileges | | Root Account Activity | AWS root account used (critical security event) | | KMS Key Compromise | KMS key deletion, disabling, or policy change | | Secrets Manager Event | Secret deletion, policy changes, or rotation changes | | Security Hub Finding | High or critical findings from AWS Security Hub | | Config Compliance Violation | AWS Config rule non-compliance | ### AWS — Storage | Type | Description | | ---------------------- | --------------------------------------------------- | | S3 Public Access | S3 bucket or object made publicly accessible | | S3 Policy Modification | Bucket policy changed in a security-relevant way | | S3 Data Exfiltration | Unusual data access patterns on S3 objects | | S3 Encryption Disabled | Bucket encryption configuration removed or weakened | ### AWS — Compute & Containers | Type | Description | | ----------------------- | ---------------------------------------------------------------------- | | EC2 Anomaly | Unusual EC2 instance behavior | | EC2 Instance Compromise | Potential EC2 instance compromise | | EC2 Unauthorized Access | Unauthorized access to EC2 instances | | Lambda Function Issue | Lambda function errors, configuration changes, or permission issues | | ECS/EKS Container Issue | Container service task failures, cluster changes, or deployment issues | ### AWS — Database | Type | Description | | ------------------ | ------------------------------------------------------------------------------------------------ | | RDS Database Issue | RDS database deletion, snapshot exposure, configuration changes, or security group modifications | | DynamoDB Issue | DynamoDB table deletion, backup issues, or policy changes | ### AWS — Network & VPC | Type | Description | | --------------------------- | ----------------------------------------------------------- | | Security Group Modification | Security group rules changed | | VPC Modification | VPC configuration changed | | VPC Route Change | Route table entry added, modified, or deleted | | VPC Route Table Change | Route table associated or disassociated | | VPC Flow Logs Disabled | VPC Flow Logs disabled | | VPC Peering Change | VPC peering connection modified | | VPC Gateway Change | Internet or NAT gateway configuration changed | | VPC Subnet Change | Subnet configuration changed | | VPC Deletion | VPC deleted | | Network Traffic Anomaly | Unusual network traffic patterns detected via VPC Flow Logs | | Network Traffic Blocked | Traffic blocked by security groups or NACLs | ### AWS — Service Health & Operations | Type | Description | | -------------------------- | ---------------------------------------------- | | Service Health Degradation | Degraded performance of AWS services | | API Throttling | API throttling across AWS services | | Resource Exhaustion | Service limit or quota exhaustion | | CloudWatch Alarm | CloudWatch alarm entered ALARM state | | Application Log Errors | Application errors detected in CloudWatch Logs | ### AWS — Cost & Billing | Type | Description | | ------------------------- | ---------------------------------------------------------------- | | Cost Anomaly Detected | AWS Cost Anomaly Detection identified unexpected spend increases | | Budget Threshold Exceeded | AWS Budget threshold breached | ### OCI — IAM & Access | Type | Description | | ------------------------ | ----------------------------------------- | | OCI IAM Compromise | Potential OCI IAM compromise detected | | OCI IAM User Change | IAM user created, modified, or deleted | | OCI IAM Group Change | IAM group membership changed | | OCI Dynamic Group Change | Dynamic group matching rules changed | | OCI Policy Change | OCI IAM policy statement modified | | OCI Compartment Change | Compartment created, moved, or deleted | | OCI Credential Change | API key or auth token modified | | OCI Privilege Escalation | Privilege escalation attempt detected | | OCI IAM Security Change | General IAM security configuration change | ### OCI — Network | Type | Description | | ------------------------------------ | ---------------------------------------------- | | OCI Security List Change | Security list ingress/egress rules changed | | OCI Network Security Group Change | Network security group rules modified | | OCI VCN Change | VCN configuration changed | | OCI Subnet Change | Subnet configuration modified | | OCI Route Table Change | Route table rules changed | | OCI Internet Gateway Change | Internet gateway created or modified | | OCI NAT Gateway Change | NAT gateway configuration changed | | OCI Service Gateway Change | Service gateway configuration changed | | OCI Load Balancer Change | Load balancer configuration modified | | OCI Network Load Balancer Change | Network load balancer modified | | OCI DRG Change | Dynamic Routing Gateway modified | | OCI DRG Attachment Change | DRG attachment modified | | OCI Local Peering Gateway Change | Local peering gateway modified | | OCI Remote Peering Connection Change | Remote peering connection modified | | OCI Public IP Change | Public IP address changed | | OCI Private IP Change | Private IP address changed | | OCI VNIC Change | Virtual NIC configuration changed | | OCI DHCP Options Change | DHCP options modified | | OCI WAF Change | Web Application Firewall configuration changed | | OCI DNS Change | DNS zone or record modified | | OCI API Gateway Change | API Gateway configuration changed | | OCI Network Security Change | General network security configuration change | ### OCI — Storage | Type | Description | | ------------------------- | ---------------------------------------------- | | OCI Bucket Public Access | Object Storage bucket made publicly accessible | | OCI Bucket Policy Change | Bucket policy modified | | OCI Object Storage Change | Object Storage configuration changed | ### OCI — Compute | Type | Description | | ---------------------- | ---------------------------------------- | | OCI Compute Anomaly | Unusual compute instance behavior | | OCI Console Connection | Console connection to a compute instance | ### OCI — Database & KMS | Type | Description | | ---------------------------- | ----------------------------------------------- | | OCI Database Security Change | Database security configuration changed | | OCI KMS Key Compromise | Potential KMS key compromise | | OCI KMS Security Change | KMS vault or key security configuration changed | ### OCI — Logging & Cloud Guard | Type | Description | | ----------------------- | -------------------------------------------- | | OCI Logging Tampering | Audit or logging configuration tampered with | | OCI Cloud Guard Problem | Cloud Guard detected a security problem | ## Custom Workflows Build custom cloud incident response workflows that trigger on cloud events. For example: * Notify a security channel when an S3 bucket is made public * Auto-create a Jira ticket for every IAM policy violation * Escalate VPC route changes to PagerDuty during change-freeze windows * Run a custom remediation script when a Security Hub finding is detected See [Workflows](/workflows/create-workflows) to build custom cloud incident response automation. ## Next Steps * [Connect an AWS account](/integrations/aws) to start detecting cloud incidents * [Connect an OCI tenancy](/integrations/oci) for Oracle Cloud monitoring * [Set up GitHub or GitLab](/integrations/github) for IaC remediation PRs * [Configure Slack notifications](/integrations/slack) for real-time cloud alerts * [Build custom workflows](/workflows/create-workflows) for automated cloud incident response # Kubernetes Incident Response Source: https://docs.usekestrel.ai/incident-response/kubernetes AI-powered detection, root cause analysis, and remediation for Kubernetes incidents Kestrel monitors your Kubernetes clusters 24/7, automatically detecting incidents as they occur. When something goes wrong — a pod enters CrashLoopBackOff, a deployment rollout stalls, a node becomes unresponsive — Kestrel's AI runs root cause analysis, generates YAML fixes, identifies causal PRs, and creates postmortems. ## Prerequisites Connect at least one Kubernetes cluster with the Kestrel operator installed. Receive incident notifications and ask follow-up questions in threads. Connect Datadog or OpenTelemetry for metrics context during root cause analysis. Route incidents to your on-call team with AI-enriched context. Enable causal PR detection and GitOps remediation workflows. Connect Confluence, Jira, Slack history, Glean, or Linear for historical incident context. ## How Detection Works The Kestrel operator runs inside your cluster and continuously monitors: * **Pod status** — CrashLoopBackOff, OOMKilled, ImagePullBackOff, evictions, probe failures * **Deployment rollouts** — stalled rollouts, failed replica scaling, rollback events * **Node health** — NotReady nodes, resource pressure, taint/toleration mismatches * **Kubernetes events** — scheduling failures, volume errors, RBAC denials, container runtime errors * **Application signals** — segfaults, panics, uncaught exceptions, null pointer errors, deadlocks When the operator detects an abnormal condition, it creates an incident and immediately begins AI-powered investigation. Kestrel distinguishes between **transient** incidents (pods that crash once and recover) and **persistent** incidents. Transient incidents are hidden by default in the dashboard but are still recorded for historical analysis. ## Root Cause Analysis When an incident is detected, Kestrel's AI analyzes multiple data sources to determine the root cause: 1. **Container logs** — Recent logs from the affected pod and its previous restarts 2. **Kubernetes events** — Events attached to the pod, deployment, node, and namespace 3. **Resource specifications** — CPU/memory requests and limits, volume mounts, environment variables, probes 4. **Metrics** (if Datadog or OpenTelemetry is connected) — CPU usage, memory consumption, request latency, error rates 5. **Knowledge sources** (if connected) — Historical incidents, runbooks, and tribal knowledge from Confluence, Jira, Slack, Glean, or Linear The AI produces: * **Investigation summary** — A step-by-step account of what was analyzed and what was found * **Root cause** — The identified cause of the incident with supporting evidence * **Timeline** — Sequence of events leading to the incident * **Affected resources** — All Kubernetes resources involved (pods, deployments, services, nodes) * **Service dependencies** — Upstream and downstream services that may be impacted ## Fix Generation After identifying the root cause, Kestrel generates YAML fixes that address the issue. Fixes are generated as Kubernetes resource patches — for example, increasing memory limits for an OOMKilled pod or correcting an image tag for an ImagePullBackOff. Each fix includes: * **Diff view** — Side-by-side comparison of the current and proposed resource spec * **Explanation** — Why this change addresses the root cause * **Apply options:** | Method | Description | | -------------- | ----------------------------------------------------------------------------------- | | **Safe Apply** | Apply the fix directly to the cluster through the Kestrel operator | | **Create PR** | Open a pull request in your connected GitHub or GitLab repository (GitOps workflow) | | **Copy YAML** | Copy the fix to your clipboard for manual application | For application-level incidents (segfaults, exceptions, panics), Kestrel also generates **code fixes** — suggested source code changes with file paths and diffs, ready to be applied as a PR. Safe Apply requires write permissions on the Kestrel operator. By default, the operator runs in read-only mode. See [Auto-Remediation](#auto-remediation) to enable write access. ## Causal PR Detection When a GitHub or GitLab integration is connected, Kestrel searches for recently merged pull requests or merge requests that may have caused the incident. The AI correlates: * The **affected resource** (deployment, namespace, image tag) * The **incident timeline** (when the issue first appeared) * **Recent merges** in connected repositories (code changes, config changes, Helm value updates) If a likely causal PR is found, the incident detail page links directly to it with an explanation of why Kestrel believes it contributed to the issue. ## Incident Types ### Infrastructure Incidents | Type | Description | | --------------------------- | ---------------------------------------------------------- | | **CrashLoopBackOff** | Container repeatedly crashes after starting | | **OOMKilled** | Container terminated due to exceeding memory limits | | **ImagePullBackOff** | Unable to pull the container image | | **RolloutFailure** | Deployment rollout failed or stalled | | **NodeNotReady** | Node is not in a Ready state | | **SchedulingFailure** | Pod cannot be scheduled to any node | | **InsufficientResources** | Cluster lacks CPU or memory to schedule the pod | | **TaintTolerationMismatch** | Pod tolerations don't match available node taints | | **NodeSelectorMismatch** | No nodes match the pod's node selector | | **PodEvicted** | Pod was evicted due to resource pressure | | **LivenessProbeFailure** | Liveness probe is failing, triggering container restarts | | **ReadinessProbeFailure** | Readiness probe is failing, removing the pod from service | | **HealthCheckFailure** | Health check endpoint is not responding | | **MissingPVC** | PersistentVolumeClaim referenced by the pod does not exist | | **VolumeError** | Volume mount or attachment failure | | **VolumeResizeError** | PersistentVolume resize operation failed | | **NetworkError** | Network connectivity issue affecting the pod | | **RBACError** | Insufficient RBAC permissions for the workload | | **ResourceExhaustion** | Cluster-wide resource exhaustion (CPU, memory, or disk) | | **ContainerRuntimeError** | Container runtime failure (containerd, CRI-O) | | **LifecycleHookError** | PreStop or PostStart lifecycle hook failure | | **ConfigError** | ConfigMap or Secret misconfiguration | ### Application Incidents | Type | Description | | ------------------------ | --------------------------------------------- | | **ApplicationError** | Generic application error detected in logs | | **Segfault** | Segmentation fault in the application process | | **NullPointerException** | Null pointer or nil reference dereference | | **Panic** | Go panic or equivalent unrecoverable error | | **UncaughtException** | Unhandled exception in the application | | **IndexOutOfBounds** | Array or slice index out of range | | **DivisionByZero** | Division by zero error | | **MemoryError** | Application-level memory allocation failure | | **StackOverflow** | Stack overflow due to excessive recursion | | **AssertionError** | Assertion failure in application code | | **Deadlock** | Thread or goroutine deadlock detected | ## Auto-Remediation By default, the Kestrel operator runs in **read-only mode** — it can detect incidents and generate fixes, but cannot apply changes to your cluster. To enable automatic or one-click fix application: 1. Navigate to **Clusters** in the Kestrel dashboard 2. Select the cluster you want to configure 3. Enable **Safe Apply** to allow one-click fix application from the dashboard 4. Optionally enable **Auto-Remediation** to let Kestrel automatically apply approved fix types without manual intervention Auto-remediation policies are configured per cluster. You can define which incident types and severities are eligible for automatic remediation. ## Custom Workflows Build custom incident response workflows that trigger when Kubernetes incidents are detected. For example: * Notify a specific Slack channel when a critical OOMKilled incident occurs in the `production` namespace * Create a Jira ticket for every incident in a specific cluster * Automatically restart a deployment after a transient CrashLoopBackOff * Escalate to PagerDuty only if the incident persists for more than 5 minutes See [Workflows](/workflows/create-workflows) to build custom Kubernetes incident response automation. ## Next Steps * [Connect a Kubernetes cluster](/integrations/kubernetes) to start detecting incidents * [Set up Slack notifications](/integrations/slack) for real-time alerts * [Connect GitHub or GitLab](/integrations/github) for causal PR detection and GitOps remediation * [Build custom workflows](/workflows/create-workflows) for automated incident response * [Configure PagerDuty](/integrations/pagerduty) for on-call routing with AI context # Argo Rollouts Source: https://docs.usekestrel.ai/integrations/argo-rollouts Connect Argo Rollouts for progressive delivery actions — promote, abort, retry, and roll back canary and blue-green rollouts in workflows The Argo Rollouts integration connects Kestrel to your in-cluster Argo Rollouts installation, enabling AI agents to promote canary deployments, abort or retry failed rollouts, and roll back to previous revisions as part of automated workflows and incident response. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * Argo Rollouts controller and CRDs installed in the same cluster (`argoproj.io/v1alpha1 Rollout`) ## Setup Argo Rollouts is auto-detected by the Kestrel Operator when the Rollout CRD is present in the cluster. No manual connection or credentials are required. If Argo Rollouts is not already installed: ```bash theme={null} kubectl create namespace argo-rollouts kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml ``` The operator probes for the Rollout CRD periodically. Check the operator logs: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "Rollouts" ``` Once detected, the integration shows as **Detected** on the Integrations page and Rollouts action blocks become available in the workflow builder. The integration is enabled by default. To opt out, set `operator.rollouts.enabled: false` in your Helm values and upgrade the operator (this sets `ROLLOUTS_DISABLED=true` on the operator deployment). ## How It's Used ### In Workflows **Action blocks:** * **Promote Rollout** — promote a paused canary/blue-green rollout to the next step (or fully with `full: true`) * **Abort Rollout** — abort an in-progress rollout and revert to the stable version * **Retry Rollout** — retry an aborted rollout from the beginning * **Rollback Rollout** — roll back a rollout to a previous revision (defaults to the last revision) * **Pause Rollout** / **Resume Rollout** — pause or resume rollout progression * **Restart Rollout** — restart the pods of a rollout without changing the version * **Get Rollout Status** — retrieve current phase, step, and canary weight of a rollout * **Wait for Rollout Healthy** — block workflow execution until a rollout reports Healthy (with configurable timeout) Example: A workflow triggered by a Datadog monitor alert on elevated error rates uses "Abort Rollout" to stop the in-progress canary, then posts the rollout status to Slack for the on-call engineer. ### Scoping Rollout actions are scoped by **cluster**, **namespace**, and **rollout name** — all populated with dropdowns from live cluster data in the workflow builder. ## Disconnecting To disable the Argo Rollouts integration, set the following in your Helm values and upgrade the operator: ```yaml theme={null} operator: rollouts: enabled: false ``` # ArgoCD Source: https://docs.usekestrel.ai/integrations/argocd Connect ArgoCD for GitOps sync actions, deployment tracking, and status monitoring in workflows The ArgoCD integration connects Kestrel to your in-cluster ArgoCD installation, enabling AI agents to sync applications, monitor deployment status, and track GitOps operations as part of automated workflows and incident investigation. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * ArgoCD installed in the same cluster ## Setup ArgoCD is auto-detected by the Kestrel Operator when it is installed in the same cluster. No manual connection is required. During cluster onboarding or in the cluster settings page: 1. Navigate to **Integrations → Kubernetes** 2. Select your cluster 3. Enable **ArgoCD Integration** in the cluster configuration Alternatively, configure via Helm values: ```yaml theme={null} operator: argocd: enabled: true namespace: "argocd" ``` Check the operator logs for ArgoCD discovery: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "ArgoCD" ``` The Kestrel Operator discovers ArgoCD by looking for the ArgoCD Application CRDs and API server in the cluster. If ArgoCD is installed in a non-default namespace, specify it in the Helm values. ## How It's Used ### In Workflows **Action blocks:** * **Trigger ArgoCD Sync** — trigger an ArgoCD sync for a specific application * **Wait for ArgoCD Sync** — block workflow execution until a sync operation completes (with configurable timeout) * **Get ArgoCD App Status** — retrieve the current sync and health status of an ArgoCD application * **Rollback ArgoCD App** — roll back an ArgoCD application to its previous deployment version * **Find ArgoCD App for Workload** — dynamically discover which ArgoCD application manages a given Kubernetes workload (by name/namespace matching) Example: A workflow triggered by a pod crash uses "Find ArgoCD App for Workload" to identify the managing ArgoCD application, checks its status, and if unhealthy, triggers a rollback to the last known good version. ## Disconnecting To disable the ArgoCD integration: 1. Navigate to **Integrations → Kubernetes** and select your cluster 2. Disable **ArgoCD Integration** in the cluster configuration Or set `operator.argocd.enabled: false` in your Helm values and upgrade the operator. # AWS Source: https://docs.usekestrel.ai/integrations/aws Connect your AWS accounts to Kestrel for resource inventory, security monitoring, and VPC Flow Logs analysis Connect your AWS accounts to Kestrel using cross-account IAM roles for secure, credential-free access to your cloud resources. Kestrel discovers and inventories AWS resources, detects security misconfigurations, and collects VPC Flow Logs for network traffic analysis. ## Prerequisites * **Organization Admin** role in Kestrel * AWS account with permissions to create IAM roles and policies * AWS CLI or access to the AWS Console ## Setup 1. Navigate to **Integrations → Cloud** in your Kestrel dashboard 2. Click **Connect AWS Account** 3. Select your AWS region 4. Select the connection purpose: | Purpose | Description | | ------------------- | ----------------------------------------------------- | | **EKS** | Kubernetes cluster discovery and monitoring | | **VPC Flow Logs** | VPC Flow Log collection and analysis | | **Cloud Ingestion** | Full cloud resource inventory and security monitoring | 5. Click **Generate CloudFormation Template** Kestrel generates a CloudFormation template that creates a cross-account IAM role with minimum required permissions. **Option A: AWS Console (One-Click)** Click the **Launch Stack** link provided in the setup dialog. Review the parameters and click **Create Stack**. **Option B: AWS CLI** ```bash theme={null} aws cloudformation create-stack \ --stack-name kestrel-ai-integration \ --template-url \ --capabilities CAPABILITY_NAMED_IAM \ --parameters ParameterKey=ExternalId,ParameterValue= ``` 1. Once the stack reaches `CREATE_COMPLETE`, copy the **Role ARN** from the stack outputs 2. Paste the Role ARN into the Kestrel verification dialog 3. Click **Verify Connection** Kestrel assumes the role and validates permissions. On success, your account appears as **Connected**. You can create multiple connections to the same AWS account with different purposes, or use a single **Cloud Ingestion** connection for comprehensive coverage. ### VPC Flow Logs After connecting with VPC Flow Logs capability: 1. Navigate to the connected account in the Cloud Integrations page 2. Expand the **VPC Flow Logs** section 3. Click **Enable** to start collecting flow logs across your VPCs 4. Kestrel creates the necessary log groups and flow log configurations automatically You can enable/disable flow logs on individual VPCs. ## How It's Used ### In Workflows **Trigger blocks:** * **IAM Security Event** — fires on IAM role, policy, or user changes detected via CloudTrail * **Root Account Activity** — fires when the AWS root account is used * **KMS Key Change** — fires on KMS key creation, deletion, or policy changes * **Secrets Manager Event** — fires on secret creation, rotation, or deletion in Secrets Manager * **Security Hub Finding** — fires when AWS Security Hub reports a new finding * **S3 Bucket Change** — fires on S3 bucket policy, ACL, or public access changes * **EC2 Instance Issue** — fires on EC2 instance state changes, status check failures, or termination * **Lambda Function Issue** — fires on Lambda invocation errors, throttles, or configuration changes * **ECS/EKS Container Issue** — fires on ECS task failures or EKS cluster events * **RDS Database Issue** — fires on RDS instance failures, failovers, or storage issues * **DynamoDB Issue** — fires on DynamoDB table throttles, capacity changes, or errors * **VPC/Network Change** — fires on VPC, security group, NACL, or route table modifications * **CloudWatch Alarm** — fires when a CloudWatch alarm transitions to ALARM state * **Application Log Errors** — fires when application log error patterns exceed a threshold * **Config Rule Non-Compliant** — fires when an AWS Config rule evaluates a resource as non-compliant * **AWS Service Health Event** — fires on AWS Health Dashboard service disruptions affecting your account * **Any AWS Cloud Incident** — fires on any AWS cloud incident detected by Kestrel * **Cost Anomaly Detected** — fires when AWS Cost Anomaly Detection identifies unusual spend * **Budget Threshold Exceeded** — fires when an AWS Budget threshold is breached * **Forecast Exceeds Budget** — fires when the Cost Explorer forecast projects a budget overrun before it happens * **Spend Spike Detected** — fires when day-over-day spend rises beyond a configurable percentage threshold * **Idle Resource Detected** — fires when a daily scan finds idle resources (unattached EBS volumes, unassociated Elastic IPs, low-CPU instances, old snapshots) **Cost optimization action blocks:** Connected accounts unlock the Cloud Cost action blocks in workflows: * *Insights (read-only)*: Query Cost Explorer, Get Cost Anomalies, Get Cost Forecast, Get Budget Status, Get Rightsizing Recommendations, Get Savings Plans Recommendations, Get Reservation Recommendations, Get Commitment Utilization, Compare Cost Periods, Find Idle Resources, Get Compute Optimizer Recommendations, Get Trusted Advisor Cost Checks * *Remediation (mutates infrastructure — place behind an Approval block)*: Stop EC2 Instances, Delete Unattached EBS Volumes, Release Elastic IPs, Delete Old Snapshots. Resources tagged `kestrel:protected` are always skipped. Cloud operations (querying resources, describing instances, checking security groups, generating Terraform fixes, creating IaC remediation PRs) are provided by Kestrel's built-in action blocks and are available in any workflow alongside these triggers. Example: A workflow triggers on a Security Hub finding, queries the affected resource details, generates a Terraform fix, and opens a pull request in the connected IaC repository. ### In Incident Response * **Cloud security monitoring** — detects misconfigurations across EC2, S3, RDS, IAM, and other services * **Terraform fix generation** — when a misconfiguration is found, Kestrel generates a Terraform fix targeting the exact IaC file in your connected repository * **VPC Flow Log analysis** — analyzes traffic patterns to identify anomalous network behavior ### In Cloud AI Copilot * Ask questions about your AWS resources: `What S3 buckets have public access?` * Investigate security findings: `Show me all EC2 instances with open SSH access` * Query costs and resource utilization across accounts ## IAM Permissions The CloudFormation-created IAM role includes **read-only** access to: * EC2 (instances, security groups, VPCs, subnets, flow logs) * S3 (bucket policies, public access settings) * RDS (instances, clusters, security) * IAM (roles, policies, users — metadata only) * CloudTrail (event history) * EKS (clusters, node groups) * Lambda, ECS, and other compute services * Cost Explorer, Budgets, Compute Optimizer, and Trusted Advisor (cost and usage data, recommendations) Write permissions are limited to VPC Flow Log management (creating/deleting flow log configurations) and the cost-remediation actions (`ec2:StopInstances`, `ec2:DeleteVolume`, `ec2:ReleaseAddress`, `ec2:DeleteSnapshot`, plus `ec2:CreateSnapshot` for pre-deletion safety snapshots) used by the Cloud Cost remediation workflow blocks. If you connected your account before the cost-optimization blocks were released, update your CloudFormation stack to the latest template to grant the new Cost Explorer, Budgets, Compute Optimizer, and remediation permissions. Blocks that lack permissions fail with an actionable error pointing at the stack update. ## Managing Connections * **Refresh**: Re-sync connection status and resource inventory * **Delete**: Removes the connection from Kestrel. Clean up the CloudFormation stack in AWS separately. # Beam Source: https://docs.usekestrel.ai/integrations/beam Connect Beam.cloud for serverless deployment, container/sandbox lifecycle, task monitoring, and AI investigation in workflows The Beam integration connects Kestrel to your Beam.cloud workspace, enabling serverless task, container, deployment, and machine events to trigger workflows and giving AI agents access to deployment state, running containers/sandboxes, and task status across your workspace. Beam does not send tenant-wide outbound webhooks. Kestrel detects task failures and container/deployment/machine lifecycle events by polling the Beam API on a configurable interval, authenticating with a workspace bearer token. ## Prerequisites * **Organization Admin** role in Kestrel * Beam.cloud account with a workspace * A Beam workspace API token ## Setup Create a workspace API token from the Beam dashboard (under **Settings**) or with the Beam CLI (`beam config`). The token is sent as a bearer token to `https://app.beam.cloud/api/v1/gateway` to read deployment, container, and task state and perform actions across your workspace. 1. Navigate to **Integrations → Beam** in your Kestrel dashboard 2. Paste your **Beam workspace token**. 3. Choose a **Poll Interval** for trigger detection. 4. Click **Connect Beam** — your Beam workspace appears as connected. The workspace token grants access to your Beam deployments and containers. Treat it as a secret — Kestrel stores it encrypted. No webhook setup is required inside Beam. Kestrel polls the Beam API on your configured interval to detect task, container, deployment, and machine events. ## How It's Used ### In Workflows **Trigger blocks:** * **Task Failed** — fires when a Beam task transitions to FAILED * **Task Timed Out** — fires when a task transitions to TIMEOUT * **Container Stopped** — fires when a container, pod, or sandbox leaves a running state * **Deployment Scaled to Zero** — fires when a deployment drops to zero warm containers * **Queue Depth High** — fires when a deployment's task queue backlog grows * **Machine Down** — fires when a machine leaves a ready state (pool capacity loss) * **Sandbox Execution Failed** — fires when a Kestrel-created sandbox ends failed/errored or with a non-zero exit code * **Sandbox Terminated** — fires when a Kestrel-created sandbox ends/terminates cleanly Filter triggers by deployment and event type. Kestrel polls the Beam API on your configured interval to detect these conditions. Sandbox triggers observe only the sandboxes Kestrel creates (via the **Create Sandbox** action), not arbitrary workspace sandboxes. **Action blocks:** * **List Deployments** — list the deployments in the workspace * **Get Deployment** — retrieve a deployment's state and version * **Scale Deployment** — set the desired container count for a deployment * **Start Deployment** — activate a stopped deployment * **Stop Deployment** — deactivate a running deployment * **List Containers** — list running containers, pods, and sandboxes * **Get Container Status** — retrieve a container or sandbox status * **Stop Container** — stop a running container, pod, or sandbox * **List Tasks** — list recent tasks with their status * **Get Task Status** — fetch the status and detail of a task * **List Machines** — list machines, optionally scoped to a compute pool * **Investigate Beam** — run an AI investigation of failing tasks, stuck containers, and scaling issues **Sandbox action blocks:** * **Create Sandbox** — provision an ephemeral sandbox (Python or a registry base image, with declared packages, CPU/memory, and exposable ports). Outputs a `sandbox_id`. * **Run Command in Sandbox** — run a shell command or Python code in a sandbox and capture `stdout`/`stderr`/`exit_code` * **Upload File to Sandbox** — write file content into a sandbox path * **Download File from Sandbox** — read a file from a sandbox into step outputs (capped at 256KB) * **Expose Sandbox Port** — expose a port and get back a public, authenticated URL * **Terminate Sandbox** — terminate a sandbox to free resources * **List Sandboxes** — list the sandboxes Kestrel is currently tracking, with last-observed status Beam deployment, container, and machine selects accept template variables like `{{signal.deployment_id}}`, `{{signal.container_id}}`, `{{signal.task_id}}`, and `{{signal.machine_id}}` from the trigger. ### Sandbox lifecycle (chaining sandbox blocks) Sandboxes are persistent across workflow steps. A **Create Sandbox** step outputs a `sandbox_id`; downstream sandbox blocks reference it (the **Sandbox** select defaults to `{{step_outputs.action-1.sandbox_id}}`) to run commands, move files, and expose ports against the same sandbox. Always add a **Terminate Sandbox** step when finished to free resources — sandboxes also auto-reap based on their Keep Warm setting and Kestrel stops tracking them after 24 hours. Example: A workflow creates a sandbox with `requests` installed, uploads an input file, runs a Python data transform, downloads the result, posts a summary to Slack, then terminates the sandbox. Example: A workflow triggers on a failed task, runs an AI investigation to diagnose the failure, posts a summary to Slack, and opens a Jira bug ticket. ## Disconnecting 1. Navigate to **Integrations → Beam** 2. Click **Disconnect** 3. Confirm the disconnection This stops Beam polling and all Beam workflow triggers. You can reconnect at any time. # CircleCI Source: https://docs.usekestrel.ai/integrations/circleci Connect CircleCI for pipeline triggering, workflow rerun/cancel, on-hold job approval, test-result retrieval, failure webhooks, and AI investigation in workflows The CircleCI integration connects Kestrel to your CircleCI organization through the CircleCI v2 API, enabling pipeline and job events to trigger workflows and letting workflows trigger pipelines on a branch or tag, rerun failed workflows, cancel stuck runs, approve on-hold jobs, fetch test results, and run read-only AI investigations across pipelines, workflows, and jobs. Events are delivered by CircleCI's **native outbound webhooks** (`workflow-completed` and `job-completed`), signed with HMAC-SHA256 (`circleci-signature` header) using the webhook secret Kestrel generates at connect time. Kestrel verifies the signature before evaluating workflows. ## Prerequisites * **Organization Admin** role in Kestrel * A CircleCI organization with at least one project * A CircleCI **personal API token** for a user that follows the projects your workflows will manage ## Setup In CircleCI (`app.circleci.com`), go to **User Settings → Personal API Tokens** and click **Create New Token**. Give it a name (e.g. `kestrel`) and copy the token — it is shown only once. The token is sent as the `Circle-Token` header to the CircleCI v2 API. Kestrel sees the projects the token's user **follows**, so follow the projects you want available in workflows. 1. Navigate to **Integrations → CircleCI** in your Kestrel dashboard. 2. Paste your **API token**. 3. Optionally set a custom **API base URL** for self-hosted CircleCI server installs (defaults to `https://circleci.com/api/v2`). 4. Click **Connect CircleCI** — Kestrel validates the token by listing your followed projects and generates a **webhook secret** for signed deliveries. Actions work as soon as the token is connected. To have CircleCI events **trigger** workflows, add an outbound webhook to each project you care about: 1. On the Kestrel CircleCI integration page, copy the **Webhook URL** and the **Webhook secret**. 2. In each CircleCI project, open **Project Settings → Webhooks → Add Webhook**. 3. Set the **Receiver URL** to the copied webhook URL and paste the webhook secret as the **Secret token**. 4. Select the **workflow-completed** and/or **job-completed** events and save. The API token can trigger pipelines and approve gated jobs that deploy real infrastructure. Treat it as a secret — Kestrel stores it encrypted. Gate destructive workflow actions (Trigger Pipeline on production branches, Approve On-Hold Job) behind Approval nodes. Project slugs use the `vcs/org/repo` form, e.g. `gh/acme/api`. The project dropdowns in the workflow builder list your followed projects automatically. ## How It's Used ### In Workflows **Trigger blocks (webhook):** * **Workflow Failed** — fires when a workflow completes with `failed` or `error` status * **Workflow Succeeded** — fires when a workflow completes successfully * **Workflow Completed (any status)** — fires when a workflow finishes with any status * **Job Failed** — fires when a job completes with `failed` status (finer-grained than workflow-completed) Filter triggers by project slug, branch, and status. Trigger events expose template variables such as `{{signal.project_slug}}`, `{{signal.pipeline_id}}`, `{{signal.workflow_id}}`, `{{signal.workflow_name}}`, `{{signal.job_name}}`, `{{signal.job_number}}`, `{{signal.status}}`, and `{{signal.branch}}` for downstream steps. **Action blocks:** * **Trigger Pipeline** — trigger a pipeline on a branch or tag, optionally with pipeline parameters (one `KEY=VALUE` per line) * **Wait for Pipeline** — poll a pipeline's workflows until they all finish and branch on the overall result * **Get Workflow Status** — fetch a pipeline's workflow statuses without waiting * **Rerun Workflow** — rerun a workflow, optionally only its failed jobs (the go-to auto-remediation for flaky builds) * **Cancel Workflow** — cancel a running workflow (stop stuck or unwanted runs to free credits) * **Approve On-Hold Job** — approve an on-hold approval job so a gated workflow can proceed (e.g. auto-approve staging deploys after checks pass; gate behind an Approval node for production) * **Get Job Test Results** — fetch a job's test results with a failure summary (great for Slack posts or AI analysis) * **Investigate CircleCI** — run a read-only AI investigation across pipelines, workflows, jobs, and test results Project and pipeline selects use dynamic dropdowns backed by the CircleCI API and accept template variables like `{{signal.project_slug}}` and `{{signal.workflow_id}}` from the trigger. Example (CI failure RCA — the flagship CircleCI flow): When a **workflow fails** on `main`, fetch the failed job's test results, run an **AI investigation**, then post the root cause to Slack `#ci` and open a Jira ticket. Example (flaky-build remediation): When a workflow fails, **rerun from failed** once automatically; if the rerun also fails, page on-call via PagerDuty with the investigation summary. Example (gated deploys): When a staging workflow succeeds, run smoke checks, then **approve the on-hold** production deploy job — or route a Slack approval first. ## Disconnecting 1. Navigate to **Integrations → CircleCI** 2. Click **Disconnect** 3. Confirm the disconnection This stops all CircleCI workflow triggers and actions. Webhooks configured in CircleCI will keep sending deliveries until you remove them there; deliveries with an invalid signature are rejected. You can reconnect at any time. # ClickHouse Source: https://docs.usekestrel.ai/integrations/clickhouse Connect ClickHouse Cloud for service lifecycle (start/stop/scale), IP access list, backups, ClickPipes ingestion, usage costs, error-spike detection, and AI investigation in workflows The ClickHouse integration connects Kestrel to your ClickHouse Cloud organization through the ClickHouse Cloud API, enabling control-plane events (service state, scaling, backups, ClickPipes, query errors, and spend) to trigger workflows and giving AI agents read-only access to your services' state. ClickHouse Cloud has no control-plane webhooks (its notifications are email/console-only, and ClickStack alert webhooks are observability alerts, not service/backup/ClickPipe events), so Kestrel polls the ClickHouse Cloud API on a per-tenant cadence to detect service state changes, scaling and version changes, backup completions/failures, ClickPipe failures, failed-query spikes, and spend thresholds. ## Prerequisites * **Organization Admin** role in Kestrel * A ClickHouse Cloud organization with at least one service * A ClickHouse Cloud **API key** (key ID + secret) with the **Admin** role, created in the ClickHouse Cloud console ## Setup In the ClickHouse Cloud console (`console.clickhouse.cloud`), click your **organization** name in the bottom-left to open the organization menu, choose **API keys**, then click **New API key**. Give it a name (e.g. `kestrel`) and select the **Admin** role (Developer keys are read-only and cannot start/stop/scale services). Copy the generated **Key ID** and **Key secret** (shown only once). The key ID and secret are sent as HTTP Basic auth to the ClickHouse Cloud API at `api.clickhouse.cloud` to read and manage services, backups, and ClickPipes. 1. Navigate to **Integrations → ClickHouse** in your Kestrel dashboard. 2. Paste your **Key ID** and **Key secret**. 3. Optionally enter your **Organization ID** — a ClickHouse Cloud API key belongs to exactly one organization, so Kestrel auto-detects it when left blank. 4. Choose a **Poll Interval** (how often Kestrel checks the ClickHouse Cloud API for service/backup/ClickPipe/error/spend events). 5. Click **Connect ClickHouse** — your ClickHouse Cloud organization appears as connected. The API key grants access to your ClickHouse Cloud services and their control plane. Treat it as a secret — Kestrel stores it encrypted. All ClickHouse triggers are detected by polling, so there may be up to one poll interval of delay before a workflow fires. The minimum poll interval is 60 seconds. ## How It's Used ### In Workflows **Trigger blocks (all poll-based):** * **Service State Changed** — fires when a service changes state (running, idle, stopped, awaking, ...) * **Service Idle** — fires when a service scales to idle/stopped (cost signal) * **Service Scaled** — fires when a service's replica count or memory limits change * **Backup Completed** / **Backup Failed** — fires when a service backup reaches a terminal state * **Query Error Spike** — fires when failed queries/inserts (Prometheus counters) increase past a configurable threshold within a poll window * **ClickPipe Failed** — fires when a managed ingestion pipeline enters a failed state * **Version Changed** — fires when a service's ClickHouse version changes after an upgrade * **Usage/Spend Threshold** — fires when month-to-date spend (ClickHouse Credits) crosses a percent of a configured monthly budget * **Too Many Parts** — fires when a partition's max part count exceeds a threshold (the classic "too many parts" incident) * **High Query Concurrency** — fires when concurrent running queries exceed a threshold Filter triggers by service ID and event type; the error-spike threshold, monthly budget, and usage percent are configurable per trigger. **Action blocks:** * **List Services** / **Get Service** — inventory (state, scaling, version, endpoints, IP access list) * **Create Service** — provision a service, e.g. a self-serve staging analytics DB (gate behind an Approval node) * **Start Service** / **Stop Service** — the cost levers: resume before working hours, stop non-prod services overnight * **Update Autoscaling** — per-replica memory limits, replica count, idle scaling, idle timeout * **Update IP Access List** — add/remove CIDR entries (security) * **List Backups** / **Get Backup** / **Update Backup Configuration** — backup visibility and schedule * **Restore Backup** — restore by provisioning a new service from a backup (the API has no in-place restore; gate behind an Approval node) * **Delete Service** — tear down a stopped service (gate behind an Approval node) * **List API Keys** — key hygiene (metadata only; secrets are never returned) * **Get Service Metrics** — fetch failed-query Prometheus counters during RCA * **Get Usage Costs** — org spend report in ClickHouse Credits * **Get / Set / Clear Autoscaling Schedule** — native scheduled autoscaling (business-hours scale-down; the race-free cost lever) * **Get / Set / Clear Upgrade Window** — control when ClickHouse version upgrades occur (maintenance governance) * **Get / Update / Reset ClickHouse Setting** — tune server/query settings, e.g. `max_concurrent_queries` (Beta API; some settings trigger a rollout restart) * **Get / Upsert / Delete Query Endpoint** — manage the query API endpoint (roles, allowed origins) * **List Members** / **Remove Member** — org access review and offboarding (gate remove behind Approval) * **List Roles** — org role inventory for access review * **List Activity** — org audit/activity log over a recent window (security RCA) * **List ClickPipes** / **Get ClickPipe** / **Start ClickPipe** / **Stop ClickPipe** / **Resync ClickPipe** / **Scale ClickPipe** — manage managed ingestion (Kafka, S3, CDC) * **Investigate ClickHouse** — run an AI investigation of services, backups, ClickPipes, metrics, and costs ClickHouse service, ClickPipe, and backup selects accept template variables like `{{signal.service_id}}`, `{{signal.clickpipe_id}}`, and `{{signal.backup_id}}` from the trigger. Example (cost — the flagship ClickHouse flow): On a weeknight schedule, stop non-production ClickHouse services and post a summary to Slack; on a morning schedule, start them again. Example (incident): When a query-error spike or ClickPipe failure is detected, run an AI investigation, page on-call via PagerDuty, and open a Jira issue with the findings. Example (provisioning): A "staging ClickHouse service" developer request gates behind an Approval, then creates the service and posts the endpoint host to Slack. ## Disconnecting 1. Navigate to **Integrations → ClickHouse** 2. Click **Disconnect** 3. Confirm the disconnection This stops all ClickHouse workflow triggers. You can reconnect at any time. # Cloudflare Source: https://docs.usekestrel.ai/integrations/cloudflare Connect Cloudflare for DNS, WAF, cache, Workers, load balancer, and tunnel automation, plus security event and health check triggers in workflows The Cloudflare integration connects Kestrel to your Cloudflare account, enabling security events, health check failures, Worker deployment failures, and SSL expirations to trigger workflows — and giving AI agents the ability to manage DNS records, WAF and firewall rules, cache purges, Workers, load balancer pools, tunnels, and Access policies. ## Prerequisites * **Organization Admin** role in Kestrel * Cloudflare account with permission to create API tokens ## Setup In the Cloudflare dashboard, go to **My Profile → API Tokens → Create Token → Custom Token** and grant the following permissions: | Scope | Permission | | -------------------------------------------- | ---------- | | Zone → Zone | Read | | Zone → DNS | Edit | | Zone → Firewall Services | Edit | | Zone → WAF | Edit | | Zone → Cache Purge | Purge | | Zone → Page Rules | Edit | | Zone → Zone Settings | Edit | | Account → Workers Scripts | Edit | | Account → Load Balancing: Monitors and Pools | Edit | | Account → Cloudflare Tunnel | Edit | | Account → Access: Apps | Edit | Set **Account Resources** to your account and **Zone Resources** to all zones (or the specific zones you want Kestrel to manage). You can omit permissions for features you don't plan to automate. Your Account ID is in the dashboard URL: `dash.cloudflare.com/[account_id]/home`. 1. Navigate to **Integrations → Cloudflare** in your Kestrel dashboard 2. Click **Connect** 3. Paste the API token and enter the Account ID 4. Save — your Cloudflare account appears as connected. To use Cloudflare trigger blocks, configure Cloudflare Notifications to send events to Kestrel: 1. Copy the **webhook URL** and **webhook secret** shown on the Cloudflare integration page 2. In the Cloudflare dashboard, go to **Notifications → Destinations → Webhooks → Create**, paste the URL, and use the secret as the secret value 3. Go to **All Notifications → Add**, select an alert type, and choose the webhook as the destination Supported alert types: **HTTP DDoS Attack**, **WAF Attack**, **Firewall Events**, **Health Check Status**, **Workers Event**, and **SSL Certificate Expiration**. Actions work without any notification setup — webhooks are only needed for Cloudflare trigger blocks. ## How It's Used ### In Workflows **Trigger blocks:** * **Security Event** — fires on WAF events, DDoS attacks, or firewall rule matches * **Health Check Failure** — fires when a Cloudflare health check detects an origin is down * **Worker Deployment Failure** — fires when a Workers deployment fails * **SSL Certificate Expiring** — fires when an SSL certificate is approaching expiration Filter triggers by zone and event type. **Action blocks:** * **DNS** — create, update, delete, and list DNS records * **Cache** — purge everything or purge selectively by URL, tag, or host * **Security** — create firewall rules, block IPs and ASNs, toggle WAF rulesets, update rate limits, update Access policies * **Workers** — deploy a Worker, roll back to a previous version, get Worker logs * **Load Balancing** — enable/disable pools, update pool weights * **Tunnels** — create and delete Cloudflare Tunnels * **Other** — create health checks, create page rules Zone, record, Worker, pool, and tunnel selects are populated dynamically from your account, and accept template variables like `{{signal.zone_name}}` from the trigger. Example: A workflow triggers on a WAF attack, blocks the offending IP, toggles a stricter WAF ruleset, and posts a summary to Slack for approval before reverting. ## Disconnecting 1. Navigate to **Integrations → Cloudflare** 2. Click **Disconnect** 3. Confirm the disconnection This disables all Cloudflare triggers and actions. You can reconnect at any time. # Confluence Source: https://docs.usekestrel.ai/integrations/confluence Connect Confluence for publishing runbooks, postmortems, and using wiki pages as incident knowledge The Confluence integration connects Kestrel to your Atlassian Confluence instance, enabling automated publishing of RCA reports, postmortems, and runbooks from workflows, plus historical knowledge search during incident investigations. ## Prerequisites * **Organization Admin** role in Kestrel * Atlassian account with Confluence access ## Setup 1. Navigate to **Integrations → Confluence** in your Kestrel dashboard 2. Click **Connect** 3. Authorize Kestrel in the Atlassian OAuth popup 4. Select the Confluence site to connect (if you have multiple) 5. The popup closes and your Confluence instance appears as connected If you've already connected Jira on the same Atlassian site, Confluence may be auto-detected. Check the Confluence integration card for a pre-authorized connection. ## How It's Used ### In Workflows **Action blocks:** * **Publish RCA** — automatically publish a root cause analysis report to a designated Confluence space * **Publish Postmortem** — generate and publish a structured postmortem document * **Publish Runbook Entry** — create or update a runbook entry with workflow-generated content * **Update Page** — modify an existing Confluence page (append, prepend, or replace content) Example: A workflow triggered by an incident resolution can automatically generate a postmortem from the incident timeline, RCA findings, and remediation steps, and publish it to your team's Confluence space. ### In Incident Response * **Knowledge source** — during root cause analysis, Kestrel searches Confluence for relevant runbooks, architecture docs, past postmortems, and team documentation about affected services (see [Knowledge Sources](/integrations/knowledge-sources)) * **Postmortem publishing** — generate and publish structured postmortems directly from the incident detail page ### In Cloud AI Copilot * Search documentation: `Find the runbook for the payments service` * Publish content: `Create a Confluence page summarizing today's incident` ## Configuration ### Knowledge Source Setup To use Confluence as a knowledge source during incident investigations: 1. Navigate to **Integrations → Knowledge Sources** 2. Click **Add Source** and select **Confluence** 3. Your connected Confluence instance is auto-detected 4. Optionally restrict search to specific spaces 5. Click **Enable** Kestrel will search Confluence pages and blog posts for relevant historical context during incident RCA. ## Disconnecting 1. Navigate to **Integrations → Confluence** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Confluence workflow actions and removes Confluence as a knowledge source. You can reconnect at any time. # Datadog Source: https://docs.usekestrel.ai/integrations/datadog Query metrics, events, hosts, and logs from your in-cluster Datadog installation for incident investigation The Datadog integration enables the Kestrel Operator to query historical metrics, events, host information, and logs directly from the Datadog API. This gives Kestrel's AI agents access to your existing Datadog data for incident investigation, root cause analysis, and infrastructure questions. The operator auto-discovers Datadog in your cluster and authenticates using existing Kubernetes secrets. No manual key configuration is required. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * Datadog Agent deployed via Helm chart or Datadog Operator * Datadog **API Key** and **Application Key** stored in Kubernetes secrets An Application Key is required for all Datadog API queries. If your Datadog installation does not include one, generate it from the [Datadog Application Keys page](https://app.datadoghq.com/organization-settings/application-keys) and add it to your Datadog secret. ## Setup During cluster onboarding in the Kestrel Dashboard, select **Datadog** as the Metrics Source and specify the namespace where Datadog is deployed (defaults to `datadog`). Alternatively, configure it via Helm values: ```yaml theme={null} operator: datadog: enabled: true namespace: "datadog" ``` Check the operator logs for successful Datadog discovery: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "Datadog" ``` A successful discovery shows: ``` [Datadog] Discovery complete namespace=datadog secret=datadog site=datadoghq.com has_api_key=true has_app_key=true ``` The operator performs cluster-wide discovery as a fallback, so it will find Datadog even if the configured namespace differs from the actual deployment. ### Optional Overrides For non-standard Datadog deployments: ```yaml theme={null} operator: datadog: enabled: true namespace: "custom-monitoring" secretName: "my-datadog-secret" site: "datadoghq.eu" ``` ## How It's Used ### In Workflows **Trigger blocks:** * **Datadog Monitor Alert** — triggers a workflow when a Datadog monitor enters Alert state * **Datadog Monitor Warning** — triggers when a monitor enters Warn state * **Datadog Monitor Recovered** — triggers when a monitor returns to OK state * **Datadog Monitor No Data** — triggers when a monitor reports No Data * **Datadog Monitor (Any Transition)** — triggers on any monitor state change The operator polls Datadog monitor statuses every 60 seconds and detects transitions automatically. No webhook configuration needed — the operator uses its existing Datadog API credentials. **Action blocks:** * **Query Metrics** — query Datadog time-series metrics with custom queries and time ranges * **Create Monitor** — create a new Datadog monitor with specified thresholds and alerting conditions * **Send Event** — post a custom event to the Datadog event stream * **Mute Monitor** — mute a Datadog monitor for a specified duration Example: A workflow triggered by a Datadog monitor entering Alert state queries additional metrics for context, runs an AI investigation, and posts the findings to Slack with remediation steps. ### In Incident Response * **Metrics correlation** — during root cause analysis, Kestrel queries Datadog for CPU, memory, network, and custom metrics around the incident timeframe * **Event context** — surfaces Datadog events (deployments, alerts) that coincide with incidents * **Log search** — searches Datadog logs for error patterns related to affected workloads ### In Cloud AI Copilot Ask questions that leverage your Datadog data: * `What was the CPU usage for the payments pod over the last hour?` * `Show me Datadog events for the production cluster today` * `Search Datadog logs for timeout errors in the API service` ## Supported Query Types | Query Type | Description | Datadog API Endpoint | | ----------- | ----------------------------------------------------- | --------------------------------- | | **Metrics** | Historical time series (CPU, memory, network, custom) | `GET /api/v1/query` | | **Events** | Infrastructure events, alerts, deployments | `GET /api/v1/events` | | **Hosts** | Monitored hosts with metadata | `GET /api/v1/hosts` | | **Logs** | Application and infrastructure log search | `POST /api/v2/logs/events/search` | ## RBAC When `operator.datadog.enabled` is set to `true`, the Helm chart creates namespace-scoped RBAC resources (Role + RoleBinding) in the Datadog namespace, granting: * **Read access** to Datadog secrets (API key discovery) * **List access** to DaemonSets and Deployments (workload discovery) No cluster-wide secret access is granted. ## Troubleshooting | Issue | Solution | | -------------------------------- | ------------------------------------------------------------------------------- | | "No Datadog workloads found" | Verify Datadog is deployed: `kubectl get daemonset -A \| grep datadog` | | "Secret missing 'api-key' field" | Check the secret exists and contains `api-key` and `app-key` fields | | Queries return no data | Verify the Datadog site matches your account region (e.g., `us5.datadoghq.com`) | # Daytona Source: https://docs.usekestrel.ai/integrations/daytona Connect Daytona for sandbox, snapshot, and volume lifecycle, command execution, and AI investigation in workflows The Daytona integration connects Kestrel to your Daytona organization, enabling sandbox, snapshot, and volume lifecycle events to trigger workflows and giving AI agents access to sandbox state, snapshots, and volumes. Daytona delivers sandbox, snapshot, and volume lifecycle events to Kestrel through a webhook you register in the Daytona dashboard. Kestrel verifies every delivery against the webhook's signing secret before it can fire a workflow. ## Prerequisites * **Organization Admin** role in Kestrel * Daytona account with an organization * A Daytona organization API key * Webhooks enabled for your Daytona organization ## Setup In the Daytona dashboard, open **API Keys** in the left sidebar and click **Create API Key** (or use the CLI: `daytona api-key create`). Then: 1. Enter a **Key Name** (e.g. `kestrel`). 2. Optionally set an **Expires** date — leave blank for a non-expiring key, or pick a date and plan to rotate the key in Kestrel before it expires. 3. For **Permissions**, choose **Full Access** (simplest), or choose **Restricted** and grant **Read + Write + Delete** on **Sandboxes** and **Snapshots** plus **Read** on **Volumes**. The **Sandboxes** preset alone is not sufficient — snapshot and volume actions require those additional grants. 4. Click **Create** and copy the generated key (shown only once). The key is sent as a bearer token to `https://app.daytona.io/api` to read sandbox, snapshot, and volume state and perform actions. 1. In the Daytona dashboard, open **Webhooks**. If webhooks aren't enabled yet, click **Enable webhooks**. 2. Click **Create Endpoint**, give it a name, and set the **Endpoint URL** to the Kestrel webhook URL shown on the **Integrations → Daytona** connect screen (it ends in `/api/webhooks/daytona`). 3. For **Events**, select all the sandbox, snapshot, and volume events, then click **Create**. 4. Click the newly created endpoint to open its details and copy its **Signing Secret**. 1. Navigate to **Integrations → Daytona** in your Kestrel dashboard. 2. Paste your **Daytona API key**. 3. (Optional) Set a custom **API URL** for self-hosted Daytona. 4. Paste the webhook **Signing Secret** from the previous step. 5. Click **Connect Daytona** — your Daytona organization appears as connected. The API key grants access to your Daytona sandboxes, snapshots, and volumes. Treat it and the signing secret as secrets — Kestrel stores both encrypted. Kestrel only fires Daytona triggers from webhook deliveries whose signature matches the signing secret you provided. To rotate the secret, update it in Daytona, then disconnect and reconnect in Kestrel with the new value. ## How It's Used ### In Workflows **Trigger blocks:** * **Sandbox Created** — fires when a new sandbox is created * **Sandbox Stopped** — fires when a sandbox transitions out of the started state * **Sandbox Error** — fires when a sandbox enters an error or build-failed state * **Sandbox Archived** — fires when a sandbox is archived * **Sandbox Execution Failed** — fires when a command run in a sandbox via a workflow exits non-zero * **Snapshot Build Failed** — fires when a snapshot enters an error/build-failed state * **Volume Error** — fires when a volume enters an error state Filter triggers by sandbox and event type. Kestrel receives these as webhook deliveries from Daytona. **Action blocks:** * **List Sandboxes** — list the sandboxes in the organization * **Create Sandbox** — create a sandbox from a prebuilt snapshot or a Docker image (image references must carry a specific version tag or digest, e.g. `python:3.12` — untagged images and `latest`/`lts`/`stable` are rejected by Daytona) * **Get Sandbox** — retrieve a sandbox's state and metadata * **Start Sandbox** — start a stopped sandbox * **Stop Sandbox** — stop a running sandbox (preserves state for restart) * **Archive Sandbox** — archive a stopped sandbox to free compute while retaining disk * **Delete Sandbox** — permanently delete a sandbox (gate behind an Approval node) * **Run Command in Sandbox** — execute a shell command in a sandbox and capture exit code and output * **Set Auto-Stop Interval** — set a sandbox's inactivity auto-stop interval for cost control * **List Snapshots** — list the snapshots in the organization * **Create Snapshot** — create a snapshot from a Docker image (the image must carry a specific version tag or digest) * **Delete Snapshot** — delete a snapshot by name or ID * **List Volumes** — list the volumes in the organization * **Get Volume** — retrieve a volume's state * **Create Volume** — create a volume for sharing data across sandboxes * **Delete Volume** — permanently delete a volume (gate behind an Approval node) * **Investigate Daytona** — run an AI investigation of failing sandboxes, snapshots, and volumes Daytona sandbox, snapshot, and volume selects accept template variables like `{{signal.sandbox_id}}`, `{{signal.snapshot}}`, and `{{signal.volume}}` from the trigger. Create Sandbox outputs `sandbox_id`, `sandbox_name`, and `sandbox_state`, so downstream blocks can reference a just-created sandbox with `{{step_outputs..sandbox_id}}`; Create Volume similarly outputs `volume_id`, `volume_name`, and `volume_state`. Example: A workflow triggers on a sandbox error, runs an AI investigation to diagnose the failure, posts a summary to Slack, and opens a Jira bug ticket. Example: A workflow triggers on a newly created sandbox and sets an auto-stop interval to keep idle compute costs down. ## Disconnecting 1. Navigate to **Integrations → Daytona** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Daytona workflow triggers. You can optionally remove the webhook endpoint in the Daytona dashboard too. You can reconnect at any time. # Flux CD Source: https://docs.usekestrel.ai/integrations/fluxcd Connect Flux CD for GitOps actions — reconcile, suspend, and resume Kustomizations, HelmReleases, and sources in workflows The Flux CD integration connects Kestrel to your in-cluster Flux installation, enabling AI agents to trigger reconciliations, suspend or resume GitOps resources, and inspect reconciliation status as part of automated workflows and incident response. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * Flux controllers and CRDs installed in the same cluster (`kustomize.toolkit.fluxcd.io`, `helm.toolkit.fluxcd.io`, `source.toolkit.fluxcd.io`) ## Setup Flux CD is auto-detected by the Kestrel Operator when the Flux CRDs are present in the cluster. No manual connection or credentials are required. If Flux is not already installed, bootstrap it with the Flux CLI: ```bash theme={null} flux bootstrap github \ --owner= \ --repository= \ --path=clusters/my-cluster ``` The operator probes for the Flux CRDs periodically. Check the operator logs: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "Flux" ``` Once detected, the integration shows as **Detected** on the Integrations page and Flux action blocks become available in the workflow builder. The integration is enabled by default. To opt out, set `operator.flux.enabled: false` in your Helm values and upgrade the operator (this sets `FLUX_DISABLED=true` on the operator deployment). ## How It's Used ### In Workflows **Action blocks:** * **Reconcile Flux Resource** — trigger an immediate reconciliation of a Kustomization, HelmRelease, GitRepository, HelmRepository, or OCIRepository (optionally reconciling its source first) * **Suspend Flux Resource** — suspend reconciliation of a resource (e.g., to freeze deployments during an incident) * **Resume Flux Resource** — resume reconciliation of a suspended resource * **Get Flux Resource Status** — retrieve ready condition, last applied revision, and suspension state * **Wait for Flux Ready** — block workflow execution until a resource reports Ready (with configurable timeout) * **Get Flux Events** — fetch recent Kubernetes events for a Flux resource to aid debugging Example: A workflow triggered by a PagerDuty incident suspends the affected HelmRelease to stop Flux from re-applying a bad change, notifies the team in Slack, and resumes reconciliation once an approval gate is passed. ### Scoping Flux actions are scoped by **cluster**, **resource kind**, **namespace**, and **resource name** — all populated with dropdowns from live cluster data in the workflow builder. ## Disconnecting To disable the Flux CD integration, set the following in your Helm values and upgrade the operator: ```yaml theme={null} operator: flux: enabled: false ``` # Fly.io Source: https://docs.usekestrel.ai/integrations/flyio Connect Fly.io for machine lifecycle triggers, secrets access, and machine actions in workflows The Fly.io integration connects Kestrel to your Fly.io account, enabling Machine lifecycle events to trigger workflows and giving AI agents access to machine state, logs, secrets, and lifecycle actions across your apps. Unlike most integrations, Fly.io does not send outbound webhooks. Kestrel detects Machine lifecycle events by polling the Fly Machines API (`https://api.machines.dev`) on a configurable interval. ## Prerequisites * **Organization Admin** role in Kestrel * Fly.io account with app access * `flyctl` installed (to mint a token) ## Setup Create an org-scoped token so Kestrel can read machine state and perform machine actions across all apps in the org: ```bash theme={null} fly tokens create org ``` A deploy token also works for a single app. Copy the token value. ```bash theme={null} fly orgs list ``` Use the **slug**, not the display name. Defaults to `personal`. 1. Navigate to **Integrations → Fly.io** in your Kestrel dashboard 2. Click **Connect** 3. Paste the API token and enter the organization slug. The token is sent as a Bearer token to `https://api.machines.dev`. 4. Choose a **Poll Interval** — faster intervals detect incidents sooner; the minimum is 30 seconds. 5. Save — your Fly.io account appears as connected. No webhook setup is required — there is nothing to configure inside Fly.io. Just keep the integration connected and Kestrel will poll for Machine lifecycle events. ## How It's Used ### In Workflows **Trigger blocks:** * **Machine Stopped** — fires when a Machine transitions to the stopped state * **Machine Failed / Crashed** — fires when a Machine enters a failed state * **Machine Started** — fires when a Machine starts * **Machine Created** — fires when a new Machine is created * **Machine Destroyed** — fires when a Machine is destroyed Filter triggers by app and event type. Kestrel polls the Fly Machines API on your configured interval to detect these transitions. **Action blocks:** * **Get Machine** — retrieve a Machine's state, config, and metadata * **Start Machine** — start a stopped Machine * **Stop Machine** — stop a running Machine * **Restart Machine** — restart a Machine * **List Machines** — list Machines for an app * **Get App** — retrieve app details * **Set Secrets** — set app secrets * **Investigate Fly.io** — run an AI investigation combining machine state, logs, and lifecycle history Fly.io app and machine selects accept template variables like `{{signal.app_name}}` and `{{signal.machine_id}}` from the trigger. Example: A workflow triggers when a Machine crashes, fetches its state and logs, runs an AI investigation, restarts the Machine, and posts a summary to Slack. ## Disconnecting 1. Navigate to **Integrations → Fly.io** 2. Click **Disconnect** 3. Confirm the disconnection This stops Fly.io polling and all Fly.io workflow triggers. You can reconnect at any time. # GitHub Source: https://docs.usekestrel.ai/integrations/github Connect GitHub for GitOps deployments, causal PR detection, and IaC remediation via pull requests The GitHub integration connects Kestrel to your GitHub repositories, enabling GitOps deployments, Infrastructure-as-Code remediation, and causal PR detection during incident investigation. ## Prerequisites * **Organization Admin** role in Kestrel * GitHub organization admin access ## Setup 1. Navigate to **Integrations → CI/CD** in your Kestrel dashboard 2. Click **Connect GitHub** 3. Select your GitHub organization in the installation page 4. Choose which repositories to grant access to (all or selected) 5. Click **Install & Authorize** To enable IaC remediation on a repository: 1. Expand the GitHub card to view your repositories 2. Click **Configure IaC** on the target repository 3. Select the IaC type: **Terraform**, **CloudFormation**, or **Pulumi** 4. Set the root path (e.g., `/` or `/terraform`) 5. Optionally link a cloud account (AWS or OCI) to match resources to the correct repository 6. Click **Enable IaC** ### Managing Repository Access Repository access is managed through your GitHub App installation settings: 1. Go to your GitHub organization settings 2. Navigate to **Installed GitHub Apps → Kestrel AI** 3. Update the repository selection ## How It's Used ### In Workflows **Action blocks:** * **Create Pull Request** — open a pull request with specified branch, title, and body * **Create Issue** — create a GitHub issue with labels, assignees, and body content * **Trigger GitHub Action** — trigger a GitHub Actions workflow run on a specified branch * **Wait for GitHub Action Run** — block workflow execution until a GitHub Actions run completes * **Get GitHub Action Status** — retrieve the current status of a GitHub Actions workflow run * **Read File** — read a file from a repository at a specified ref * **Search Code** — search repository code for patterns or keywords * **AI Code Investigation** — run an AI-powered investigation across repository code * **AI Code Fix** — generate an AI-powered code fix and commit it to a branch * **Wait for PR Approval** — block workflow execution until a pull request is approved * **Wait for PR Merge** — block workflow execution until a pull request is merged * **Investigate GitHub Action Failure** — fetch failed job logs from a GitHub Actions run and use AI to analyze the root cause, providing a summary of what failed and suggested fixes Example: A workflow triggered by a Kubernetes incident searches the repo for the affected service code, runs an AI investigation, creates a fix PR, and waits for approval before merging. ### In Incident Response * **Causal PR detection** — during root cause analysis, Kestrel identifies recent PRs/deployments that may have caused the incident * **YAML fix PRs** — click **Create Pull Request** on any AI-generated Kubernetes fix to open a PR with the change * **IaC remediation** — when Kestrel detects a cloud misconfiguration, it identifies the Terraform file managing that resource and opens a PR with the fix * **Knowledge source** — repository content provides context during incident investigation ### In Cloud AI Copilot * Deploy AI-generated configurations via pull request from the chat interface * Ask about recent deployments: `What PRs were merged to production in the last 24 hours?` * Browse repository contents for IaC and Kubernetes manifests ## IaC Remediation Workflow For cloud incidents with connected IaC repositories: 1. Kestrel detects a cloud misconfiguration (e.g., S3 bucket with public access) 2. Kestrel identifies the Terraform file managing that resource across all connected repositories 3. Kestrel generates a Terraform fix 4. Click **Create Pull Request** to open a PR with the remediation 5. The PR includes the full diff, fix description, and a link back to the incident ## Disconnecting To modify or remove the integration, manage the GitHub App installation from your GitHub organization settings. # GitLab Source: https://docs.usekestrel.ai/integrations/gitlab Connect GitLab for GitOps deployments and IaC remediation via merge requests The GitLab integration connects Kestrel to your GitLab projects, enabling GitOps deployments and Infrastructure-as-Code remediation through merge requests. ## Prerequisites * **Organization Admin** role in Kestrel * GitLab account with access to the target projects ## Setup 1. Navigate to **Integrations → CI/CD** in your Kestrel dashboard 2. Click **Connect GitLab** 3. Authorize Kestrel in the GitLab OAuth page 4. You're redirected back to the Kestrel dashboard 1. Click the GitLab card to expand it 2. Click **Add Projects** 3. Select the GitLab projects you want to manage 4. Click **Sync Projects** 1. Click **Configure IaC** on a synced GitLab project 2. Select the IaC type: **Terraform**, **CloudFormation**, or **Pulumi** 3. Set the root path 4. Optionally link a cloud account (AWS or OCI) 5. Click **Enable IaC** ## How It's Used ### In Workflows **Action blocks:** * **Create Merge Request** — open a merge request with specified source branch, title, and description * **Create Issue** — create a GitLab issue with labels, assignees, and description * **Trigger Pipeline** — trigger a CI/CD pipeline run on a specified branch * **Wait for Pipeline** — block workflow execution until a pipeline run completes * **Get Pipeline Status** — retrieve the current status of a CI/CD pipeline * **Wait for MR Approval** — block workflow execution until a merge request is approved * **Wait for MR Merge** — block workflow execution until a merge request is merged Example: A workflow triggered by an incident creates a merge request with a fix, triggers the CI pipeline, waits for it to pass, and then waits for MR approval before notifying the team. ### In Incident Response * **YAML fix MRs** — click **Create Merge Request** on any AI-generated Kubernetes fix to open an MR with the change * **IaC remediation** — when Kestrel detects a cloud misconfiguration, it identifies the Terraform file managing that resource and opens an MR with the fix * **Knowledge source** — repository content provides context during incident investigation ### In Cloud AI Copilot * Deploy AI-generated configurations via merge request from the chat interface * Browse repository contents for IaC and Kubernetes manifests ## Disconnecting Disconnecting GitLab removes all synced projects and their IaC configurations. This action cannot be undone. 1. Click the GitLab card to expand it 2. Click **Disconnect** 3. Confirm the disconnection # Glean Source: https://docs.usekestrel.ai/integrations/glean Connect Glean for enterprise-wide knowledge search during incident investigations The Glean integration connects Kestrel to your organization's Glean instance, enabling enterprise-wide knowledge search during incident root cause analysis. Glean aggregates knowledge from across your tools — Confluence, Google Drive, Slack, Notion, and more — providing a single search interface for incident context. ## Prerequisites * **Organization Admin** role in Kestrel * Glean Enterprise plan with API access enabled * Glean API key with `search:read` scope ## Setup 1. Contact your Glean administrator to enable API access 2. Go to your **Glean Admin Console → API → API Keys** 3. Click **Create API Key** 4. Select the `search:read` scope 5. Copy the generated API key 1. Navigate to **Integrations → Knowledge Sources** in your Kestrel dashboard 2. Click **Add Source** and select **Glean** 3. Paste the API key 4. Optionally enter a **Workspace ID** 5. Click **Add Source** Glean API access requires an Enterprise plan. Contact your Glean account manager if you don't see the API option in the admin console. ## How It's Used ### In Incident Response * **Enterprise knowledge search** — during root cause analysis, Kestrel searches Glean for relevant documents, past incidents, runbooks, and team discussions across all of your organization's connected tools * **Broad context** — because Glean indexes content from multiple sources (Confluence, Drive, Slack, Notion, etc.), it often surfaces context that wouldn't be found by searching individual tools Glean results are presented alongside Kestrel's automated investigation, giving you both real-time diagnostics and historical insight. ### In Cloud AI Copilot * Search your organization's knowledge base from the chat: `Search Glean for the payments service architecture` Glean is most valuable when you have knowledge spread across many tools. If your team primarily uses Confluence and Slack, connecting those directly as knowledge sources may be sufficient. ## Managing the Connection ### Testing Click the **Test** button on the Glean source to verify credentials and connectivity. ### Enabling/Disabling Use the toggle switch on the Glean source to temporarily enable or disable it without removing the configuration. ### Deleting Click the delete icon and confirm to permanently remove the Glean source. # Helm Source: https://docs.usekestrel.ai/integrations/helm Install, upgrade, rollback, and manage Helm releases as part of automated workflows The Helm integration enables workflow blocks that manage Helm releases on your Kubernetes clusters. The operator executes Helm commands directly using the `helm` binary installed in the operator pod. No additional setup is required — Helm is available on all clusters where the Kestrel Operator is deployed. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * For private chart repositories: Helm repo credentials configured in the operator (via init container or Helm values) ## Setup Helm is automatically available when the Kestrel Operator is deployed. No manual configuration is needed for public charts. For private Helm repositories or OCI registries, configure access via the operator's Helm values: ```yaml theme={null} operator: helm: repositories: - name: my-private-repo url: https://charts.example.com username: "${HELM_USER}" password: "${HELM_PASS}" ``` ## Workflow Blocks ### Helm Upgrade Deploy a new version of an existing release with updated chart version or values. The most common use case for CI/CD automation. **Fields:** * **Cluster** — target cluster * **Release Name** — name of the Helm release (e.g., `my-app`) * **Namespace** — Kubernetes namespace * **Chart** — chart reference (e.g., `bitnami/nginx`, `oci://ghcr.io/org/chart`) * **Set Values** — inline `--set` values (e.g., `image.tag={{signal.commit_sha}},replicas=3`) * **Values YAML** — full YAML values content (from previous step output or inline) * **Chart Version** — version constraint (leave empty for latest) * **Wait for Rollout** — wait until all resources are ready (default: true) * **Timeout** — max time to wait (default: 5m) **Outputs:** `release_name`, `namespace`, `revision`, `status`, `summary` ### Helm Install Install a new Helm release from a chart repository or OCI registry. **Fields:** Same as Upgrade, plus: * **Create Namespace** — create the namespace if it doesn't exist (default: true) **Outputs:** `release_name`, `namespace`, `revision`, `status`, `summary` ### Helm Rollback Rollback a Helm release to a previous revision. **Fields:** * **Cluster** — target cluster * **Release Name** — name of the release to rollback * **Namespace** — Kubernetes namespace * **Revision** — revision number (leave empty to rollback to previous) * **Wait** / **Timeout** **Outputs:** `release_name`, `revision`, `status`, `summary` ### Helm Uninstall Remove a Helm release and all associated Kubernetes resources. **Fields:** * **Cluster** — target cluster * **Release Name** — name of the release to remove * **Namespace** — Kubernetes namespace **Outputs:** `release_name`, `status`, `summary` ### Helm Status Get the current status of a Helm release. Useful for condition gates and health checks. **Fields:** * **Cluster** — target cluster * **Release Name** — name of the release * **Namespace** — Kubernetes namespace **Outputs:** `release_name`, `namespace`, `revision`, `status`, `chart`, `app_version`, `summary` ## Example Workflows ### Deploy on GitHub Action Success When a CI build completes, upgrade the Helm release with the new image tag: ``` Trigger: GitHub Action Completed (workflow: build-and-push) → Helm Upgrade (release: my-app, chart: oci://ghcr.io/org/app, set: image.tag={{signal.commit_sha}}) → Slack Notify (#deploys, "Deployed {{signal.commit_sha}} to production") ``` ### Rollback on Pod Crash When pods start crashing after a deploy, automatically rollback: ``` Trigger: Pod CrashLoopBackOff → Helm Status (release: my-app) → Condition: status == "deployed" (only rollback if it was a Helm-managed deploy) → True: Helm Rollback (release: my-app) → True: Slack Notify (#incidents, "Rolled back my-app due to crashes") ``` ## Supported Chart Sources | Source | Example | Notes | | -------------- | ------------------------- | ------------------------------------------ | | Public repos | `bitnami/nginx` | Must be added with `helm repo add` first | | OCI registries | `oci://ghcr.io/org/chart` | Requires registry access from operator pod | | Inline values | Via "Values YAML" field | YAML content piped to `helm ... -f -` | ## Troubleshooting | Issue | Solution | | ------------------------------- | ------------------------------------------------------------------------------------------ | | "release not found" | Verify the release name and namespace are correct | | "chart not found" | Ensure the chart repository is accessible from the operator pod | | "timeout waiting for condition" | Increase the timeout or check pod health separately | | OCI registry auth errors | Ensure the operator pod has registry credentials (imagePullSecrets or helm registry login) | # Infisical Source: https://docs.usekestrel.ai/integrations/infisical Connect Infisical for secret change monitoring, approval-request and failed-sync alerts, machine-identity tracking, secret and folder management, and AI investigation in workflows The Infisical integration connects Kestrel to your Infisical organization (Infisical Cloud or self-hosted) through the Infisical REST API, enabling secret changes, approval requests, failed secret syncs, and new machine identities to trigger workflows and letting workflows read and manage secrets, organize folders, trigger secret syncs, review approval requests, and query audit logs. Infisical has no outbound control-plane webhooks for organization-level events (its per-project webhooks only signal that *something* changed), so Kestrel polls the Infisical **audit log** and project APIs on a per-tenant cadence to detect secret creates/updates/deletes, new approval requests, failed syncs, and new identities. Polling reads **only metadata and audit events** — secret values are never read or stored by the pollers. ## Prerequisites * **Organization Admin** role in Kestrel * An Infisical organization with at least one project * A **Universal Auth machine identity** for Kestrel with access to the projects your workflows manage * Audit-log triggers (secret created/updated/deleted, identity created) require an Infisical plan with **audit log API access** ## Setup In Infisical, go to **Organization → Access Control → Machine Identities** and press **Create Organization Machine Identity** (e.g. `kestrel`). Universal Auth is added as its authentication method by default. On the identity's page, open the **Universal Auth** method under **Authentication** and copy its **Client ID** (this is different from the identity's ID shown in Details), then create a **Client Secret** — the secret is shown only once. Press **Add to Project** in the **Projects** section (or go to **Project → Access Control → Machine Identities**) for each project your workflows should manage, with a role that matches what your workflows do — **Viewer** for read-only monitoring, or a role with secret write permissions if workflows create or update secrets. 1. Navigate to **Integrations → Infisical** in your Kestrel dashboard. 2. Paste the **Client ID** and **Client Secret**. 3. Optionally set a custom **Site URL** for self-hosted Infisical (defaults to `https://app.infisical.com`). 4. Click **Connect Infisical** — Kestrel validates the credentials by logging in and listing your projects, then starts polling. The machine identity can read and write real secrets, depending on the project roles you grant it. Give it the narrowest project access your workflows need, and gate destructive workflow actions (Delete Secret) behind Approval nodes. Kestrel stores the credentials encrypted. All Infisical triggers are detected by polling, so there may be up to one poll interval of delay before a workflow fires. The poll cadence is configurable per trigger (1m–30m, default 5 minutes). Secret-change triggers are derived from the audit log; pollers never read secret values. ## How It's Used ### In Workflows **Trigger blocks (all poll-based):** * **Secret Created / Updated / Deleted** — fire when secrets change in a project (derived from the audit log; values are never read) * **Approval Requested** — fires when a secret-change approval request is opened in a project * **Secret Sync Failed** — fires when a secret sync to an external destination (AWS, GitHub, etc.) fails * **Identity Created** — fires when a new machine identity appears in the organization Filter secret triggers by project, environment slug, and folder-path prefix. Trigger events expose template variables such as `{{signal.project_id}}`, `{{signal.project_name}}`, `{{signal.environment}}`, `{{signal.secret_path}}`, `{{signal.secret_key}}`, `{{signal.sync_id}}`, `{{signal.sync_name}}`, and `{{signal.actor}}` for downstream steps. **Action blocks:** * **Get Secret** — read a secret (the value output is marked sensitive: it is redacted in run history but can be templated into downstream steps, e.g. to sync into another system) * **Create Secret** / **Update Secret** — store or rotate a secret; the value can reference previous step outputs, e.g. a freshly generated credential * **Delete Secret** — delete a secret (gate behind an Approval node) * **List Secrets** — enumerate secret key names (never values) in a project/environment/path * **List Projects** / **List Environments** / **List Folders** / **Create Folder** — inventory and organize the secret hierarchy * **List Secret Syncs** / **Trigger Secret Sync** — check sync health and re-run a failed sync * **List Approval Requests** — review open secret-change approval requests * **Get Audit Logs** — query recent audit events, optionally filtered by project and event type * **List Identities** — inventory the organization's machine identities * **Investigate Infisical** — run a read-only AI investigation across projects, syncs, approvals, identities, and audit logs (the investigation can never read secret values) Project, environment, folder, and sync selects use dynamic dropdowns backed by the Infisical API and accept template variables like `{{signal.project_id}}` and `{{signal.sync_id}}` from the trigger. Example (sync recovery — the flagship Infisical flow): When a **secret sync fails**, post the sync name and project to Slack with approve/reject buttons, then **Trigger Secret Sync** to retry on approval. Example (change review): When a **secret is updated** in the `prod` environment, fetch recent audit logs for the project and post a summary — who changed what, and when — to your security Slack channel. Example (governance): When an **approval request** is opened, notify the reviewers' Slack channel with the project and environment, and page via PagerDuty if it is still open after an escalation delay. ## Disconnecting 1. Navigate to **Integrations → Infisical** 2. Click **Disconnect** 3. Confirm the disconnection This stops polling and all Infisical workflow triggers and actions. The stored credentials are deleted from Kestrel; you can also revoke the machine identity's client secret in Infisical. You can reconnect at any time. # Jenkins Source: https://docs.usekestrel.ai/integrations/jenkins Connect Jenkins for build triggering (with parameters), build monitoring, console log retrieval, stopping runaway builds, build-failure webhooks, and AI investigation in workflows The Jenkins integration connects Kestrel to your Jenkins controller through the Jenkins REST API, enabling build lifecycle events to trigger workflows and letting workflows trigger jobs (optionally with parameters), wait on builds, fetch console logs, stop runaway builds, and run read-only AI investigations across jobs, builds, and the build queue. Build events are delivered by a **webhook notification plugin** (such as the Notification plugin) or a simple post-build `curl` step POSTing to Kestrel. Deliveries are authenticated with a per-tenant **shared secret** (the `X-Kestrel-Webhook-Secret` header, or a `?secret=` query parameter for plugins that cannot set custom headers) generated at connect time. ## Prerequisites * **Organization Admin** role in Kestrel * A Jenkins controller reachable from Kestrel (Jenkins is self-hosted; the controller URL must be accessible) * A Jenkins user and **API token** with permission to read jobs and trigger builds ## Setup In Jenkins, click your username → **Configure** (or **Security**), then under **API Token** click **Add new Token**. Give it a name (e.g. `kestrel`) and copy the generated token — it is shown only once. The username + API token authenticate to the Jenkins REST API via HTTP basic auth. The user needs at least **Job/Read** and **Job/Build** permissions for the jobs your workflows will manage. 1. Navigate to **Integrations → Jenkins** in your Kestrel dashboard. 2. Enter your **Jenkins URL** (e.g. `https://jenkins.example.com`). 3. Enter the **Username** the token belongs to. 4. Paste the **API token**. 5. Click **Connect Jenkins** — Kestrel validates the credentials by listing jobs and generates a **webhook secret** for build-event deliveries. Actions work as soon as the credentials are connected. To have Jenkins builds **trigger** workflows, configure Jenkins to POST build events to Kestrel: 1. On the Kestrel Jenkins integration page, copy the **Webhook URL** and the **Webhook secret**. 2. **Option A (Notification plugin):** install the Notification plugin and add a notification endpoint to each job with the webhook URL, format JSON, protocol HTTP. Append `?secret=` to the URL if the plugin cannot set custom headers. 3. **Option B (post-build step):** add a post-build shell step that POSTs to the webhook URL with the `X-Kestrel-Webhook-Secret` header and a JSON body containing `job_name`, `build_number`, `phase` (`STARTED`/`COMPLETED`), `status` (`SUCCESS`/`FAILURE`/`UNSTABLE`/`ABORTED`), and `build_url`. The API token can trigger and abort builds. Treat it as a secret — Kestrel stores it encrypted. Gate build-triggering workflow actions on production deploy jobs behind Approval nodes. Folder-nested jobs (including multibranch pipeline branches) are addressed by their full path, e.g. `platform/deploy-api`. The job dropdowns in the workflow builder list nested jobs automatically. ## How It's Used ### In Workflows **Trigger blocks (webhook):** * **Build Failed** — fires when a build completes with `FAILURE` * **Build Unstable** — fires when a build completes `UNSTABLE` (e.g. test failures) * **Build Succeeded** — fires when a build completes with `SUCCESS` * **Build Completed (any result)** — fires when a build finishes with any result * **Build Started** — fires when a build starts Filter triggers by job name (full path for folder-nested jobs) and build status. Trigger events expose template variables such as `{{signal.job_name}}`, `{{signal.build_number}}`, `{{signal.build_status}}`, and `{{signal.build_url}}` for downstream steps. **Action blocks:** * **Trigger Build** — trigger a job build, optionally with build parameters (one `KEY=VALUE` per line) * **Wait for Build** — poll a build until it finishes and branch on the result (`SUCCESS`, `FAILURE`, `UNSTABLE`, `ABORTED`) * **Get Build Status** — fetch a build's status without waiting (`BUILDING` while in progress) * **Stop Build** — abort a running build (kill hung or runaway builds to free executors; gate behind an Approval node when appropriate) * **Get Console Log** — fetch the tail of a build's console log (great before Slack posts or AI analysis) * **Investigate Jenkins** — run a read-only AI investigation across jobs, builds, console logs, and the build queue Job selects use dynamic dropdowns backed by the Jenkins API and accept template variables like `{{signal.job_name}}` and `{{signal.build_number}}` from the trigger. Example (CI failure RCA — the flagship Jenkins flow): When a **build fails**, fetch the console log, run an **AI investigation** of the failure, then post the root cause to Slack `#ci` and open a Jira ticket. Example (remediation): When a nightly job fails, **trigger** a cleanup job with parameters, **wait** for it, and page on-call via PagerDuty only if the retry also fails. Example (hygiene): On a schedule, investigate builds that have been running longer than an hour and **stop** the hung ones, posting a summary to Slack. ## Disconnecting 1. Navigate to **Integrations → Jenkins** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Jenkins workflow triggers and actions. Notification endpoints configured in Jenkins will keep sending deliveries until you remove them there; deliveries with an unknown secret are rejected. You can reconnect at any time. # Jira Source: https://docs.usekestrel.ai/integrations/jira Connect Jira for ticket creation, status tracking, and incident knowledge during investigations The Jira integration connects Kestrel to your Atlassian Jira instance, enabling automated ticket creation from workflows, status tracking, and historical issue search during incident investigations. ## Prerequisites * **Organization Admin** role in Kestrel * Atlassian account with Jira access ## Setup 1. Navigate to **Integrations → Jira** in your Kestrel dashboard 2. Click **Connect** 3. Authorize Kestrel in the Atlassian OAuth popup 4. Select the Jira site to connect (if you have multiple) 5. The popup closes and your Jira instance appears as connected Kestrel uses OAuth 2.0 with scoped permissions. Only the projects and issue types you authorize are accessible. ## How It's Used ### In Workflows **Action blocks:** * **Create Ticket** — create a Jira ticket with customizable project, issue type, summary, description, priority, labels, and assignee * **Add Comment** — add a comment to an existing Jira issue with workflow context * **Transition Ticket** — move a Jira issue through workflow states (e.g., Open → In Progress → Done) Example: A workflow triggered by a Kubernetes incident can automatically create a Jira ticket with the incident summary, assign it to the namespace owner, and transition it to "In Progress." ### In Incident Response * **Knowledge source** — during root cause analysis, Kestrel searches Jira for similar past issues, related post-mortems, and historical context about affected services (see [Knowledge Sources](/integrations/knowledge-sources)) * **Incident tracking** — create Jira tickets directly from incident detail pages to track remediation ### In Cloud AI Copilot * Create Jira tickets from the chat: `Create a Jira ticket for the payments service outage` * Search for related issues: `Find Jira issues related to OOM errors in production` ## Configuration ### Knowledge Source Setup To use Jira as a knowledge source during incident investigations: 1. Navigate to **Integrations → Knowledge Sources** 2. Click **Add Source** and select **Jira** 3. Your connected Jira instance is auto-detected — no additional credentials needed 4. Click **Enable** Kestrel will search Jira issues and comments for relevant historical context during incident RCA. ## Disconnecting 1. Navigate to **Integrations → Jira** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Jira workflow actions and removes Jira as a knowledge source. You can reconnect at any time. # Karpenter Source: https://docs.usekestrel.ai/integrations/karpenter Connect Karpenter for node autoscaling actions — scale NodePool limits, tune disruption settings, and recycle nodes in workflows The Karpenter integration connects Kestrel to your in-cluster Karpenter installation, enabling AI agents to inspect and scale NodePools, tune consolidation behavior, and recycle nodes as part of automated workflows and incident response. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * Karpenter installed in the same cluster with its CRDs present (`nodepools.karpenter.sh`, `nodeclaims.karpenter.sh`) ## Setup Karpenter is auto-detected by the Kestrel Operator when the Karpenter CRDs are present in the cluster. No manual connection or credentials are required. If Karpenter is not already installed, follow the [Karpenter getting started guide](https://karpenter.sh/docs/getting-started/) for your cloud provider. The operator probes for the Karpenter CRDs periodically. Check the operator logs: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "Karpenter" ``` Once detected, the integration shows as **Detected** on the Integrations page and Karpenter action blocks become available in the workflow builder. The integration is enabled by default. To opt out, set `operator.karpenter.enabled: false` in your Helm values and upgrade the operator (this sets `KARPENTER_DISABLED=true` on the operator deployment). ## How It's Used ### In Workflows **Trigger blocks:** * **Node Provisioning Failed** — fires when Karpenter fails to launch or register a node for a NodeClaim (insufficient capacity, launch failure, registration failure) * **Node Interrupted** — fires when a Karpenter-managed node is interrupted or forcibly terminated (spot reclaim, drain failure, termination-grace expiry) * **NodePool Limit Reached** — fires when pods cannot be scheduled because a NodePool's resource limits are exceeded **Action blocks:** * **List NodePools** — list all NodePools with their limits, current resource usage, disruption settings, and readiness * **Get NodePool Status** — read a NodePool's limits, usage, disruption settings, and live NodeClaim counts * **List NodeClaims** — list provisioned nodes with instance type, capacity type, zone, and readiness, optionally scoped to one NodePool * **Scale NodePool Limits** — update a NodePool's resource limits (`spec.limits`) to allow more capacity or cap spend — this is how you scale up/down with Karpenter * **Set Disruption Policy** — update consolidation policy and consolidate-after delay to control how aggressively Karpenter removes underutilized nodes * **Apply NodePool** — create or update a NodePool from a full manifest * **Delete NodeClaim** — recycle a node; Karpenter drains it gracefully and provisions a replacement if capacity is still needed Example: A developer asks in the request chat to scale up the `default` NodePool. The workflow raises the NodePool's CPU limit, waits for capacity, and posts a confirmation to the platform team's Slack channel. ### Scoping Karpenter actions are scoped by **cluster** and **NodePool** — both populated with dropdowns from live cluster data in the workflow builder. ## Disconnecting To disable the Karpenter integration, set the following in your Helm values and upgrade the operator: ```yaml theme={null} operator: karpenter: enabled: false ``` # Knowledge Sources Source: https://docs.usekestrel.ai/integrations/knowledge-sources Connect your organization's knowledge sources to enhance incident investigations with historical context Knowledge Sources connects Kestrel to the tools where your team's operational knowledge already lives. When an incident is detected, Kestrel automatically searches your connected sources for similar past incidents, runbooks, postmortems, and relevant discussions — providing historical context alongside real-time diagnostics. ## How It Works When an incident is detected, Kestrel automatically: 1. Searches connected knowledge sources for relevant context 2. Finds similar past incidents and their resolutions 3. Locates runbooks and documentation for affected services 4. Surfaces team discussions about the affected systems 5. Retrieves postmortems and lessons learned from related incidents This context is presented alongside Kestrel's automated investigation, giving you both real-time diagnostics and historical insight. ## Supported Sources | Source | Auth Method | What It Searches | | ---------------------------------------------------- | ---------------------------------- | ---------------------------------------------------- | | [**Slack**](/integrations/slack) | Kestrel App (no extra setup) | Messages and threads for relevant discussions | | [**Confluence**](/integrations/confluence) | OAuth (via Confluence integration) | Wiki pages, runbooks, postmortems, architecture docs | | [**Jira**](/integrations/jira) | OAuth (via Jira integration) | Issues, comments, and postmortems | | [**Linear**](/integrations/linear) | OAuth (via Linear integration) | Issues, projects, and comments | | [**Notion**](/integrations/knowledge-sources#notion) | Internal Integration Token | Pages and databases | | [**Glean**](/integrations/glean) | API Key | Enterprise-wide search across all connected tools | ## Setup ### Adding a Source Go to **Integrations → Knowledge Sources** in your Kestrel dashboard. Click **Add Source** and select the source type from the dropdown. Follow the source-specific instructions below. Sources that have a dedicated integration (Slack, Confluence, Jira, Linear) are auto-detected when that integration is already connected. Click **Add Source** to save and enable the knowledge source. ### Slack Slack uses the same Kestrel app as incident notifications. No additional credentials are required. **Prerequisite**: Your Slack workspace must be connected via the [Slack Integration](/integrations/slack). 1. Select **Slack** as the source type 2. If connected, you'll see a green confirmation 3. Click **Enable Slack Search** Invite @Kestrel to the channels you want to be searchable for knowledge source lookups. ### Confluence If Confluence is connected via the [Confluence Integration](/integrations/confluence), it's auto-detected. Otherwise: 1. Go to [id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens) 2. Create an API token 3. Enter your **Email Address**, **API Token**, and **Base URL** (e.g., `https://yourcompany.atlassian.net`) ### Jira If Jira is connected via the [Jira Integration](/integrations/jira), it's auto-detected. Otherwise: 1. Go to [id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens) 2. Create or reuse an Atlassian API token 3. Enter your **Email Address**, **API Token**, and **Base URL** ### Linear If Linear is connected via the [Linear Integration](/integrations/linear), it's auto-detected. Otherwise: 1. Go to [linear.app/settings/account/security](https://linear.app/settings/account/security) 2. Create a **Personal API Key** 3. Paste the key in the **API Key** field Personal API keys have the same permissions as your user account. Consider using a service account for production. ### Notion 1. Go to [notion.so/my-integrations](https://notion.so/my-integrations) and create a new integration 2. Copy the **Internal Integration Secret** token 3. **Important**: Share the pages/databases you want searchable with your integration 4. Paste the token in the **API Key** field 5. Optionally enter a **Workspace ID** You must explicitly share each Notion page or database with the integration for it to be searchable. ### Glean See the [Glean integration page](/integrations/glean) for full setup instructions. ## Managing Sources ### Testing a Connection Click the **Test** button on any source to verify credentials and connectivity. ### Enabling/Disabling Use the toggle switch on each source to temporarily enable or disable it without removing the configuration. ### Editing Click the edit icon to update the display name, credentials, or base URL. Leave credential fields blank to keep existing values. ### Deleting Click the delete icon and confirm to permanently remove a source. Incident analysis will no longer include context from that source. # Kubernetes Source: https://docs.usekestrel.ai/integrations/kubernetes Connect your Kubernetes clusters to Kestrel for real-time incident detection, traffic visibility, and AI-powered operations The Kestrel Operator is a lightweight Kubernetes agent that connects your clusters to the Kestrel AI platform. It streams resource metadata, events, logs, and network traffic telemetry over mTLS — enabling 24/7 real-time incident detection, AI-powered workflows, and an infrastructure copilot without any changes to your existing applications. ## Prerequisites * Kubernetes cluster (v1.24+) * `kubectl` configured with cluster admin access * `helm` CLI (v3.0+) * A Kestrel AI account with **Organization Admin** role * Network connectivity to `grpc.platform.usekestrel.ai:443` ## Setup 1. Log in to the [Kestrel AI Dashboard](https://platform.usekestrel.ai) 2. Navigate to **Integrations → Kubernetes** 3. Click **Generate New Credential** 4. Set a **Cluster Name** (e.g., `production-cluster`) — alphanumeric, dots, dashes, and underscores only 5. Choose your **Flow Collection Source** (Cilium L3/L4 or Istio L7) 6. Choose your **Metrics Source** (Kubernetes Metrics Server, OpenTelemetry, or Datadog) 7. Optionally enable **Safe-Apply Permissions** for AI-powered resource management Click **Download** to save the pre-configured values file: ``` kestrel-ai-operator-values-.yaml ``` ```bash theme={null} helm install kestrel-operator \ oci://ghcr.io/kestrelai/charts/kestrel-operator \ --version latest \ --namespace kestrel-ai \ --create-namespace \ -f kestrel-ai-operator-values-.yaml ``` Check pod status: ```bash theme={null} kubectl get pods -n kestrel-ai -l app=kestrel-operator ``` Expected output: ``` NAME READY STATUS RESTARTS AGE kestrel-operator-xxxxxxxxx-xxxxx 1/1 Running 0 1m ``` Then confirm the cluster shows as **Connected** (green badge) in **Integrations → Kubernetes**. The operator token automatically renews every 24 hours. No manual rotation is required. ## Flow Collection Sources Collects Layer 3/4 network flows from Cilium CNI via Hubble Relay. This is the default — no extra configuration needed if Cilium is your CNI. Collects Layer 7 HTTP/gRPC access logs from Envoy proxies via Istio's Access Log Service. Requires registering Kestrel as an Istio extension provider — see the [Configuration Reference](#istio-configuration) below. ## How It's Used ### In Workflows **Trigger blocks:** * **Deployment Replicas Failing** — fires when a Deployment cannot reach its desired replica count * **Nodes Unavailable** — fires when one or more cluster nodes become NotReady * **Pod CrashLoopBackOff** — fires when a pod enters a CrashLoopBackOff restart cycle * **Pod ImagePullBackOff** — fires when a pod cannot pull its container image * **Pod OOMKilled** — fires when a container is terminated due to an out-of-memory condition * **Pods Failing** — fires when pods transition to a failed state * **Pods Restarting** — fires when pods exceed a restart count threshold * **StatefulSet Replicas Failing** — fires when a StatefulSet cannot reach its desired replica count * **Node Memory Pressure** — fires when a node reports a MemoryPressure condition * **Node Disk Pressure** — fires when a node reports a DiskPressure condition * **DaemonSet Failing** — fires when a DaemonSet has unavailable pods * **Any Kubernetes Incident** — fires on any Kubernetes incident detected by Kestrel Kubernetes operations (kubectl commands, resource scaling, rolling restarts, network policy application, safe-apply YAML changes) are provided by Kestrel's built-in action blocks and are available in any workflow alongside these triggers. Example: A workflow triggers on Pod CrashLoopBackOff, runs an AI root cause analysis, applies a fix via safe-apply, and notifies the namespace owner on Slack. ### In Incident Response * **24/7 real-time detection** from Kubernetes events, pod logs, node conditions, and network telemetry * **Automated root cause analysis** correlating events, metrics, logs, and network flows * **YAML fix generation** with one-click GitOps deployment via connected GitHub/GitLab repos * **Causal PR detection** — identifies recent deployments that may have caused the incident ### In Cloud AI Copilot * Ask questions about any connected cluster: `What pods are crashing in production?` * Query metrics and resource utilization: `Show memory usage for the payments namespace` * Generate and apply Kubernetes resources through conversational AI * Use the `/kestrel` Slack command for quick cluster queries ## Configuration Reference ### Helm Values #### Authentication | Key | Type | Default | Description | | ------------- | ------- | ----------------------------- | ---------------------------------- | | `auth.token` | string | `""` | JWT token (generated in dashboard) | | `server.host` | string | `grpc.platform.usekestrel.ai` | gRPC endpoint | | `server.port` | integer | `443` | gRPC port | #### Cilium | Key | Type | Default | Description | | ----------------------------------------- | ------- | ------- | ---------------------------------- | | `operator.cilium.disableFlows` | boolean | `false` | Disable Cilium flow collection | | `operator.cilium.hubble.tls.forceDisable` | boolean | `false` | Disable TLS for Hubble connections | #### Istio Configuration | Key | Type | Default | Description | | ------------------------ | ------- | ------- | ----------------------------- | | `operator.istio.enabled` | boolean | `false` | Enable Istio ALS for L7 flows | | `operator.istio.alsPort` | integer | `8080` | ALS gRPC server port | When enabling Istio, register Kestrel as an extension provider in your Istio mesh config: ```yaml theme={null} meshConfig: extensionProviders: - name: kestrel-operator-als envoyHttpAls: service: kestrel-operator-als.kestrel-ai.svc.cluster.local port: 8080 - name: kestrel-operator-als-tcp envoyTcpAls: service: kestrel-operator-als.kestrel-ai.svc.cluster.local port: 8080 ``` #### Metrics Source | Key | Type | Default | Description | | ---------------------------- | ------- | ----------- | ------------------------------------- | | `operator.otel.enabled` | boolean | `false` | Enable OpenTelemetry metrics receiver | | `operator.otel.receiverPort` | integer | `4317` | OTLP gRPC receiver port | | `operator.datadog.enabled` | boolean | `false` | Enable Datadog metrics integration | | `operator.datadog.namespace` | string | `"datadog"` | Namespace where Datadog is deployed | #### Safe-Apply | Key | Type | Default | Description | | ---------------------------- | ------- | ------- | ---------------------------------------------------- | | `operator.safeApply.enabled` | boolean | `false` | Grant RBAC permissions for applying approved changes | Safe-Apply permissions allow the operator to create, update, and delete Kubernetes resources. Only enable if you plan to use AI-powered resource management features. ### Quick Configuration Examples **Cilium with Datadog metrics:** ```yaml theme={null} auth: token: "your-token-here" operator: cluster: id: "cluster-uuid" name: "production-cluster" datadog: enabled: true namespace: "datadog" ``` **Istio with OpenTelemetry metrics:** ```yaml theme={null} auth: token: "your-token-here" operator: cluster: id: "cluster-uuid" name: "production-cluster" cilium: disableFlows: true istio: enabled: true alsPort: 8080 otel: enabled: true ``` ## Managing Clusters ### Multiple Clusters Connect multiple clusters to a single Kestrel organization. Generate a unique credential for each cluster and deploy the operator with its specific token. Your subscription tier determines the maximum number of clusters. Contact [hello@usekestrel.ai](mailto:hello@usekestrel.ai) to upgrade. ### Updating ```bash theme={null} helm upgrade kestrel-operator \ oci://ghcr.io/kestrelai/charts/kestrel-operator \ --version latest \ --namespace kestrel-ai \ -f kestrel-ai-operator-values-.yaml ``` ### Uninstalling ```bash theme={null} helm uninstall kestrel-operator -n kestrel-ai ``` Then revoke the credential in **Integrations → Kubernetes** by clicking the delete icon next to your cluster. ### Connection Status | Status | Meaning | | --------------------- | --------------------------------------- | | **Connected** (Green) | Actively sending data | | **Offline** (Gray) | Previously connected, currently offline | | **Pending** (Red) | Never connected with this token | # Kyverno Source: https://docs.usekestrel.ai/integrations/kyverno Connect Kyverno for policy-as-code automation — react to policy violations and admission blocks, audit compliance, and manage policy enforcement in workflows The Kyverno integration connects Kestrel to your in-cluster Kyverno installation, enabling policy violations and admission blocks to trigger workflows and letting workflows list policies, audit violations from PolicyReports, switch policies between Audit and Enforce, and deploy or remove policies as code. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * Kyverno installed in the same cluster with its CRDs present (`clusterpolicies.kyverno.io`, `policies.kyverno.io`) ## Setup Kyverno is auto-detected by the Kestrel Operator when the Kyverno CRDs are present in the cluster. No manual connection or credentials are required. If Kyverno is not already installed, follow the [Kyverno installation guide](https://kyverno.io/docs/installation/). The operator probes for the Kyverno CRDs periodically. Check the operator logs: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "Kyverno" ``` Once detected, the integration shows as **Detected** on the Integrations page and Kyverno trigger and action blocks become available in the workflow builder. The integration is enabled by default. To opt out, set `operator.kyverno.enabled: false` in your Helm values and upgrade the operator (this sets `KYVERNO_DISABLED=true` on the operator deployment). ## How It's Used ### In Workflows **Trigger blocks:** * **Policy Violation** — fires when a Kyverno PolicyReport records a new failing result (an Audit-mode violation on a live resource). The operator watches `PolicyReports` and `ClusterPolicyReports` and streams new fail/warn/error results in near-real time. * **Admission Blocked** — fires when an Enforce-mode policy blocks a resource at admission. Scope triggers by cluster, namespace, policy name, and severity (`critical`, `high`, `medium`, `low`, `info` — as annotated on the policy). Trigger events expose template variables such as `{{signal.policy_name}}`, `{{signal.rule_name}}`, `{{signal.severity}}`, `{{signal.resource_kind}}`, `{{signal.resource_name}}`, `{{signal.namespace}}`, and `{{signal.message}}` for downstream steps. **Action blocks:** * **List Policies** — list ClusterPolicies and namespaced Policies with enforcement mode (Audit/Enforce), rule counts, categories, and readiness * **Get Policy** — read one policy's enforcement mode, rules (validate/mutate/generate/verifyImages), matched resource kinds, and readiness * **List Violations** — list current violations from PolicyReports, filterable by namespace, policy, severity, and result (fail/warn/error) * **Set Enforcement Mode** — switch a policy between Audit (report violations) and Enforce (block non-compliant resources at admission) — how you promote a guardrail to blocking mode * **Apply Policy** — create or update a ClusterPolicy/Policy from a full manifest, deploying admission guardrails as code * **Delete Policy** — remove a policy that is blocking legitimate deployments or is no longer needed * **Investigate Kyverno** — run a read-only AI investigation across policies, rules, and PolicyReport violations via the cluster operator (the investigation can never apply, delete, or re-mode a policy) Set Enforcement Mode, Apply Policy, and Delete Policy change live admission behavior cluster-wide — gate them behind Approval nodes in your workflows. Example (compliance): When a **critical policy violation** fires, post the policy, rule, and violating resource to your security Slack channel and open a Jira ticket. Example (guardrail promotion): A developer asks in the request chat to enforce the `require-labels` policy. The workflow lists its current violations, requests approval in Slack, then **Set Enforcement Mode** to Enforce and confirms. Example (audit): On a daily schedule, **List Violations** across the cluster, summarize with AI, and post a compliance digest to Slack. ### Scoping Kyverno actions are scoped by **cluster** and **policy** — both populated with dropdowns from live cluster data in the workflow builder. Namespaced Policies additionally take a **namespace**; leave it empty for cluster-scoped ClusterPolicies. ## Disconnecting To disable the Kyverno integration, set the following in your Helm values and upgrade the operator: ```yaml theme={null} operator: kyverno: enabled: false ``` # Linear Source: https://docs.usekestrel.ai/integrations/linear Connect Linear for issue creation, tracking, and historical context during incident investigations The Linear integration connects Kestrel to your Linear workspace, enabling automated issue creation from workflows and historical issue search during incident investigations. ## Prerequisites * **Organization Admin** role in Kestrel * Linear account with workspace access ## Setup 1. Navigate to **Integrations → Linear** in your Kestrel dashboard 2. Click **Connect** 3. Authorize Kestrel in the Linear OAuth popup 4. The popup closes and your Linear workspace appears as connected ## How It's Used ### In Workflows * **Action blocks**: * **Create Issue** — create Linear issues with customizable team, project, title, description, priority, labels, and assignee * **Update Issue** — modify fields on existing issues * **Search Issues** — query Linear for related issues * **Add Comment** — add comments to existing issues with workflow context Example: A workflow triggered by a deployment failure can automatically create a Linear issue assigned to the owning team with incident context and logs. ### In Incident Response * **Knowledge source** — during root cause analysis, Kestrel searches Linear for similar past issues, related bugs, and historical context about affected services (see [Knowledge Sources](/integrations/knowledge-sources)) * **Issue creation** — create Linear issues directly from incident detail pages to track remediation work ### In Cloud AI Copilot * Create issues from the chat: `Create a Linear issue for the API latency regression` * Search for related issues: `Find Linear issues related to database connection timeouts` ## Configuration ### Knowledge Source Setup To use Linear as a knowledge source during incident investigations: 1. Navigate to **Integrations → Knowledge Sources** 2. Click **Add Source** and select **Linear** 3. Your connected Linear workspace is auto-detected 4. Click **Enable** Kestrel will search Linear issues and comments for relevant historical context during incident RCA. ## Disconnecting 1. Navigate to **Integrations → Linear** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Linear workflow actions and removes Linear as a knowledge source. You can reconnect at any time. # Nebius Source: https://docs.usekestrel.ai/integrations/nebius Connect Nebius AI Cloud for GPU/node lifecycle triggers, instance actions, and managed Kubernetes scaling in workflows The Nebius integration connects Kestrel to your Nebius AI Cloud account, enabling GPU and node lifecycle events to trigger workflows and giving AI agents access to instance state, managed Kubernetes clusters, and node-group scaling across your projects. Nebius does not send outbound webhooks. Kestrel detects GPU errors and node/instance lifecycle events by polling the Nebius API on a configurable interval, authenticating with a service-account key exchanged for an IAM token via RFC-8693 token exchange. ## Prerequisites * **Organization Admin** role in Kestrel * Nebius AI Cloud account with project access * Nebius CLI (`nebius`) installed to generate a service-account key ## Setup Create (or choose) a service account with read access to instances and Kubernetes node state and permission to perform instance/scaling actions in your projects, then generate an authorized key JSON: ```bash theme={null} nebius iam auth-public-key generate \ --service-account-id \ --output authorized-key.json ``` The resulting JSON contains a `subject-credentials` object with the signing key: ```json theme={null} { "subject-credentials": { "alg": "RS256", "private-key": "...", "kid": "...", "iss": "...", "sub": "..." } } ``` 1. Navigate to **Integrations → Nebius** in your Kestrel dashboard 2. Click **Connect** 3. Paste the **service account authorized key JSON**. It is exchanged for an IAM token against `https://api.nebius.cloud` and used to read instance/node state and perform actions across your projects. 4. Set the **Region** (defaults to `eu-north1`). 5. Choose a **Poll Interval** for trigger detection. 6. Save — your Nebius account appears as connected. The authorized key JSON contains a private signing key. Treat it as a secret — Kestrel stores it encrypted and only uses it to mint short-lived IAM tokens. No webhook setup is required inside Nebius. Kestrel polls the Nebius API on your configured interval to detect GPU errors and node/instance lifecycle events. ## How It's Used ### In Workflows **Trigger blocks:** * **GPU Error** — fires when a node reports a GPU error condition * **Maintenance Scheduled** — fires when maintenance is scheduled for a node * **Node Not Ready** — fires when a node transitions into a not-ready state * **Instance Stopped** — fires when an instance transitions into a stopped state Filter triggers by project, cluster, and event type. Kestrel polls the Nebius API on your configured interval to detect these conditions. **Action blocks:** * **Get Instance** — retrieve an instance's state, config, and region * **Start Instance** — start a stopped instance * **Stop Instance** — gracefully stop a running instance * **Restart Instance** — restart an instance in place without recreating it * **List Instances** — list instances for a project * **List Clusters** — list managed Kubernetes clusters for a project * **List Node Groups** — list node groups for a managed Kubernetes cluster * **Scale Node Group** — scale a node group to a target size * **Create Instance** — provision a new GPU or CPU instance with its boot disk (platform, preset, subnet, image, disk size, optional SSH key) * **Delete Instance** — delete an instance and, by default, its boot disk * **Create Node Group** — provision a new GPU/CPU worker pool in a managed Kubernetes cluster * **Delete Node Group** — delete a node group, tearing down its nodes * **Investigate Nebius** — run an AI investigation of GPU errors, node failures, and configuration issues Nebius project, instance, cluster, and node-group selects accept template variables like `{{signal.project_id}}`, `{{signal.instance_id}}`, `{{signal.cluster_id}}`, and `{{signal.node_name}}` from the trigger. Provisioning and deletion blocks (Create/Delete Instance, Create/Delete Node Group) create or destroy billable GPU capacity; add an Approval node before them if you want a human sign-off step. Create Instance outputs an `instance_id` that downstream blocks can reference via `{{step_outputs..instance_id}}`. Example: A workflow triggers on a GPU error, runs an AI investigation to diagnose the failure, scales the affected node group to replace the bad node, and posts a summary to Slack. Example: A developer requests an H100 instance via Slack (`req-nebius` trigger); the workflow waits for approval, provisions the instance with Create Instance, and DMs the requester the new instance ID. ## Disconnecting 1. Navigate to **Integrations → Nebius** 2. Click **Disconnect** 3. Confirm the disconnection This stops Nebius polling and all Nebius workflow triggers. You can reconnect at any time. # Neon Source: https://docs.usekestrel.ai/integrations/neon Connect Neon for serverless Postgres branch lifecycle, point-in-time restore, compute suspend/resume and autoscaling, reset-from-parent, credential rotation, and AI investigation in workflows The Neon integration connects Kestrel to your Neon organization through the Neon API, enabling control-plane events (branches, compute endpoints, operations, and usage) to trigger workflows and giving AI agents read-only access to your projects' state. Neon has no control-plane webhooks (its only webhooks are Neon Auth application-auth events like `send.otp` / `user.created`, which are not infrastructure events), so Kestrel polls the Neon API on a per-tenant cadence to detect branch creation/readiness, operation failures, compute suspend/resume, and usage thresholds. ## Prerequisites * **Organization Admin** role in Kestrel * A Neon account with at least one project * A Neon **API key** (personal or organization) created in the Neon Console * Your Neon **Organization ID** if your account is organization-scoped (found in the Console URL, `console.neon.tech/app/org-.../projects`, or under Organization settings) ## Setup In the Neon Console (`console.neon.tech`), open **Account settings → API keys**, click **Create new API key**, give it a name (e.g. `kestrel`), and copy the generated key (shown only once). The key is sent as a bearer token to the Neon API at `console.neon.tech/api/v2` to read and manage projects, branches, compute endpoints, and operations. A personal key covers all projects you can access; an organization key scopes to that organization. 1. Navigate to **Integrations → Neon** in your Kestrel dashboard. 2. Paste your **API key**. 3. Enter your **Organization ID** if your Neon account is organization-scoped (a personal account can leave this blank). Find your org ID in the Neon Console URL when you open the org (`console.neon.tech/app/org-.../projects`), or under **Organization settings → Settings**. 4. Choose a **Poll Interval** (how often Kestrel checks the Neon API for branch/compute/operation/usage events). 5. Click **Connect Neon** — your Neon organization appears as connected. The API key grants access to your Neon projects and their control plane. Treat it as a secret — Kestrel stores it encrypted. All Neon triggers are detected by polling, so there may be up to one poll interval of delay before a workflow fires. The minimum poll interval is 60 seconds. ## How It's Used ### In Workflows **Trigger blocks (all poll-based):** * **Branch Created** — fires when a new Neon branch is created * **Branch Ready** — fires when a branch becomes ready to connect * **Operation Failed** — fires when a Neon operation (create branch, start compute, etc.) enters a failed state * **Compute Suspended** — fires when a compute endpoint autosuspends to idle (cost signal) * **Compute Active** — fires when a compute endpoint resumes to active * **Usage/Quota Threshold** — fires when a project crosses a storage or compute usage threshold Filter triggers by project ID, branch ID, and event type. **Action blocks:** * **List Projects** / **Get Project**, **List Branches** / **Get Branch** — inventory * **Create Branch** — instant copy-on-write branch (optionally with a compute endpoint), ideal for ephemeral PR preview databases * **Delete Branch** — tear down a branch (gate behind an Approval node) * **Reset Branch from Parent** — the "reset my preview DB" action (gate behind an Approval node) * **Restore Branch (Point-in-Time)** — restore a branch to an earlier timestamp or LSN ("restore main to 2 hours ago"); the pre-restore state is preserved under a backup branch (gate behind an Approval node) * **List Compute Endpoints** / **Create Compute Endpoint** / **Delete Compute Endpoint** * **Suspend Compute** / **Start Compute** — cut/restore compute cost * **Set Autoscaling Limits** — update a compute endpoint's autoscaling min/max compute units * **Rotate Role Credentials** — reset a role's password and return a fresh connection URI (gate behind an Approval node) * **Get Connection URI** — fetch a branch's connection string (for posting to Slack) * **Investigate Neon** — run an AI investigation of projects, branches, compute endpoints, and failed operations Neon project and branch selects accept template variables like `{{signal.project_id}}`, `{{signal.branch_id}}`, and `{{signal.endpoint_id}}` from the trigger. Example (the flagship Neon flow): When a pull request opens, create a Neon copy-on-write preview branch and post the connection URI to Slack; when the PR closes, delete the branch. Example (cost): When a compute endpoint suspends, notify the team; or on idle detection, suspend the compute to cut cost. Example (dev requests): A "reset my preview DB" Slack request gates behind an Approval, then resets the branch from its parent. Example (recovery): After a bad migration or accidental data deletion, an Approval-gated workflow restores the branch to a timestamp just before the incident — the pre-restore state stays available under an auto-created backup branch. ## Disconnecting 1. Navigate to **Integrations → Neon** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Neon workflow triggers. You can reconnect at any time. # OCI Source: https://docs.usekestrel.ai/integrations/oci Connect your Oracle Cloud Infrastructure tenancies to Kestrel for resource inventory and security monitoring Connect your Oracle Cloud Infrastructure (OCI) tenancies to Kestrel for cloud resource inventory, security monitoring, and VCN Flow Log analysis. ## Prerequisites * **Organization Admin** role in Kestrel * OCI tenancy with permissions to create API keys and policies * Access to the OCI Console or OCI CLI ## Setup 1. In the OCI Console, navigate to **Identity → Users → Your User → API Keys** 2. Click **Add API Key** 3. Download the private key file 4. Note the **fingerprint** displayed after adding the key Grant Kestrel read access to your tenancy resources: ``` Allow group KestrelGroup to read all-resources in tenancy Allow group KestrelGroup to manage vcn in tenancy Allow group KestrelGroup to manage flow-log-configs in tenancy ``` 1. Navigate to **Integrations → Cloud** in your Kestrel dashboard 2. Click **Connect OCI Tenancy** 3. Fill in the connection form: * **Connection Name**: A friendly name (e.g., "Production") * **Tenancy OCID**: Found in OCI Console → Tenancy Details * **User OCID**: Found in OCI Console → User Settings * **Fingerprint**: The API key fingerprint from Step 1 * **Private Key**: Paste the contents of the private key PEM file * **Region**: Select your OCI home region 4. Click **Verify & Connect** Kestrel validates the credentials and begins discovering your cloud resources. ### VCN Flow Logs After connecting: 1. Navigate to the connected tenancy in the Cloud Integrations page 2. Expand the **VCN Flow Logs** section 3. Click **Enable** to start collecting flow logs across your VCNs 4. Kestrel creates the necessary log groups and flow log configurations You can manage flow logs per-VCN. ## How It's Used ### In Workflows * **Trigger blocks**: OCI Audit events, security list changes, VCN configuration changes * **Action blocks**: Query OCI resources, describe instances, check security lists, generate Terraform fixes ### In Incident Response * **Cloud security monitoring** — detects misconfigurations across compute instances, object storage, VCNs, and IAM * **Terraform fix generation** — generates IaC remediation targeting connected repositories * **VCN Flow Log analysis** — analyzes traffic patterns for anomalous network behavior ### In Cloud AI Copilot * Ask questions about your OCI resources: `What compute instances are publicly accessible?` * Investigate security findings across tenancies * Query resource configurations and compliance posture ## Managing Connections * **Refresh**: Re-sync connection status and resource inventory * **Delete**: Removes the connection from Kestrel. Clean up the API key and policy in OCI separately. # Okta Source: https://docs.usekestrel.ai/integrations/okta Connect Okta for identity security automation — react to lockouts, MFA resets, and privileged group changes, and respond with suspensions, session revocation, and group changes in workflows The Okta integration connects Kestrel to your Okta organization through the Okta Management API, enabling System Log security events (lockouts, suspicious activity, admin grants, MFA changes, group membership changes) to trigger workflows and letting workflows look up users, query the System Log, and take identity-response actions such as suspending users, clearing sessions, resetting MFA, and changing group membership. Okta triggers are poll-based: Kestrel tails your org's System Log on a per-tenant cadence, so no webhooks or Event Hooks are needed and triggers work on every Okta plan. ## Prerequisites * **Organization Admin** role in Kestrel * An Okta admin account for your org * An Okta **API token** (SSWS) — the token inherits the permissions of the admin who creates it ## Setup In the Okta Admin Console, go to **Security → API → Tokens → Create token**. Copy the token — it is shown only once. A read-only admin token is enough for triggers and read blocks; lifecycle and session blocks (suspend, deactivate, clear sessions, reset MFA, group changes) need a super admin or org admin token. Your Okta domain, e.g. `https://your-org.okta.com`, shown in the Admin Console header. 1. Navigate to **Integrations → Okta** in your Kestrel dashboard. 2. Enter your **org URL** and paste the **API token**. 3. Click **Connect Okta** — Kestrel validates the token and starts polling the System Log. The token can change live identity state, depending on the admin role it inherits. Use a dedicated service-account admin with the narrowest role that covers your workflows, and gate destructive actions (Suspend, Deactivate, Reset MFA, group removals) behind Approval nodes. Kestrel stores the token encrypted. All Okta triggers are detected by polling, so there may be up to one poll interval of delay before a workflow fires. The poll cadence is configurable per trigger (1m–30m; 1m is recommended for security-response triggers). ## How It's Used ### In Workflows **Trigger blocks (all poll-based via the System Log):** * **User Locked Out** — `user.account.lock` (e.g. after repeated sign-in failures) * **Suspicious Activity Reported** — end-user report or ThreatInsight `security.threat.detected` * **Admin Privilege Granted** — `user.account.privilege.grant` * **MFA Factor Changed** — factor deactivated or all factors reset * **User Created / User Deactivated** — lifecycle events * **Group Membership Changed** — user added to / removed from a group * **App Assignment Changed** — user assigned to / removed from an application Scope triggers by event type, target user, actor, target group, and outcome (SUCCESS/FAILURE). Trigger events expose template variables such as `{{signal.target_user}}`, `{{signal.actor}}`, `{{signal.target_group}}`, `{{signal.outcome}}`, `{{signal.client_ip}}`, and `{{signal.client_location}}` for downstream steps. **Action blocks (read):** * **Get User** — a user's profile and status (ACTIVE, LOCKED\_OUT, SUSPENDED, ...) * **List Users** — search users by name/email prefix and/or status * **Query System Log** — pull recent security events during an investigation * **List User's Groups** — blast-radius / privileged-membership check for a user * **List Group Members** — audit who is in a (privileged) group **Action blocks (response):** * **Suspend / Unsuspend User** — reversible suspension that keeps app assignments * **Unlock User** — clear a LOCKED\_OUT state after a verified lockout * **Deactivate User** — deprovision (removes app assignments; not simply reversible) * **Clear Sessions** — revoke all of a user's sessions, optionally including OAuth tokens — the first response to session hijack or credential compromise * **Expire Password** — force a password reset at next sign-in * **Reset MFA** — reset all enrolled factors (suspected factor compromise, SIM swap) * **Add / Remove User to/from Group** — quarantine a user or strip an unexpected privileged membership **Action blocks (AI):** * **Investigate Okta** — run a read-only AI investigation across users, groups, app assignments, MFA factors, and the System Log (the investigation can never suspend users, change groups, or revoke sessions) User and group selects use dynamic dropdowns backed by the Okta API and default to `{{signal.target_user}}` when the workflow starts from an Okta trigger. Example (brute force): When a **user is locked out**, query the System Log for that user's sign-in failures; if the failure count is high, clear their sessions, expire their password, and alert the security Slack channel. Example (account takeover): When **suspicious activity is reported**, immediately clear the user's sessions, request approval in Slack, then suspend the user and page the on-call via PagerDuty. Example (privileged access): When someone is **added to the "Okta Administrators" group**, post the actor and target to the security channel for review, with an approval-gated removal path. ## Disconnecting 1. Navigate to **Integrations → Okta** 2. Click **Disconnect** 3. Confirm the disconnection This stops System Log polling and all Okta workflow triggers and actions. The stored API token is deleted from Kestrel; also revoke it in the Okta Admin Console (**Security → API → Tokens**) if it is no longer needed. # OpenTelemetry Source: https://docs.usekestrel.ai/integrations/opentelemetry Receive and store OpenTelemetry metrics from your collectors for incident root cause analysis The OpenTelemetry integration enables the Kestrel Operator to receive and store OTLP metrics from your OpenTelemetry Collectors. These metrics are used during incident root cause analysis to correlate infrastructure and application performance data with Kubernetes events. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * OpenTelemetry Collector deployed in your cluster with the `k8sattributes` processor The `k8sattributes` processor is required for metrics to include Kubernetes context (namespace, workload, pod names). Without it, the operator cannot map metrics to workloads and RCA agents cannot correlate metrics with Kubernetes resources. ## Setup During cluster onboarding, select **OpenTelemetry** as the Metrics Source in the Kestrel Dashboard. Alternatively, configure via Helm values: ```yaml theme={null} operator: otel: enabled: true receiverPort: 4317 metricsStore: retention: "30m" maxSeries: 100000 ringSize: 60 ``` Ensure your OTEL Collector has the `k8sattributes` processor configured: ```yaml theme={null} processors: k8sattributes: auth_type: "serviceAccount" passthrough: false extract: metadata: - k8s.namespace.name - k8s.pod.name - k8s.pod.uid - k8s.deployment.name - k8s.statefulset.name - k8s.daemonset.name - k8s.replicaset.name - k8s.container.name - k8s.node.name pod_association: - sources: - from: resource_attribute name: k8s.pod.ip - from: resource_attribute name: ip - from: connection ``` Add a Kestrel exporter to your OTEL Collector config: ```yaml theme={null} exporters: otlp/kestrel: endpoint: kestrel-operator.kestrel-ai.svc.cluster.local:4317 tls: insecure: true service: pipelines: metrics: receivers: [otlp, prometheus] processors: [k8sattributes, batch] exporters: [otlp/kestrel] ``` The operator supports gzip compression. The OTEL Collector enables gzip by default. If using the Go SDK directly, enable it via `OTEL_EXPORTER_OTLP_COMPRESSION=gzip`. ## How It's Used ### In Workflows * **Action blocks**: Query stored OTEL metrics as workflow steps to check thresholds, compare metric values, or enrich workflow context with performance data ### In Incident Response * **Metrics correlation** — during root cause analysis, Kestrel queries stored OTEL metrics for CPU, memory, network, and application-level metrics around the incident timeframe * **Workload-aware queries** — thanks to `k8sattributes`, metrics are queryable by namespace, deployment, pod, and container ### In Cloud AI Copilot Ask questions that leverage your OpenTelemetry metrics: * `What was the request latency for the API service in the last 30 minutes?` * `Show me CPU usage trends for the payments namespace` * `Are there any workloads with high error rates right now?` ## Configuration Reference | Key | Type | Default | Description | | --------------------------------- | ------- | -------- | ------------------------- | | `operator.otel.enabled` | boolean | `false` | Enable OTLP gRPC receiver | | `operator.otel.receiverPort` | integer | `4317` | OTLP gRPC receiver port | | `operator.metricsStore.retention` | string | `"30m"` | Rolling retention window | | `operator.metricsStore.maxSeries` | integer | `100000` | Max unique metric series | | `operator.metricsStore.ringSize` | integer | `60` | Data points per series | # PagerDuty Source: https://docs.usekestrel.ai/integrations/pagerduty Route Kestrel incidents to PagerDuty with full root cause analysis, severity mapping, and service routing The PagerDuty integration connects Kestrel's automated incident detection to your existing on-call workflows. When Kestrel detects a Kubernetes or cloud incident, it creates a PagerDuty incident with full root cause analysis, affected resources, and AI-generated remediation steps. ## Prerequisites * **Organization Admin** role in Kestrel * PagerDuty account with admin access ## Setup Kestrel supports two connection methods: ### Method 1: API Token (Recommended) Enables full service management, namespace routing, and cloud resource routing. In PagerDuty: **User Icon → My Profile → User Settings → Create API User Token**. Copy the token. 1. Navigate to **Integrations → PagerDuty** in your Kestrel dashboard 2. Ensure **Use API Token** is checked 3. Paste the API token 1. Copy the **Webhook URL** shown in the setup dialog 2. In PagerDuty, go to **Integrations → Generic Webhooks (V3)** and create a subscription using the URL 3. Copy the **Signing Secret** from the webhook configuration 4. Paste the signing secret in Kestrel Click **Connect PagerDuty**, then click **Sync Services** to import your PagerDuty services. ### Method 2: Routing Key (Legacy) For simple setups that route alerts to a single service: 1. Uncheck **Use API Token** 2. In PagerDuty, add an **Events API v2** integration to your service 3. Copy the **Integration Key** (routing key) 4. Paste it in the **Integration/Routing Key** field 5. Enter a **Service Name** and click **Connect PagerDuty** ## How It's Used ### In Workflows **Trigger blocks:** * **Incident Triggered** — fires when a new PagerDuty incident is created * **Incident Acknowledged** — fires when a PagerDuty incident is acknowledged by a responder * **Incident Resolved** — fires when a PagerDuty incident is resolved * **Any PagerDuty Incident** — fires on any PagerDuty incident state change * **High Urgency Incident** — fires when a high-urgency PagerDuty incident is triggered **Action blocks:** * **Create Alert** — create a new PagerDuty alert on a specified service * **Acknowledge Alert** — acknowledge an existing PagerDuty alert * **Add Note to Alert** — attach a note with investigation context to a PagerDuty alert * **Resolve Alert** — resolve a PagerDuty alert * **Escalate** — escalate a PagerDuty incident to the next escalation level Example: A workflow triggers on a high-urgency incident, runs an automated root cause analysis, adds the findings as a note, and escalates if the issue matches a known critical pattern. ### In Incident Response * **Automatic alerting** — when Kestrel detects an incident, a PagerDuty incident is created with full RCA context * **Auto-resolution** — when incidents resolve in Kestrel, PagerDuty incidents are automatically resolved * **Bidirectional sync** — PagerDuty incidents are updated as Kestrel's investigation progresses * **Service routing** — route alerts to specific PagerDuty services based on namespace or cloud resource * **Severity mapping** — map Kestrel severity levels (Low, Medium, High, Critical) to PagerDuty severity levels ### In Cloud AI Copilot * Query PagerDuty incident status from the chat: `What PagerDuty incidents are open for the payments service?` * Trigger PagerDuty incidents from the copilot for manual escalation ## Configuration ### Namespace Routing Route Kubernetes incidents to specific PagerDuty services based on affected namespace: 1. Go to the **Namespaces** tab 2. Click **Add Assignment** → select a **Cluster**, **Namespace**, and target **PagerDuty Service** ### Cloud Resource Routing Route cloud incidents to specific PagerDuty services: 1. Go to the **Cloud Resources** tab 2. Click **Add Assignment** → select a **Cloud Resource** and target **PagerDuty Service** ### Notification Settings Configure which incidents trigger PagerDuty alerts in the **Overview** tab: * **Notify on Kubernetes incidents** — enable/disable K8s alerts * **Notify on Cloud incidents** — enable/disable AWS/OCI alerts * **Skip transient incidents** — filter out auto-resolving incidents * **Minimum severity** — set the threshold (Low, Medium, High, Critical) # PlanetScale Source: https://docs.usekestrel.ai/integrations/planetscale Connect PlanetScale for database branch, deploy-request (schema migration), backup, and credential automation in workflows The PlanetScale integration connects Kestrel to your PlanetScale organization through the PlanetScale API, enabling branch and deploy-request (schema migration) events to trigger workflows and giving AI agents read-only access to your databases' state. PlanetScale delivers branch, deploy-request, and storage events to Kestrel via native webhooks you register per-database (the webhook receiver verifies the `X-PlanetScale-Signature` against the signing secret). PlanetScale has no backup webhook, so **Backup Completed** / **Backup Failed** are detected by polling the backups API on a per-tenant cadence. Because the poller only synthesizes backup events, it never double-fires with webhook-delivered events. ## Prerequisites * **Organization Admin** role in Kestrel * A PlanetScale account with at least one database * A PlanetScale **service token** (ID + token) with `read_databases` organization access plus the database accesses for the blocks you'll use (see below) * The PlanetScale **organization** name (slug) * (For branch/deploy-request triggers) a PlanetScale webhook registered per database ## Setup In the PlanetScale dashboard at **app.planetscale.com**, open **Settings → Service tokens → New service token** (or run `pscale service-token create`). Copy the **token ID** and **token** (shown once), then grant permissions in two scopes: **Organization access** (click *Add organization permissions*): * `read_databases` — list and inspect the organization's databases (required; Kestrel uses it to validate the connection and populate dropdowns) **Database access** (click *Select permissions*; use *Permissions for all databases* to cover every database, or scope to specific ones): * `read_branch`, `create_branch`, `delete_branch` — branch blocks (create/get/list/delete, promote, safe migrations) * `read_deploy_request`, `create_deploy_request`, `approve_deploy_request` — deploy-request blocks (create/get/list/deploy/revert/close/approve) * `read_backups`, `write_backups` — backup blocks (list/create) * `connect_branch` (and `connect_production_branch` for production) — branch password blocks (list/create/delete) For a read-only setup (investigate + list/get blocks only), the `read_*` permissions alone are enough. The token is sent in the `Authorization: :` header to the PlanetScale API at `api.planetscale.com` to read and manage branches, deploy requests, and backups. 1. Navigate to **Integrations → PlanetScale** in your Kestrel dashboard. 2. Paste your **Service Token ID** and **Service Token**. 3. Enter your **Organization** name. 4. Choose a **Backup Poll Interval** (how often Kestrel checks the backups API for backup completed/failed events). 5. Paste the **webhook signing secret(s)** PlanetScale generated when you registered the webhook(s) in the next step — one per database. (You can connect first and add secrets later by reconnecting.) 6. Click **Connect PlanetScale** — your PlanetScale organization appears as connected. Needed for the branch and deploy-request triggers: 1. In the PlanetScale dashboard at **app.planetscale.com**, open the database's **Settings → Webhooks → Add webhook** (or run `pscale webhook create `). 2. Set the **URL** to the Kestrel webhook URL shown on the **Integrations → PlanetScale** connect screen (it ends in `/api/webhooks/planetscale`). 3. Under **Events**, select the ones to forward. The available events depend on the database engine: all databases expose the branch events (`branch.ready`, `branch.anomaly`, `branch.primary_promoted`, `branch.sleeping`) and `cluster.storage`; **Vitess (MySQL) databases also expose the deploy-request lifecycle** (`deploy_request.opened`, `deploy_request.errored`, `deploy_request.closed`, …), which **Postgres databases do not**. Keep **Enable this webhook** checked and **Save webhook**. 4. PlanetScale generates a unique **signing secret** for the webhook (shown on creation, and under **⋯ → Show secret**). Copy it and paste it into the **Webhook secret** field on the Kestrel connect screen. Repeat per database — paste each database's secret (Kestrel verifies every delivery against any of the stored secrets). To add a database later, paste its secret on the **Reconnect** form; existing secrets are kept. The service token grants access to your PlanetScale databases. Treat the token and the webhook signing secrets as secrets — Kestrel stores them encrypted. Branch and deploy-request triggers fire as soon as the webhook delivery is verified. Backup triggers are detected by polling, so there may be up to one poll interval of delay before a backup workflow fires. The minimum poll interval is 60 seconds. ## How It's Used ### In Workflows **Trigger blocks:** * **Deploy Request Opened / Queued / In Progress / Schema Applied / Errored / Reverted / Closed** — the deploy-request (schema migration) lifecycle (webhook; **Vitess/MySQL databases only** — Postgres databases don't emit deploy-request webhook events) * **Branch Ready** — fires when a branch is created and ready to connect (webhook) * **Branch Anomaly (Insights)** — fires on a new insights anomaly, e.g. a slow query or replication lag (webhook) * **Branch Primary Promoted** / **Branch Sleeping** — branch lifecycle (webhook) * **Storage Threshold** — fires when a cluster reaches a storage threshold (webhook) * **Backup Completed** / **Backup Failed** — fires when a branch backup completes or fails (poll-based) Filter triggers by database, branch, and event type. **Action blocks:** * **List Databases** / **Get Database** — inspect databases * **List Branches** / **Get Branch** / **Create Branch** / **Delete Branch** — manage branches * **Promote Branch to Production** — promote a branch (gate behind an Approval node) * **Set Safe Migrations** — enable/disable safe migrations on a branch * **List Deploy Requests** / **Get Deploy Request** — inspect deploy requests * **Create Deploy Request** — open a schema migration * **Deploy Deploy Request** — apply the schema change (gate behind an Approval node) * **Revert Deploy Request** — revert a deployed change (auto-revert on failure; gate behind an Approval node) * **Close Deploy Request** / **Approve Deploy Request** — close or approve a deploy request * **List Backups** / **Create Backup** — manage branch backups * **List/Create/Delete Branch Password** — manage branch credentials * **Investigate PlanetScale** — run an AI investigation of databases, branches, deploy requests, and backups PlanetScale database and branch selects accept template variables like `{{signal.database}}`, `{{signal.branch}}`, and `{{signal.deploy_request_number}}` from the trigger. Example (the flagship CI/CD flow): When a pull request is merged, create a PlanetScale deploy request, require approval, deploy the schema change, and post the result to `#releases` on Slack. Example (incident): When a deploy request errors, run an AI investigation, revert the deploy request, and page on-call via PagerDuty. ## Disconnecting 1. Navigate to **Integrations → PlanetScale** 2. Click **Disconnect** 3. Confirm the disconnection This stops all PlanetScale workflow triggers (polling and webhook). You can optionally remove the webhooks in the PlanetScale dashboard too. You can reconnect at any time. # PostHog Source: https://docs.usekestrel.ai/integrations/posthog Connect PostHog for product analytics triggers, session replay, and error investigation in workflows The PostHog integration connects Kestrel to your PostHog instance, enabling product analytics events to trigger workflows and giving AI agents access to session recordings, error data, and event queries for investigation. ## Prerequisites * **Organization Admin** role in Kestrel * PostHog account with API access * PostHog **API key** (Personal API Key) and **Project ID** ## Setup 1. In PostHog, go to **Settings → Project → Project API Key** and copy your **Project ID** 2. Go to **Settings → User → Personal API Keys** and create a new key 3. Copy the generated API key 1. Navigate to **Integrations → PostHog** in your Kestrel dashboard 2. Click **Connect** 3. Paste your **API Key** and **Project ID** 4. Enter your PostHog instance URL (e.g., `https://app.posthog.com` or your self-hosted URL) 5. Click **Connect PostHog** ## How It's Used ### In Workflows **Trigger blocks:** * **Session Error/Exception** — fires when PostHog captures an unhandled exception in a session * **Console Error** — fires on console error patterns matching your filter * **Rage Click** — fires when a user exhibits rage click behavior * **Any PostHog Event** — fires on any PostHog event matching your configured criteria * **Log Error Alert** — fires when PostHog log error patterns exceed your configured threshold **Action blocks:** * **Get Session Summary** — retrieve an AI-generated summary of a user session * **Get Session Recording** — fetch session recording details and metadata * **Query Events** — run PostHog event queries with filters and date ranges * **List Session Recordings** — list session recordings matching specified filters * **Get Error Issue** — retrieve error group details and exception information Example: A workflow triggers on a spike in rage clicks, queries the relevant session recordings, summarizes the user experience issue, and creates a Linear issue with the findings. ## Disconnecting 1. Navigate to **Integrations → PostHog** 2. Click **Disconnect** 3. Confirm the disconnection # Pulumi Cloud Source: https://docs.usekestrel.ai/integrations/pulumi Connect Pulumi Cloud for stack update and deployment automation, drift detection and remediation, deployment queue control, stack tags, outputs, and AI investigation in workflows The Pulumi Cloud integration connects Kestrel to your Pulumi Cloud organization through the Pulumi Cloud REST API, enabling stack update, deployment, drift, and policy events to trigger workflows and letting workflows run deployments, gate destructive operations behind approvals, pause/resume deployment queues, manage stack tags, and read stack outputs. Stack and deployment events are delivered by **Pulumi Cloud webhooks** that you register at the organization or stack level. Each delivery carries a `Pulumi-Webhook-Kind` header identifying the event and is signed with HMAC-SHA256 (`Pulumi-Webhook-Signature` header) using the webhook secret Kestrel generates at connect time, and Kestrel verifies the signature before evaluating workflows. ## Prerequisites * **Organization Admin** role in Kestrel * A Pulumi Cloud organization with at least one stack * A Pulumi Cloud **access token** with access to the organization — an **organization token** or **team token** is recommended over a personal token * Deployment actions (Run Deployment, Cancel, Pause/Resume) require **Pulumi Deployments** to be configured on the target stacks ## Setup In Pulumi Cloud (`app.pulumi.com`), go to your organization's **Settings → Access Tokens**. Create an **organization token** (or a team token scoped to the stacks your workflows manage). Copy the token — it is shown only once. 1. Navigate to **Integrations → Pulumi Cloud** in your Kestrel dashboard. 2. Paste your **access token**. 3. Enter your **Organization** name (e.g. `my-org`). 4. Optionally set a custom **API base URL** for self-hosted Pulumi (defaults to `https://api.pulumi.com`). 5. Click **Connect Pulumi Cloud** — Kestrel validates the token by listing your organization's stacks and generates a **webhook secret** for delivery verification. Actions work as soon as the token is connected. To have Pulumi Cloud events **trigger** workflows, add a webhook at the organization level (all stacks) or per stack: 1. On the Kestrel Pulumi Cloud integration page, copy the **Webhook URL** and the **Webhook secret**. 2. In Pulumi Cloud, open your organization's **Settings → Integrations**, scroll to the **Webhooks** section, and click **Add webhook** (for per-stack events, use the stack's **Settings → Webhooks** instead). 3. Under **Destination**, choose **Webhook** (not Slack or Microsoft Teams) and give it a display name (e.g. `kestrel`). 4. Set the **Payload URL** to the copied webhook URL and paste the webhook secret into the **Secret** field. The secret is required — Kestrel rejects unsigned deliveries. 5. Under **Triggers**, check these boxes — Kestrel needs all of these event groups: * **All Stack events** — powers **Stack Created**, **Stack Deleted**, **Update Succeeded**, **Update Failed**, **Preview Failed**, and **Destroy Succeeded**. * **All Deployment events** — powers **Deployment Started / Succeeded / Failed**, **Drift Detected**, and **Drift Run Failed** (the drift detection and remediation checkboxes live in this section). * **All Policy events** — powers **Policy Violation** (both mandatory and advisory violations). * **All Environment events** and **All Change requests** — also needed; Kestrel consumes these event groups as well. 6. Click **Create**. Pulumi Cloud sends a `ping` delivery on creation, which Kestrel acknowledges — the integration page's webhook event counter increments once the first delivery is verified. The access token can run deployments that change or destroy real infrastructure. Treat it as a secret — Kestrel stores it encrypted. Gate destructive workflow actions (Run Deployment with destroy or remediate-drift operations) behind Approval nodes. Deployment and drift events (`deployment_*`, `drift_detected`) require **Pulumi Deployments** with drift detection schedules configured on the stack (Pulumi Cloud Team/Enterprise). Update events (`update_succeeded`, `update_failed`) fire for any `pulumi up`, including CLI-driven updates. ## How It's Used ### In Workflows **Trigger blocks (webhook):** * **Update Succeeded** — fires when a stack update (`pulumi up`) succeeds * **Update Failed** — fires when a stack update, refresh, or destroy fails * **Preview Failed** — fires when a stack preview fails (broken program or config before anything is applied) * **Destroy Succeeded** — fires when a stack destroy succeeds (all managed resources torn down) * **Deployment Started / Succeeded / Failed** — fire as a Pulumi Deployments run progresses * **Drift Detected** — fires when a scheduled drift detection run finds infrastructure drift from the declared program state * **Drift Run Failed** — fires when a drift detection or remediation run fails to complete * **Policy Violation** — fires when a CrossGuard policy violation is detected during a stack operation * **Stack Created / Stack Deleted** — fire when stacks are created or removed in the organization Filter triggers by stack (project/stack reference), project name, and event type. Trigger events expose template variables such as `{{signal.stack}}`, `{{signal.project}}`, `{{signal.operation}}`, `{{signal.result}}`, `{{signal.update_version}}`, `{{signal.deployment_id}}`, `{{signal.organization}}`, `{{signal.user}}`, and `{{signal.update_url}}` for downstream steps. **Action blocks:** * **List Stacks** / **Get Stack** — inventory (last update time, resource counts, version, tags, in-flight operations) * **List Updates** / **Get Update** — update history and an update's result plus resource change counts (great for Slack notifications and approval context) * **Run Deployment** — start a Pulumi Deployments run (update, preview, refresh, destroy, detect-drift, or remediate-drift) using the stack's saved deployment settings (gate destroy and remediate-drift behind an Approval node) * **Get Deployment** / **Wait for Deployment** — check a deployment's status, or poll it until it reaches a terminal state (succeeded, failed, skipped) and branch on the result * **Cancel Deployment** — cancel a queued or running deployment * **Pause Deployments** / **Resume Deployments** — freeze or unfreeze a stack's deployment queue during incidents and maintenance windows * **Get Stack Outputs** — read the stack's current outputs from its latest state (secret outputs are redacted) to wire infrastructure values into later steps * **Get Drift Status** — check the latest drift detection runs for a stack * **Set Stack Tag** / **Delete Stack Tag** — mark stacks (e.g. quarantined, incident ID) or clean up tags * **Investigate Pulumi** — run a read-only AI investigation across stacks, updates, deployments, outputs, and drift Stack selects use dynamic dropdowns backed by the Pulumi Cloud API and accept template variables like `{{signal.stack}}` and `{{signal.deployment_id}}` from the trigger. Example (failure triage — the flagship Pulumi flow): When an **update fails** on a production stack, run an AI investigation, post the findings to Slack with approve/reject buttons, then **Run Deployment** (update) to retry on approval. Example (drift): When **drift is detected**, fetch the drift status, post a summary to Slack and ask for approval, then **Run Deployment** with the remediate-drift operation on approval. Example (incident): When a **deployment fails**, investigate the failure with AI, page on-call via PagerDuty with the summary and update URL, and **Pause Deployments** on the stack until the incident is resolved. ## Disconnecting 1. Navigate to **Integrations → Pulumi Cloud** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Pulumi Cloud workflow triggers and actions. Webhooks in Pulumi Cloud will keep sending deliveries until you remove them there; unverified deliveries are ignored. You can reconnect at any time. # Railway Source: https://docs.usekestrel.ai/integrations/railway Connect Railway for deployment event triggers, log access, and rollback/redeploy actions in workflows The Railway integration connects Kestrel to your Railway account, enabling deployment and platform events to trigger workflows and giving AI agents access to deployment logs, status, and rollback/redeploy capabilities across your projects, services, and environments. ## Prerequisites * **Organization Admin** role in Kestrel * Railway account with project access ## Setup 1. Go to **railway.com/account/tokens** (Account Settings → Tokens) 2. Leave the **Workspace** dropdown set to **"No workspace"** to mint an account-scoped token. An account-scoped token can read projects, services, deployments, and logs across your whole account. 3. Create the token and copy it — you won't be able to view it again. 1. Navigate to **Integrations → Railway** in your Kestrel dashboard 2. Click **Connect** 3. Paste the API token. It is sent as a Bearer token to Railway's GraphQL API (`https://backboard.railway.com/graphql/v2`). 4. Save — your Railway account appears as connected. Railway delivers deployment and platform events via outbound webhooks. Kestrel exposes a unique webhook URL on the **Integrations → Railway** page. 1. Copy the webhook URL from Kestrel and choose a secret. 2. Add the webhook URL to your Railway project's **Settings → Webhooks**. Railway does not sign webhooks, so the secret embedded in the URL is how Kestrel attributes incoming events to your account. Keep the URL private. ## How It's Used ### In Workflows **Trigger blocks:** * **Deployment Failed** — fires when a Railway deployment build fails * **Deployment Crashed** — fires when a running deployment crashes * **Deployment Succeeded** — fires when a deployment completes successfully * **Volume Usage Alert** — fires when a volume crosses a usage threshold * **CPU/RAM Monitor Alert** — fires when a service exceeds CPU or memory thresholds Filter triggers by project, environment, and event type. **Action blocks:** * **Get Deployment** — retrieve deployment details including status and git commit * **Get Deployment Logs** — fetch build/runtime log output for a deployment * **Rollback** — roll a service back to its previous deployment * **Redeploy** — redeploy the current deployment * **Restart** — restart a service * **List Deployments** — list recent deployments with optional filters * **Set Variables** — set service environment variables * **Investigate Railway** — run an AI investigation combining deployment logs, status, and history Railway service and environment selects accept template variables like `{{signal.service_id}}` and `{{signal.environment_id}}` from the trigger. Example: A workflow triggers on a crashed deployment, fetches the logs, analyzes the failure with AI, rolls the service back, and posts a summary to Slack. ## Disconnecting 1. Navigate to **Integrations → Railway** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Railway workflow triggers. Remove the webhook from your Railway project to stop event delivery. You can reconnect at any time. # Slack Source: https://docs.usekestrel.ai/integrations/slack Connect Slack for real-time incident notifications, interactive Q&A, workflow triggers, and AI copilot commands The Kestrel Slack integration brings incident response, workflow automation, and AI copilot capabilities directly into your team's chat. Receive real-time notifications, trigger workflows from Slack, and use slash commands to interact with your infrastructure. ## Prerequisites * **Organization Admin** role in Kestrel * Slack workspace admin permissions (to authorize the app) ## Setup 1. Navigate to **Integrations → Slack** in your Kestrel dashboard 2. Click **Connect Slack Workspace** 3. Authorize the Kestrel app in the Slack OAuth popup 4. The popup closes and your workspace appears as connected Invite the Kestrel bot to channels where you want notifications: ``` /invite @Kestrel ``` Click **Sync Now** on the User Mapping card to sync Slack users with Kestrel accounts. This enables smart @mentions for namespace owners. Create a dedicated `#incidents` channel for Kestrel notifications to keep them organized. ## How It's Used ### In Workflows **Trigger blocks:** * **Create K8s Resource** — fires when a user requests Kubernetes resource creation via `/kestrel-workflow` * **Edit K8s Resource** — fires when a user requests a Kubernetes resource edit via `/kestrel-workflow` * **General Kubernetes Request** — fires on any Kubernetes-related `/kestrel-workflow` request * **Create AWS Resource** — fires when a user requests AWS resource creation via `/kestrel-workflow` * **Edit AWS Resource** — fires when a user requests an AWS resource edit via `/kestrel-workflow` * **General Cloud Request** — fires on any cloud-related `/kestrel-workflow` request * **Any Slack /kestrel-workflow Request** — fires on any `/kestrel-workflow` command invocation **Action blocks:** * **Send Message** — post a message to a Slack channel or DM * **Update Message** — update an existing Slack message in place * **Request Justification** — send an interactive approval/justification prompt and wait for a response Example: A workflow triggers on a Create K8s Resource request from Slack, sends a justification prompt to the requester's manager, and upon approval creates the resource and posts a confirmation. The `/kestrel-workflow` command lets any team member trigger pre-built workflows without leaving Slack — for example, `/kestrel-workflow restart-pod namespace=payments pod=api-server`. ### In Incident Response * **Real-time notifications** — posts to your configured channel with incident summary, severity, affected resources, root cause analysis, and remediation steps * **Smart @mentions** — namespace owners are automatically @mentioned when incidents affect their namespaces * **Interactive Q\&A** — reply in the incident notification thread to ask follow-up questions with full incident context * **Knowledge source** — Slack message history is searchable during incident RCA (see [Knowledge Sources](/integrations/knowledge-sources)) ### In Cloud AI Copilot Use the `/kestrel` slash command anywhere in Slack to ask questions about your infrastructure: * `/kestrel what pods are crashing in production?` * `/kestrel show me the network policies for the payments namespace` * `/kestrel what Terraform code manages my S3 buckets?` ## Disconnecting 1. Navigate to **Integrations → Slack** 2. Click **Disconnect Slack** 3. Confirm the disconnection This immediately stops all notifications and removes the bot from your workspace. You can reconnect at any time. # SonarCloud Source: https://docs.usekestrel.ai/integrations/sonarcloud Connect SonarCloud for code quality automation — react to failed quality gates, triage vulnerabilities and hotspots, and track code health in workflows The SonarCloud integration connects Kestrel to your SonarCloud organization through the SonarCloud Web API, enabling completed or failed analyses to trigger workflows and letting workflows check quality gates, list issues and security hotspots, fetch code quality metrics, and triage findings (transition, assign, comment, review) with human approval. ## Prerequisites * **Organization Admin** role in Kestrel * A SonarCloud account with access to the organization you want to automate * A SonarCloud **API token** and your **organization key** ## Setup In SonarCloud, go to **your avatar → My Account → Access Tokens → Generate Token**. Copy the token — it is shown only once. The key appears in your organization's URL (`sonarcloud.io/organizations/`) or under **Administration → Organization settings**. 1. Navigate to **Integrations → SonarCloud** in your Kestrel dashboard. 2. Enter your **organization key** and paste the **API token**. 3. Click **Connect SonarCloud** — Kestrel validates the token against your organization and returns a **webhook secret**. In SonarCloud, open your organization and select **Webhooks** in the left sidebar (for an individual project: **Administration → Webhooks**), then click **Create** with: * **URL**: `https:///api/webhooks/sonarcloud` * **Secret**: the webhook secret returned by the connect step SonarCloud signs each delivery with the secret via the `X-Sonar-Webhook-HMAC-SHA256` header; Kestrel rejects unsigned or mis-signed payloads. Analysis triggers are webhook-delivered in real time. Action blocks work without the webhook — only trigger blocks require it. ## How It's Used ### In Workflows **Trigger blocks:** * **Analysis Completed** — fires when a SonarCloud analysis finishes, with the quality gate result attached * **Analysis Failed** — fires when an analysis task breaks Scope triggers by project key, branch, and quality gate status (`OK`, `ERROR`, `NONE`). Trigger events expose template variables such as `{{signal.project_key}}`, `{{signal.branch}}`, `{{signal.quality_gate_status}}`, `{{signal.failed_conditions}}`, and `{{signal.project_url}}` for downstream steps. **Action blocks (read):** * **Get Quality Gate Status** — a project's quality gate result with its failed conditions * **List Issues** — open issues filtered by type (vulnerability/bug/code smell), severity, and new-code-only * **List Security Hotspots** — security hotspots awaiting review * **Get Project Metrics** — code quality measures such as coverage, duplication, and technical debt **Action blocks (triage):** * **Transition Issue** — confirm, resolve, reopen, mark false positive, or won't-fix an issue * **Assign Issue** — assign an issue to a user * **Comment on Issue** — add a comment to an issue * **Review Hotspot** — resolve a hotspot as SAFE/FIXED/ACKNOWLEDGED or send it back to review **Action blocks (AI):** * **Investigate SonarCloud** — run a read-only AI investigation across quality gates, issues, hotspots, metrics, and analysis history (the investigation can never transition issues or change settings) Project and branch selects use dynamic dropdowns backed by the SonarCloud API and default to `{{signal.project_key}}` / `{{signal.branch}}` when the workflow starts from a SonarCloud trigger. Triage actions take an issue or hotspot key — typically `{{item.key}}` inside a For Each over a list step's output — and pair naturally with Approval nodes for human-in-the-loop triage. Example (security): When the **quality gate fails on main**, list new BLOCKER/CRITICAL vulnerabilities, post them to the security Slack channel, and open a Jira ticket. Example (triage): A developer asks in the request chat to mark an issue false positive. The workflow requests approval in Slack, transitions the issue, and comments with the approver's name. Example (code health): On a weekly schedule, fetch project metrics and post a coverage/duplication digest to the team channel. ## Disconnecting 1. Navigate to **Integrations → SonarCloud** 2. Click **Disconnect** 3. Confirm the disconnection This disables all SonarCloud workflow triggers and actions. The stored API token is deleted from Kestrel; also delete the webhook in SonarCloud and revoke the token under **My Account → Access Tokens** if it is no longer needed. # Supabase Source: https://docs.usekestrel.ai/integrations/supabase Connect Supabase for project, branch, backup, and read-replica lifecycle, network restrictions, and AI investigation in workflows The Supabase integration connects Kestrel to your Supabase organization through the Supabase Management API, enabling control-plane events (project health, backups, read replicas, and database branches) to trigger workflows and giving AI agents read-only access to your projects' state. Supabase has no tenant-wide outbound webhook for control-plane events, so Kestrel polls the Supabase Management API on a per-tenant cadence to detect project health degradation, backup failures, read-replica health, and branch status. The one webhook-delivered trigger is **Database Row Event**: you register a Supabase Database Webhook that posts table `INSERT`/`UPDATE`/`DELETE` events to Kestrel. ## Prerequisites * **Organization Admin** role in Kestrel * A Supabase account with at least one project * A Supabase **Personal Access Token** * (Optional) A Database Webhook signing secret, only needed for the Database Row Event trigger ## Setup In the Supabase dashboard, open **Account → Access Tokens**, click **Generate new token**, give it a name (e.g. `kestrel`), and copy the generated token (shown only once). The token is sent as a bearer token to `https://api.supabase.com` to read project, branch, backup, and health state and perform control-plane actions. 1. Navigate to **Integrations → Supabase** in your Kestrel dashboard. 2. Paste your **Personal Access Token**. 3. (Optional) Set a custom **API URL** for self-hosted/enterprise endpoints. 4. Choose a **Poll Interval** (how often Kestrel checks the Management API for health/backup/replica/branch events). 5. (Optional) Set a **Database Webhook Secret** if you plan to use the Database Row Event trigger. 6. Click **Connect Supabase** — your Supabase organization appears as connected. Only needed for the **Database Row Event** trigger: 1. In the Supabase dashboard, open **Database → Webhooks** and create a webhook on the table(s) you want to watch. 2. Set the **URL** to the Kestrel webhook URL shown on the **Integrations → Supabase** connect screen (it ends in `/api/webhooks/supabase`). 3. Add an HTTP header **`X-Supabase-Webhook-Secret`** whose value matches the signing secret you set when connecting. 4. Select the table events (insert/update/delete) you want to forward. The Personal Access Token grants access to your Supabase projects and their control plane. Treat it and the webhook secret as secrets — Kestrel stores both encrypted. Control-plane triggers are detected by polling, so there may be up to one poll interval of delay before a workflow fires. The minimum poll interval is 60 seconds. Database Row Event triggers fire as soon as the webhook delivery is verified. ## How It's Used ### In Workflows **Trigger blocks:** * **Project Health Degraded** — fires when a project service (db, auth, rest, realtime, storage, pooler) is unhealthy (poll-based) * **Backup Failed** — fires when a project's most recent database backup failed (poll-based) * **Read Replica Unhealthy** — fires when a project read replica is unhealthy (poll-based) * **Usage/Quota Threshold** — fires when a project crosses a usage/quota threshold (poll-based) * **Branch Created** — fires when a new database branch is created (poll-based) * **Branch Migration Failed** — fires when a database branch enters a failed migration state (poll-based) * **Database Row Event** — fires on table INSERT/UPDATE/DELETE delivered by a Supabase Database Webhook Filter triggers by project ref and event type (and by table, for Database Row Events). **Action blocks:** * **List Projects** / **Get Project** / **Get Project Health** — inspect projects and per-service health * **Create Branch** — create an ephemeral preview database branch (ideal for PR previews) * **List Branches** / **Get Branch** — inspect branches and their connection refs * **Merge Branch** / **Reset Branch** — run or redo a branch's migrations against its parent * **Delete Branch** — permanently delete a branch (gate behind an Approval node) * **List Backups** — list backups and PITR availability * **Create Restore Point** — create a named checkpoint before a risky migration * **Restore Backup (PITR)** — point-in-time restore (destructive; gate behind an Approval node) * **Setup Read Replica** / **Remove Read Replica** — manage read replicas * **Pause Project** / **Restore Project** — pause/un-pause a project for cost control * **Get Network Restrictions** / **Update Network Restrictions** — review and lock down a project's allowed CIDRs * **List API Keys** — list API key metadata (secret values are never returned) * **Investigate Supabase** — run an AI investigation of unhealthy projects, failed backups, and branch failures Supabase project and branch selects accept template variables like `{{signal.project_ref}}` and `{{signal.branch_id}}` from the trigger. Example: When a pull request opens, create a Supabase preview database branch, run migrations, and post the branch connection details to Slack; when the PR closes, delete the branch. Example: When a project backup fails, run an AI investigation, page on-call via PagerDuty, and open a Jira ticket. ## Disconnecting 1. Navigate to **Integrations → Supabase** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Supabase workflow triggers (polling and webhook). You can optionally remove the Database Webhook in the Supabase dashboard too. You can reconnect at any time. # Terraform Cloud Source: https://docs.usekestrel.ai/integrations/terraform Connect Terraform Cloud (HCP Terraform) for run lifecycle automation (plan/apply/discard/cancel), workspace locking, variables, state outputs, drift detection, and AI investigation in workflows The Terraform Cloud integration connects Kestrel to your Terraform Cloud (HCP Terraform) organization through the Terraform Cloud API, enabling run lifecycle and drift events to trigger workflows and letting workflows queue plans, gate applies behind approvals, manage workspace locks and variables, and read state outputs. Run and health-assessment events are delivered by **workspace notification configurations** (webhooks) that you register per workspace. Each delivery is signed with HMAC-SHA512 (`X-TFE-Notification-Signature` header) using the notification token Kestrel generates at connect time, and Kestrel verifies the signature before evaluating workflows. ## Prerequisites * **Organization Admin** role in Kestrel * A Terraform Cloud (or Terraform Enterprise) organization with at least one workspace * A Terraform Cloud **API token** with access to the organization — a **team token** or **organization token** is recommended over a user token ## Setup In Terraform Cloud (`app.terraform.io`), go to your organization's **Settings → API tokens** (or **Account Settings → Tokens** for a user token). Create a **team token** for a team that has the workspace permissions your workflows need (queue plans, apply runs, lock workspaces, manage variables). Copy the token — it is shown only once. 1. Navigate to **Integrations → Terraform Cloud** in your Kestrel dashboard. 2. Paste your **API token**. 3. Enter your **Organization** name (e.g. `my-org`). 4. Optionally set a custom **API base URL** for Terraform Enterprise (defaults to `https://app.terraform.io/api/v2`). 5. Click **Connect Terraform Cloud** — Kestrel validates the token against your organization and generates a **notification token** for webhook verification. Actions work as soon as the token is connected. To have Terraform Cloud events **trigger** workflows, add a notification configuration to each workspace you care about: 1. On the Kestrel Terraform Cloud integration page, copy the **Webhook URL** and the **Notification token**. 2. In Terraform Cloud, go to **Workspaces** and open the workspace you want events from. Notifications are a **per-workspace** setting — you won't find them under Organization Settings. 3. In the workspace's sidebar, open **Settings → Notifications** and click **Create a Notification**. 4. Choose the **Webhook** destination, set the URL to the copied webhook URL, and paste the notification token as the **Token**. 5. Select the run events you want (Created, Planning, Needs Attention, Applying, Completed, Errored). 6. If the workspace has **health assessments** enabled (HCP Terraform Plus/Premium), a **Health Events** section also appears — select **Drifted** and **Check Failed** there to enable drift triggers. On workspaces without health assessments, these options are not shown. 7. Repeat for each workspace that should trigger workflows. The API token can queue and apply runs that change real infrastructure. Treat it as a secret — Kestrel stores it encrypted. Gate destructive workflow actions (Apply Run, Create Destroy Run, Force-Unlock, Set Variable) behind Approval nodes. Drift events require **health assessments** to be enabled on the workspace (Terraform Cloud Plus/Premium). Health assessment notifications (`assessment:drifted`, `assessment:check_failure`) are only sent for workspaces with assessments turned on. ## How It's Used ### In Workflows **Trigger blocks (webhook):** * **Run Created** — fires when a new run is queued on a workspace * **Run Planning** — fires when a run starts planning * **Run Needs Attention** — fires when a plan finishes and the run awaits confirmation (the moment to route an approval) * **Run Applying** — fires when a run starts applying * **Run Completed** — fires when a run applies successfully or finishes as planned-and-finished * **Run Errored** — fires when a run fails (plan error, apply error, or policy hard-fail) * **Drift Detected** — fires when a workspace health assessment detects infrastructure drift from state * **Assessment Check Failed** — fires when a workspace health assessment fails to run Filter triggers by workspace name, event type, and run status. Trigger events expose template variables such as `{{signal.workspace}}`, `{{signal.run_id}}`, `{{signal.run_status}}`, `{{signal.organization}}`, and `{{signal.run_url}}` for downstream steps. **Action blocks:** * **List Workspaces** / **Get Workspace** — inventory (lock state, execution mode, Terraform version, resource counts) * **Lock Workspace** / **Unlock Workspace** — block or allow runs during incidents and maintenance windows * **Force-Unlock Workspace** — recover a workspace stuck by a crashed run's lock (gate behind an Approval node) * **List Runs** / **Get Run** — run history and a run's status plus planned adds/changes/destroys (great for Slack notifications and approval context) * **Create Run (Plan)** — queue a plan run on a workspace, optionally auto-apply * **Create Destroy Run** — queue a destroy plan (gate behind an Approval node) * **Apply Run** — confirm and apply a run awaiting confirmation (gate behind an Approval node for production) * **Discard Run** / **Cancel Run** — discard a pending plan or interrupt an in-progress run * **Wait for Run** — poll a run until it reaches a terminal state (applied, errored, discarded, canceled) and branch on the result * **Get State Outputs** — read the workspace's current state outputs (sensitive values are redacted) to wire infrastructure values into later steps * **List Variables** / **Set Variable** — inspect and update workspace Terraform/environment variables (sensitive values are never returned; gate Set Variable behind an Approval node) * **Get Drift Assessment** — check the latest health assessment result for drift * **Investigate Terraform** — run a read-only AI investigation across workspaces, runs, state outputs, variables, and drift Workspace and run selects use dynamic dropdowns backed by the Terraform Cloud API and accept template variables like `{{signal.workspace}}` and `{{signal.run_id}}` from the trigger. Example (approval gating — the flagship Terraform flow): When a run **needs attention** on a production workspace, fetch the run's planned changes, post them to Slack with approve/reject buttons, then **Apply Run** on approval or **Discard Run** on rejection. Example (drift): When **drift is detected**, run an AI investigation, queue a **plan run** to preview remediation, and open a Jira ticket with the findings. Example (incident): When a run **errors**, investigate the failure with AI and page on-call via PagerDuty with the summary and run URL. ## Disconnecting 1. Navigate to **Integrations → Terraform Cloud** 2. Click **Disconnect** 3. Confirm the disconnection This stops all Terraform Cloud workflow triggers and actions. Notification configurations in Terraform Cloud will keep sending deliveries until you remove them there; unverified deliveries are ignored. You can reconnect at any time. # Trivy Source: https://docs.usekestrel.ai/integrations/trivy Connect Trivy for cluster security scanning automation — react to new CVEs, exposed secrets, misconfigurations, and compliance failures, and query scan results in workflows The Trivy integration connects Kestrel to your in-cluster [trivy-operator](https://github.com/aquasecurity/trivy-operator) installation, enabling new security findings to trigger workflows and letting workflows list image vulnerabilities, exposed secrets, and misconfigurations, read compliance reports, and force workload rescans. ## Prerequisites * Kestrel Operator deployed in your cluster (see [Kubernetes](/integrations/kubernetes)) * trivy-operator installed in the same cluster with its CRDs present (`vulnerabilityreports.aquasecurity.github.io` and friends) ## Setup Trivy is auto-detected by the Kestrel Operator when the trivy-operator CRDs are present in the cluster. No manual connection or credentials are required. If trivy-operator is not already installed: ```bash theme={null} helm repo add aquasecurity https://aquasecurity.github.io/helm-charts helm install trivy-operator aquasecurity/trivy-operator \ -n trivy-system --create-namespace ``` The operator probes for the trivy-operator CRDs periodically. Check the operator logs: ```bash theme={null} kubectl logs -n kestrel-ai deploy/kestrel-operator | grep -i "Trivy" ``` Once detected, the integration shows as **Detected** on the Integrations page and Trivy trigger and action blocks become available in the workflow builder. The integration is enabled by default. To opt out, set `operator.trivy.enabled: false` in your Helm values and upgrade the operator (this sets `TRIVY_DISABLED=true` on the operator deployment). ## How It's Used ### In Workflows **Trigger blocks:** * **Vulnerability Detected** — fires when trivy-operator records new or changed image CVEs in a workload's `VulnerabilityReport` (a vulnerable image shipped to the cluster) * **Exposed Secret Detected** — fires when a secret (AWS keys, tokens, private keys) is found baked into a container image * **Config Audit Failed** — fires when a workload or RBAC object fails security checks (privileged containers, missing runAsNonRoot, wildcard RBAC) * **Compliance Check Failed** — fires when a cluster compliance report (CIS, NSA, Pod Security Standards) has failing controls Triggers deduplicate by report content, so trivy-operator's periodic report recreation (its report TTL) does not re-fire unchanged findings — only new or changed findings trigger workflows. Scope triggers by cluster, namespace, severity (`critical`, `high`, `medium`, `low`), and scanned resource kind. Trigger events expose template variables such as `{{signal.image}}`, `{{signal.workload_kind}}`, `{{signal.workload_name}}`, `{{signal.namespace}}`, `{{signal.severity}}`, `{{signal.cve_ids}}`, `{{signal.critical_count}}`, `{{signal.high_count}}`, `{{signal.check_ids}}`, `{{signal.secret_rules}}`, and `{{signal.compliance_report}}` for downstream steps. **Action blocks:** * **List Vulnerabilities** — aggregate image CVEs across the cluster, filterable by namespace, workload, severity, and fix availability * **Get Vulnerability Report** — read one workload's full vulnerability picture: per-container images, severity counts, and the CVE list * **List Exposed Secrets** — list secrets baked into container images with rule, severity, target file, and workload * **List Misconfigurations** — list failed security checks from config audit, RBAC assessment, and infra assessment reports * **Get Compliance Report** — read one cluster compliance report's pass/fail counts and failing controls * **Rescan Workload** — force trivy-operator to rescan a workload by deleting its VulnerabilityReports (verify a patched image actually cleared its CVEs) * **Investigate Trivy** — run a read-only AI investigation across CVEs, exposed secrets, misconfigurations, and compliance via the cluster operator (the investigation can never rescan or mutate anything) Example (vulnerability response): When a **critical CVE** ships to production, post the image, workload, and top CVEs to your security Slack channel and open a Jira ticket with the fix versions. Example (secret hygiene): When an **exposed secret** is detected in an image, alert the owning team immediately so the credential can be rotated, and file a tracking ticket. Example (patch verification): A developer asks in the request chat to rescan the `nginx` deployment after deploying a patched image. The workflow **Rescan Workload**, waits, then **Get Vulnerability Report** and confirms the CVEs are gone. Example (posture reporting): On a weekly schedule, **List Vulnerabilities** with fixable-only, summarize with AI, and post a patch-priority digest to Slack. ### Scoping Trivy actions are scoped by **cluster** (populated with a dropdown of clusters where trivy-operator was detected) and, for workload-level actions, **namespace** and **workload** (accepts `kind/name` like `deployment/nginx`). Compliance reads are scoped by **report name** (e.g. `cis`, `nsa`), also populated from live cluster data. ## Disconnecting To disable the Trivy integration, set the following in your Helm values and upgrade the operator: ```yaml theme={null} operator: trivy: enabled: false ``` # HashiCorp Vault Source: https://docs.usekestrel.ai/integrations/vault Connect HashiCorp Vault for secret rotation automation, seal-status monitoring, stale-secret detection, policy and auth-method change alerts, lease management, and AI investigation in workflows The HashiCorp Vault integration connects Kestrel to your Vault cluster (self-hosted or HCP Vault Dedicated) through the Vault HTTP API, enabling seal-status, secret-version, policy, and auth-method changes to trigger workflows and letting workflows read and write KV v2 secrets, rotate database static roles, manage ACL policies, revoke leases and tokens, and run read-only AI investigations. Vault has no outbound control-plane webhooks (Vault Enterprise event notifications require a WebSocket subscriber inside your network), so Kestrel polls the Vault API on a per-tenant cadence to detect seal-status changes, new secret versions, stale secrets, and policy or auth-method changes. Polling reads **only metadata** — secret values are never read or stored by the pollers. ## Prerequisites * **Organization Admin** role in Kestrel * A reachable Vault cluster (the Kestrel control plane must be able to reach the Vault address over HTTPS) * A Vault **token** for Kestrel — a periodic service token bound to a dedicated policy is recommended over a root token * KV v2 secrets engines for secret triggers and secret actions; the database secrets engine for static-role rotation ## Setup Create a dedicated policy that grants only what your workflows need. A typical policy grants `read`, `list` on `sys/health`, `sys/mounts`, `sys/policies/acl/*`, and `sys/auth`, plus `read`, `list` on the metadata paths of the KV mounts you want monitored (e.g. `secret/metadata/*`), and — only if your workflows write secrets or rotate credentials — `create`, `update` on the matching `secret/data/*` and `database/rotate-role/*` paths. Then create a token bound to that policy: ```bash theme={null} vault token create -policy=kestrel -period=768h -orphan ``` 1. Navigate to **Integrations → Vault** in your Kestrel dashboard. 2. Enter your **Vault address** (e.g. `https://vault.example.com:8200`). 3. Paste the **token**. 4. Optionally set a **namespace** (Vault Enterprise / HCP Vault). 5. Click **Connect Vault** — Kestrel validates the connection by reading `sys/health` and starts polling. The token can read and write real secrets, depending on the policy you attach. Scope the policy to the narrowest set of paths your workflows need, and gate destructive workflow actions (Delete Secret, Revoke Lease, Revoke Token) behind Approval nodes. Kestrel stores the token encrypted. All Vault triggers are detected by polling, so there may be up to one poll interval of delay before a workflow fires. The poll cadence is configurable per trigger (1m–30m; seal-status checks default to 1 minute, secret and policy checks to 5 minutes). Pollers read only KV **metadata** — never secret values. ## How It's Used ### In Workflows **Trigger blocks (all poll-based):** * **Vault Sealed / Vault Unsealed** — fire when the cluster's seal status changes (a sealed Vault cannot serve secrets) * **Health Degraded** — fires when Vault becomes unreachable or reports an unhealthy status * **Secret Version Created** — fires when a KV v2 secret gets a new version (metadata diff; values are never read) * **Secret Stale** — fires when a secret's latest version exceeds a rotation age threshold (default 90 days, configurable per trigger) * **Policy Created / Deleted** — fires when ACL policies change * **Auth Method Enabled / Disabled** — fires when authentication methods change Filter secret triggers by mount and secret-path prefix. Trigger events expose template variables such as `{{signal.mount}}`, `{{signal.secret_path}}`, `{{signal.secret_version}}`, `{{signal.secret_age_days}}`, `{{signal.policy_name}}`, and `{{signal.auth_method}}` for downstream steps. **Action blocks:** * **Read Secret** — read a KV v2 secret (the value output is marked sensitive: it is redacted in run history but can be templated into downstream steps, e.g. to sync into another system) * **Write Secret** — create or update a KV v2 secret (creates a new version); the data can reference previous step outputs, e.g. a freshly generated credential * **Delete Secret** — soft-delete the latest version, or permanently destroy all versions (gate behind an Approval node) * **List Secrets** / **Get Secret Metadata** — enumerate secret names and check version/age metadata without reading values * **Rotate Static Role** — rotate the credentials of a database static role immediately * **List Mounts** / **List Auth Methods** — inventory of secrets engines and auth methods * **List Policies** / **Read Policy** / **Write Policy** — inspect and manage ACL policies from HCL * **List Leases** / **Revoke Lease** / **Renew Lease** — manage dynamic-credential leases (revoke during credential-leak response) * **Get Health** — check seal/init status and server version * **List Token Accessors** / **Revoke Token Accessor** — audit and revoke tokens without knowing the token itself * **Investigate Vault** — run a read-only AI investigation across health, mounts, secret metadata, policies, and leases (the investigation can never read secret values) Mount, secret path, policy, and static-role selects use dynamic dropdowns backed by the Vault API and accept template variables like `{{signal.mount}}` and `{{signal.secret_path}}` from the trigger. Example (rotation — the flagship Vault flow): When a **secret goes stale** in `secret/app/prod`, post the secret's metadata to Slack with approve/reject buttons, then **Rotate Static Role** (or write a newly generated value with **Write Secret**) on approval and confirm the new version. Example (incident): When **Vault seals**, page on-call via PagerDuty with the health summary and open a Jira ticket, then post an all-clear to Slack when the **Unsealed** trigger fires. Example (security): When an **auth method is enabled**, run an AI investigation of recent policy and auth changes and post the findings to your security Slack channel. ## Disconnecting 1. Navigate to **Integrations → Vault** 2. Click **Disconnect** 3. Confirm the disconnection This stops polling and all Vault workflow triggers and actions. The stored token is deleted from Kestrel; revoke it in Vault (`vault token revoke`) if it is no longer needed. You can reconnect at any time. # Vercel Source: https://docs.usekestrel.ai/integrations/vercel Connect Vercel for deployment event triggers, build log access, and rollback actions in workflows The Vercel integration connects Kestrel to your Vercel projects, enabling deployment events to trigger workflows and giving AI agents access to build logs, deployment status, and rollback capabilities. ## Prerequisites * **Organization Admin** role in Kestrel * Vercel account with project access ## Setup 1. Navigate to **Integrations → Vercel** in your Kestrel dashboard 2. Click **Connect** 3. Authorize Kestrel in the Vercel OAuth popup 4. Select the Vercel team/account and projects to authorize 5. The popup closes and your Vercel account appears as connected After OAuth connection, Kestrel automatically registers a webhook with your Vercel account to receive deployment and error events. No manual webhook configuration is required. ## How It's Used ### In Workflows **Trigger blocks:** * **Deployment Failed** — fires when a Vercel deployment build fails * **Deployment Succeeded** — fires when a deployment completes successfully * **Deployment Created** — fires when a new Vercel deployment is initiated * **Error Anomaly** — fires when Vercel detects an anomalous spike in runtime errors * **Usage Anomaly** — fires when Vercel detects unusual usage patterns * **Domain Issue** — fires on domain configuration or certificate problems * **Firewall Attack Detected** — fires when Vercel's firewall detects a potential attack * **Deployment Checks Failed** — fires when deployment integration checks fail * **Production Rollback** — fires when a production deployment is rolled back **Action blocks:** * **Get Deployment** — retrieve deployment details including status, URL, and git commit * **Get Build Logs** — fetch build log output for a specific deployment * **Rollback Production** — roll back production to a previous stable deployment * **Promote to Production** — promote a preview deployment to production * **List Deployments** — list recent deployments with optional filters * **Investigate Vercel** — run an AI investigation combining build logs, error data, and deployment history Example: A workflow triggers on a failed deployment, fetches the build logs, analyzes the failure with AI, and posts a summary to Slack with a one-click rollback button. ## Disconnecting 1. Navigate to **Integrations → Vercel** 2. Click **Disconnect** 3. Confirm the disconnection This removes the webhook and stops all Vercel workflow triggers. You can reconnect at any time. # Introduction Source: https://docs.usekestrel.ai/introduction Kestrel documentation - AI Agents for Platform Engineering Kestrel is the agentic platform for platform engineering. Build workflows that automate incident response, cloud provisioning, CI/CD pipelines, and developer requests — using natural language, a drag-and-drop canvas, CLI, SDK, or MCP. Connect 30+ integrations, describe what you want, and let AI agents build the automation for you. ## What Kestrel Does Build workflows with natural language or a drag-and-drop canvas that automate incident response, cloud resource provisioning, CI/CD pipelines, and on-demand developer requests. Chain actions across Kubernetes, AWS, Slack, GitHub, PagerDuty, Datadog, Vercel, and more. Trigger from K8s signals, cloud events, Slack requests, custom webhooks, or any integration — with built-in observability. AI agents that monitor 24/7 for Kubernetes and cloud incidents. Automatic root cause analysis; YAML, Terraform, and code fix generation, causal PR detection, and postmortem creation — with optional auto-remediation. Ask questions about your infrastructure and applications in plain English. Investigate K8s clusters, cloud resources, metrics, network traffic, costs and more through a chat interface or the Slack `/kestrel` command. ## How It Works Link Kubernetes clusters, cloud accounts, PaaS and deployment platforms, databases, CI/CD pipelines, observability tools, notification channels, code repositories, and knowledge sources from the Kestrel platform. Each integration takes less than two minutes to setup. Once connected, Kestrel automatically starts monitoring for Kubernetes and cloud incidents across both infrastructure and application layers. The Cloud AI Copilot also becomes available immediately — ask any question about your infrastructure through the dashboard chat or the Slack `/kestrel` command. Describe what you want in natural language to the Workflow Agent — or drag and drop actions on the canvas, configuring triggers and actions using template variables. Kestrel's AI assembles and validates the workflow for you. Workflows run automatically when triggered. AI agents execute each step, route notifications, generate fixes, create tickets, and escalate when needed — with full audit logs and observability. ## Integrations Kestrel connects to your clouds, clusters, PaaS and deployment platforms, AI compute and sandbox providers, databases, CI/CD pipelines, IaC platforms, observability tools, notification channels, and knowledge sources. ### Clouds & Infrastructure | Integration | Purpose | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Kubernetes](/integrations/kubernetes) | Cluster monitoring, incident detection, resource inventory. **Workflows:** trigger on pod crashes, node failures, rollout issues; actions include RCA, fix generation, manifest creation, kubectl investigations | | [Karpenter](/integrations/karpenter) | Node autoscaling. **Workflows:** trigger on provisioning failures, node interruptions, NodePool limit exhaustion; actions include scaling NodePool limits, tuning disruption policy, applying NodePools, recycling nodes | | [AWS](/integrations/aws) | Cloud resource inventory, incident detection, VPC Flow Logs. **Workflows:** trigger on IAM, S3, EC2, RDS, Lambda events; actions include cloud RCA, Terraform/CLI generation, cost queries, anomaly detection, forecasting | | [OCI](/integrations/oci) | Oracle Cloud resource inventory, incident detection, VCN Flow Logs | | [Cloudflare](/integrations/cloudflare) | Edge, DNS, and security automation. **Workflows:** trigger on security events, health check failures, Worker failures, SSL expiration; actions include DNS records, WAF/firewall rules, cache purges, Worker deploys/rollbacks, load balancer pools, tunnels | ### PaaS & Deployment Platforms | Integration | Purpose | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Vercel](/integrations/vercel) | Deployment events and build management. **Workflows:** trigger on deployment failures/successes, error anomalies; actions include getting deployments, build logs, rollbacks, promotions, AI investigation | | [Railway](/integrations/railway) | Deployment automation. **Workflows:** trigger on deployment events; actions include deployment logs, redeploys, restarts, rollbacks, environment variables, AI investigation | | [Fly.io](/integrations/flyio) | Machine lifecycle management. **Workflows:** trigger on machine start/stop/crash/create/destroy; actions include machine start/stop/restart, secrets, AI investigation | ### AI Compute & Sandboxes | Integration | Purpose | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Nebius](/integrations/nebius) | AI cloud GPU infrastructure. **Workflows:** trigger on GPU/node lifecycle events; actions include instance start/stop/restart, managed Kubernetes node-group scaling, AI investigation | | [Beam](/integrations/beam) | Serverless AI compute. **Workflows:** trigger on task, container, deployment, and machine events; actions include deployment scaling/start/stop, sandbox lifecycle, running commands, AI investigation | | [Daytona](/integrations/daytona) | Dev sandboxes. **Workflows:** trigger on sandbox, snapshot, and volume lifecycle events; actions include sandbox management, command execution, file transfer, AI investigation | ### Databases | Integration | Purpose | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Supabase](/integrations/supabase) | Project and branch automation. **Workflows:** actions include project/branch/backup/read-replica lifecycle, network restrictions, AI investigation | | [Neon](/integrations/neon) | Serverless Postgres. **Workflows:** actions include branch lifecycle, point-in-time restore, compute suspend/resume and autoscaling, reset-from-parent, credential rotation, AI investigation | | [PlanetScale](/integrations/planetscale) | MySQL branching and schema migrations. **Workflows:** trigger on branch/deploy-request events; actions include branch, deploy-request (schema migration), backup, and credential automation | | [ClickHouse](/integrations/clickhouse) | ClickHouse Cloud services. **Workflows:** trigger on error spikes; actions include service start/stop/scale, IP access lists, backups, ClickPipes ingestion, usage costs, AI investigation | ### CI/CD & GitOps | Integration | Purpose | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [GitHub](/integrations/github) | GitOps deployments, causal PR detection, IaC remediation. **Workflows:** actions include creating PRs/issues, triggering Actions, waiting for runs, AI code investigation, AI code fix generation | | [GitLab](/integrations/gitlab) | GitOps deployments, merge request remediation. **Workflows:** actions include creating MRs/issues, triggering pipelines, waiting for pipeline completion | | [Jenkins](/integrations/jenkins) | Build automation. **Workflows:** trigger on build failures; actions include triggering builds (with parameters), monitoring builds, console logs, stopping runaway builds, AI investigation | | [CircleCI](/integrations/circleci) | Pipeline automation. **Workflows:** trigger on pipeline/workflow failures; actions include triggering pipelines, rerunning/canceling workflows, approving on-hold jobs, test results, AI investigation | | [ArgoCD](/integrations/argocd) | GitOps deployment management. **Workflows:** actions include triggering syncs, waiting for sync completion, checking application health status | | [Argo Rollouts](/integrations/argo-rollouts) | Progressive delivery. **Workflows:** actions include promoting, aborting, retrying, and rolling back canary and blue-green rollouts | | [Flux CD](/integrations/fluxcd) | GitOps reconciliation. **Workflows:** actions include reconciling, suspending, and resuming Kustomizations, HelmReleases, and sources | | [Helm](/integrations/helm) | Release management. **Workflows:** actions include installing, upgrading, rolling back, and uninstalling Helm releases | ### Infrastructure as Code | Integration | Purpose | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Terraform Cloud](/integrations/terraform) | Run lifecycle automation. **Workflows:** trigger on run events; actions include plan/apply/discard/cancel, workspace locking, variables, state outputs, drift detection, AI investigation | | [Pulumi Cloud](/integrations/pulumi) | Stack automation. **Workflows:** trigger on stack/deployment events; actions include stack updates, drift detection and remediation, deployment queue control, stack tags and outputs, AI investigation | ### Secret Management | Integration | Purpose | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [HashiCorp Vault](/integrations/vault) | Secret lifecycle automation. **Workflows:** trigger on seal-status, secret-version, stale-secret, policy, and auth-method changes (poll-based, metadata only); actions include reading/writing KV secrets, rotating static roles, managing policies and leases, revoking tokens, AI investigation | | [Infisical](/integrations/infisical) | Secret management automation. **Workflows:** trigger on secret changes, approval requests, failed syncs, and new identities (poll-based via audit log); actions include managing secrets and folders, triggering syncs, reviewing approvals, querying audit logs, AI investigation | ### Platform Security | Integration | Purpose | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [SonarCloud](/integrations/sonarcloud) | Code quality and security automation. **Workflows:** trigger on completed/failed analyses with quality gate results (webhook-delivered); actions include quality gate checks, listing issues and security hotspots, project metrics, and approval-gated triage (transition/assign/comment issues, review hotspots) | | [Okta](/integrations/okta) | Identity security automation. **Workflows:** trigger on lockouts, suspicious activity, admin grants, MFA changes, and group membership changes (poll-based via the System Log); actions include user lookups, System Log queries, suspensions, session revocation, password expiry, MFA resets, and group changes | | [Kyverno](/integrations/kyverno) | Policy-as-code enforcement. **Workflows:** trigger on policy violations and admission blocks (auto-detected in-cluster, streamed from PolicyReports); actions include listing policies and violations, switching policies between Audit and Enforce, applying and deleting policies | ### Observability & Analytics | Integration | Purpose | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Datadog](/integrations/datadog) | Metrics, monitors, and observability context. **Workflows:** actions include querying metrics, creating monitors, sending events, muting monitors | | [OpenTelemetry](/integrations/opentelemetry) | Metrics for incident root cause analysis and copilot context | | [PostHog](/integrations/posthog) | Product analytics events and session replay. **Workflows:** trigger on exceptions, rage clicks, console errors, log alerts; actions include session summaries, event queries, error issue details | ### Incident Management & Ticketing | Integration | Purpose | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [PagerDuty](/integrations/pagerduty) | On-call alerting with RCA context and auto-resolution. **Workflows:** trigger on PD incidents; actions include creating/acknowledging/resolving alerts, adding notes, escalation | | [Jira](/integrations/jira) | Ticket creation and incident tracking. **Workflows:** actions include creating tickets, adding comments, transitioning ticket status | | [Linear](/integrations/linear) | Issue tracking. **Workflows:** actions include creating issues. Also used as a knowledge source | ### Notifications & Knowledge | Integration | Purpose | | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Slack](/integrations/slack) | Notifications, interactive Q\&A, `/kestrel` copilot, `/kestrel-workflow` triggers. **Workflows:** trigger from Slack requests; actions include sending messages, threaded replies, approval requests, DMs | | [Confluence](/integrations/confluence) | RCA publishing, postmortems, runbooks. **Workflows:** actions include publishing RCA pages, postmortems, runbook entries, updating existing pages. Also used as a knowledge source during incident investigation | | [Glean](/integrations/glean) | Enterprise knowledge search during incident investigation | | [Knowledge Sources](/integrations/knowledge-sources) | Connect Slack, Confluence, Jira, Linear, Notion, and Glean as knowledge sources so incident investigations include similar past incidents, runbooks, and postmortems | ## Enterprise Security Built for organizations that demand the highest standards of data privacy, security, and compliance. * **SOC 2 Compliant** — Enterprise-grade security controls audited by an independent third party * **Complete Data Isolation** — Your data is never combined with other customers * **No Cross-Customer Training** — AI models are never trained on another customer's data * **Read-Only by Default** — No write access unless you explicitly enable auto-remediation * **Full Control Over Fixes** — Every AI-generated change requires your explicit approval * **No Sensitive Data Storage** — Secrets are never sent to or stored on Kestrel servers ## Getting Started Build your first workflow automation in seconds. Set up agentic incident detection and response. Start asking questions about your infrastructure and applications. # On-Premise Overview Source: https://docs.usekestrel.ai/on-premise/overview Deploy Kestrel in your own cloud environment with complete control and data sovereignty For organizations with strict data residency requirements, regulatory compliance needs, or air-gapped environments, Kestrel supports full self-hosted deployment in your own cloud infrastructure. ## Deployment Models ### Standard Deployment Your Kubernetes cluster has internet access. Pull images directly from our container registry, access LLM services via regional endpoints, and integrate with GitHub, Slack, and PagerDuty via public APIs. This is the simplest deployment model for organizations without strict network isolation requirements. * All data stays within your environment * Uses cloud-native LLM services (Amazon Bedrock, Vertex AI, Azure OpenAI, OCI Generative AI) * Container images pulled from Kestrel's private registry * Integrates with your existing CI/CD via GitHub.com or GitLab.com ### Air-Gapped Deployment No internet access required. All external services are accessed via VPC/Private endpoints - no public internet needed. Integrate with GitHub Enterprise Server for IaC workflows. Used by customers in defense, financial services, and other highly regulated industries. * **No internet access required** * LLM services accessed via private endpoints (AWS VPC Endpoints for Bedrock, GCP Private Service Connect for Vertex AI, Azure Private Endpoints for Azure OpenAI, OCI Service Gateway for OCI Generative AI) * Container images pulled from your private registry (ECR, Artifact Registry, ACR, OCI Container Registry) * Secrets stored in AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or OCI Secret Management * Integrates with GitHub Enterprise Server for IaC workflows ## Supported Cloud Providers | Provider | Standard | Air-Gapped | Native Services | | --------- | -------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | | **AWS** | ✅ | ✅ | EKS, ECR, Amazon Bedrock, Secrets Manager, RDS, ElastiCache | | **GCP** | ✅ | ✅ | GKE, Artifact Registry, Vertex AI, Secret Manager, Cloud SQL, Memorystore | | **Azure** | ✅ | ✅ | AKS, ACR, Azure OpenAI, Key Vault, Azure Database, Azure Cache | | **OCI** | ✅ | ✅ | OKE, OCI Container Registry, OCI Generative AI, OCI Secret Management, OCI Managed Relational Databases, OCI Cache with Redis | ## Architecture The on-premise deployment includes the following components: * **Kestrel Server** - Core API server and incident engine * **Kestrel Frontend** - Web dashboard * **Kestrel Router** - WebSocket and gRPC routing layer * **Kestrel Agents** - AI agent workers for incident analysis * **PostgreSQL** - Primary database (can use managed RDS/Cloud SQL/Azure Database/OCI PostgreSQL DB) * **Redis** - Caching and message queue (can use managed ElastiCache/Memorystore/Azure Cache/OCI Cache with Redis) * **Elasticsearch/OpenSearch** (Optional) - Enhanced search and log analysis All components are deployed via a single unified Helm chart in your Kubernetes cluster. ## Native Workload Identity Kestrel integrates with each cloud provider's native workload identity system for secure, keyless authentication: | Provider | Workload Identity | | --------- | ------------------------------------------------------------------------- | | **AWS** | IRSA (IAM Roles for Service Accounts), EKS Pod Identity, or Node IAM Role | | **GCP** | Workload Identity Federation for GKE service accounts | | **Azure** | Azure AD Workload Identity or Managed Identity for AKS pods | | **OCI** | OKE Workload Identity or Instance Principal authentication | ## Prerequisites * Kubernetes cluster (v1.24+) in your cloud environment * `helm` (v3.0+) installed * Container registry access (ECR, Artifact Registry, ACR, or OCI Container Registry) * LLM provider configured (Bedrock, Vertex AI, Azure OpenAI, OCI Generative AI, or OpenAI) * Secrets management service (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or OCI Secret Management) * IAM role, service account, managed identity, or workload identity with required permissions * On-Premise license enabled for your Kestrel organization ## Next Steps * [Follow the setup guide](/on-premise/setup) to deploy Kestrel in your environment # On-Premise Setup Guide Source: https://docs.usekestrel.ai/on-premise/setup Step-by-step guide to deploying Kestrel in your own cloud environment This guide walks you through the Kestrel on-premise deployment wizard, which generates a complete Helm values file for your environment. ## Accessing the Setup Wizard 1. Log in to your Kestrel dashboard at [platform.usekestrel.ai](https://platform.usekestrel.ai) 2. Navigate to **Integrations → On-Premise Deployment** 3. The guided setup wizard walks you through each configuration step On-premise deployment requires a license. If you don't see the On-Premise option, contact your Kestrel account manager. ## Step 1: Choose Your Cloud Provider Select your target cloud provider: * **Amazon Web Services (AWS)** * **Google Cloud Platform (GCP)** * **Microsoft Azure** * **Oracle Cloud Infrastructure (OCI)** This selection determines the available LLM services, secrets providers, container registry, and managed database options. ## Step 2: Select Deployment Type Choose your deployment model: | Type | Description | | -------------- | --------------------------------------------------------------------------------------------------------- | | **Standard** | Outbound internet access available. Uses cloud LLM APIs directly. | | **Air-Gapped** | No internet access. All services accessed via VPC/Private endpoints. Requires private container registry. | ## Step 3: Configure LLM Select and configure your LLM provider: ### Amazon Bedrock (AWS) * Select a model (e.g., Claude Sonnet 4.5) * Set the Bedrock region * Optionally provide a custom endpoint URL (for VPC endpoints in air-gapped mode) ### Vertex AI (GCP) * Configure Vertex AI region and model settings * For air-gapped mode, use Private Service Connect ### Azure OpenAI * Configure Azure OpenAI resource and deployment settings * For air-gapped mode, use Azure Private Endpoints ### OCI Generative AI * Configure OCI Generative AI service settings * For air-gapped mode, use OCI Service Gateway ### OpenAI * Enter your OpenAI API key * Select the model (e.g., GPT-5.2) ## Step 4: Configure Infrastructure ### Container Registry Enter your private container registry URL where Kestrel images will be stored: * **AWS**: `123456789012.dkr.ecr.us-east-1.amazonaws.com` * **GCP**: `us-docker.pkg.dev/project-id/repo` * **Azure**: `kestrel.azurecr.io` * **OCI**: `.ocir.io//kestrel` ### IAM / Workload Identity Provide the identity that Kestrel will use for cloud service access: * **AWS**: IAM Role ARN (IRSA, EKS Pod Identity, or Node IAM Role) * **GCP**: Service Account (Workload Identity Federation) * **Azure**: Managed Identity (Azure AD Workload Identity) * **OCI**: OKE Workload Identity or Instance Principal ### Database Choose between: * **Bundled PostgreSQL** - Included in the Helm deployment (suitable for dev/test) * **External PostgreSQL** - Use a managed database: * AWS: RDS * GCP: Cloud SQL * Azure: Azure Database * OCI: OCI Managed Relational Databases ### Redis Choose between bundled or external managed Redis: * AWS: ElastiCache * GCP: Memorystore * Azure: Azure Cache * OCI: OCI Cache with Redis ### Elasticsearch/OpenSearch (Optional) Enable for enhanced search and log analysis: * Endpoint URL * Username and password ### Ingress * **Ingress Class Name**: Your ingress controller class (default: `nginx`) * **Ingress Host**: The domain name for your Kestrel installation ### Secrets Provider Select where credentials will be stored: * AWS Secrets Manager * GCP Secret Manager * Azure Key Vault * OCI Secret Management ## Step 5: GitHub Integration (Air-Gapped Only) For air-gapped deployments, configure GitHub Enterprise Server for IaC workflows: 1. Enter your **GitHub Enterprise Server URL** (e.g., `https://github.internal.company.com`) 2. Create a GitHub App on your GHE instance: * Go to **Settings → Developer Settings → GitHub Apps → New GitHub App** * Set the webhook URL to your Kestrel ingress host * Grant required permissions (repository contents: read/write, pull requests: read/write) 3. Enter the **GitHub App ID** and **App Slug** 4. Store credentials securely in AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or OCI Secret Management Kestrel generates a TLS CA certificate for secure communication with GitHub Enterprise Server. Download it from the setup wizard and configure it in your GHE instance. ## Step 6: Generate Configuration Click **Generate Helm Values** to produce a complete `values.yaml` file customized for your environment. The generated configuration includes: * All service configurations * Database connection strings * LLM provider settings * Registry and image references * Ingress configuration * Secrets provider settings * Workload identity bindings ## Step 7: Download and Deploy ### Pull Container Images Click **Get Registry Credentials** to obtain credentials for pulling Kestrel images from the private registry. ```bash theme={null} # Log in to the Kestrel registry docker login ghcr.io -u -p # Pull and retag images for your private registry docker pull ghcr.io/kestrelai/kestrel-server:latest docker tag ghcr.io/kestrelai/kestrel-server:latest /kestrel-server:latest docker push /kestrel-server:latest # Repeat for all Kestrel images ``` For air-gapped environments, download both the Helm chart and values file, then install from local files. ### Create Secrets Create the necessary Kubernetes secrets: ```bash theme={null} # Create the namespace kubectl create namespace kestrel-ai # Create registry credentials (if using private registry) kubectl create secret docker-registry kestrel-registry \ --namespace kestrel-ai \ --docker-server= \ --docker-username= \ --docker-password= ``` ### Install with Helm ```bash theme={null} helm install kestrel \ oci://ghcr.io/kestrelai/charts/kestrel \ --version 1.0.0 \ --namespace kestrel-ai \ --create-namespace \ -f kestrel-values.yaml ``` ### Verify the Deployment ```bash theme={null} # Check all pods are running kubectl get pods -n kestrel-ai # Check the ingress kubectl get ingress -n kestrel-ai ``` Navigate to your configured domain to access the Kestrel dashboard. ## Air-Gapped Private Endpoints For air-gapped deployments, Kestrel guides you through creating the required private endpoints for your cloud provider: ### LLM Service Access AWS VPC Endpoints for Bedrock, GCP Private Service Connect for Vertex AI, Azure Private Endpoints for Azure OpenAI, or OCI Service Gateway for OCI Generative AI. ### Container Registry Private access to ECR (AWS), Artifact Registry (GCP), Azure Container Registry, or OCI Container Registry without traversing the public internet. ### Secrets Management Private endpoints for AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or OCI Secret Management to retrieve credentials at runtime. The wizard provides cloud-specific, step-by-step instructions for creating each endpoint, including security group configuration and private DNS settings. ## Updating To update your on-premise deployment: 1. Return to the setup wizard and click **Generate Helm Values** to get the latest configuration 2. Pull the newest container images 3. Run Helm upgrade: ```bash theme={null} helm upgrade kestrel \ oci://ghcr.io/kestrelai/charts/kestrel \ --namespace kestrel-ai \ -f kestrel-values.yaml ``` ## Next Steps * [Connect your Kubernetes clusters](/kubernetes/onboarding) to the on-premise Kestrel instance * [Set up cloud integrations](/cloud/overview) for cloud resource monitoring # Cloud AI Copilot Quickstart Source: https://docs.usekestrel.ai/quickstart/copilot Ask questions about your infrastructure in plain English The Cloud AI Copilot lets you investigate your infrastructure by asking questions in plain English. Query Kubernetes clusters, cloud resources, metrics, costs, and code — from the Kestrel dashboard or directly in Slack with the `/kestrel` command. No query syntax, no CLI flags, no context-switching between dashboards. ## What the Copilot Can Do * **Investigate Kubernetes** — pod status, crash reasons, resource usage, network policies, recent changes * **Query cloud resources** — AWS EC2, S3, IAM, VPC, and more * **Analyze metrics** — pull Datadog or OpenTelemetry data into the conversation * **Analyze costs** — identify expensive resources, cost trends, and optimization opportunities * **Investigate code** — trace infrastructure changes back to PRs and commits * **Generate config changes** — produce YAML manifests, Terraform plans, and Helm values on demand ## Prerequisites The copilot answers are only as good as the integrations connected. At minimum: * A Kestrel account at [platform.usekestrel.ai](https://platform.usekestrel.ai) * At least one data source connected: * [Kubernetes cluster](/integrations/kubernetes) for cluster investigations * [AWS](/integrations/aws) or [OCI](/integrations/oci) for cloud resource queries * [Datadog](/integrations/datadog) or [OpenTelemetry](/integrations/opentelemetry) for metrics questions * [GitHub](/integrations/github) or [GitLab](/integrations/gitlab) for code investigations The more integrations you connect, the more context the copilot has. Connect your notification and knowledge source integrations too — Slack history, Confluence, and Jira give the copilot organizational context that improves answers. ## Using the Dashboard Chat 1. Open the Kestrel dashboard at [platform.usekestrel.ai](https://platform.usekestrel.ai) 2. Navigate to **Cloud AI Copilot → Chat** in the sidebar 3. Type your question and press Enter Use `@cluster-name` to target specific Kubernetes clusters and `@cloud:connection-name` to target specific AWS or OCI accounts. You can mention multiple targets in a single query. The copilot maintains conversation context across messages, so you can ask follow-up questions naturally. **Example conversation:** > **You:** `@prod-eks-cluster` Why are pods in the payments namespace restarting? Cross-reference with Datadog APM latency metrics for the payments-api service over the last 2 hours. > > **Kestrel:** There are 2 pods in CrashLoopBackOff in the `payments` namespace: `payments-api-7f8b9c-x4k2m` and `payments-worker-6d4a8b-j9n3p`. Both started crashing 23 minutes ago after deployment `payments-api` was updated to image tag `v2.14.0`. Datadog shows P99 latency for `payments-api` spiked from 120ms to 4.2s at 14:32 UTC, coinciding with the deployment... > > **You:** What PR caused that deployment change, and who approved it? > > **Kestrel:** The deployment was updated via ArgoCD sync at 14:32 UTC. The change originated from PR #847 in `acme/payments-service`, authored by @jane.doe and approved by @john.smith. The PR modified the database connection pool settings, reducing `maxConnections` from 50 to 10... ## Using via Slack Once the [Slack integration](/integrations/slack) is connected, use the `/kestrel` slash command in any channel or DM: ``` /kestrel @prod-eks-cluster @staging-eks-cluster compare the resource limits for the auth-service deployment across both clusters — are there any discrepancies in CPU or memory requests? ``` ``` /kestrel @cloud:prod-aws-380983552929 which EC2 instances in us-east-1 and eu-west-1 have been running for more than 90 days without being part of an auto-scaling group? Include estimated monthly cost for each. ``` ``` /kestrel @prod-eks-cluster show me all pods that have been OOMKilled in the last 48 hours, grouped by deployment, with their current memory limits vs actual peak usage from Datadog metrics ``` ``` /kestrel @cloud:prod-aws-380983552929 @cloud:staging-aws-131131252671 compare the IAM role policies attached to the EKS node groups across both accounts — flag any permissions that exist in staging but not in production ``` Kestrel responds in-thread so conversations stay organized. Other team members can see the answers and ask follow-ups in the same thread. ## Example Questions ### Kubernetes Investigations * `@prod-eks-cluster` "Why is the auth-service deployment failing readiness probes? Check the container logs, recent config changes, and any related network policy updates in the last 6 hours" * `@prod-eks-cluster` "Show me all pods consuming more than 80% of their memory limit across the payments and checkout namespaces, sorted by how close they are to OOMKill" * `@staging-gke-cluster` "What Helm releases were upgraded in the last 24 hours and did any of them cause pod restart spikes?" * `@prod-eks-cluster` "Trace the full request path from the frontend ingress to the database for the order-service — which services are involved and what network policies apply at each hop?" ### Cloud Resource Queries * `@cloud:prod-aws-380983552929` "Which security groups in the production VPC have inbound rules allowing 0.0.0.0/0 on ports other than 80 and 443?" * `@cloud:prod-aws-380983552929` "Show me all RDS instances that don't have automated backups enabled, multi-AZ deployment, or encryption at rest" * `@cloud:prod-aws-380983552929` @cloud:staging-aws-131131252671\` "Compare the Lambda function configurations between prod and staging for the payment-processor functions — are there memory, timeout, or environment variable differences?" ### Cost Analysis * `@cloud:prod-aws-380983552929` "Break down our top 10 most expensive services this month compared to last month, and flag any that increased by more than 20%" * `@cloud:prod-aws-380983552929` "Estimate how much we'd save if we right-sized all EC2 instances in us-east-1 that have averaged less than 15% CPU utilization over the last 30 days" * `@cloud:prod-aws-380983552929` "What's our projected EKS spend for the next quarter based on current growth trends? Break it down by cluster." ### Code Investigations * "What PRs were merged to the main branch of acme/infrastructure-terraform in the last 48 hours that touched any IAM policy or security group definitions?" * "Show me the git blame for the Kubernetes resource limits in acme/payments-service — when were they last changed and what was the reasoning in the PR description?" * "Search across all connected repos for any Terraform code that references the prod-vpc VPC ID — I need to understand all resources deployed into it" ## Configuration Changes When you ask the copilot to make a change — for example, "scale the checkout deployment to 5 replicas" or "add a network policy to block egress from the sandbox namespace" — it generates the configuration change and presents it for your review. You can: * **Apply directly** to the cluster from the dashboard * **Create a PR** to your GitOps repository via GitHub or GitLab * **Copy the manifest** to apply manually The copilot never applies changes without your explicit approval. All generated configurations are presented for review first. ## Enhancing Copilot Context Connect additional integrations to improve the quality and depth of copilot answers: | Integration | What It Adds | | --------------------------------------------------------------- | ----------------------------------------- | | [Datadog](/integrations/datadog) | Metrics, APM traces, and monitor status | | [OpenTelemetry](/integrations/opentelemetry) | Traces and custom metrics | | [Confluence](/integrations/confluence) | Runbooks and architecture documentation | | [Jira](/integrations/jira) | Related tickets and known issues | | [Glean](/integrations/glean) | Cross-tool enterprise knowledge search | | [GitHub](/integrations/github) / [GitLab](/integrations/gitlab) | Code context, PR history, IaC definitions | # Incident Response Quickstart Source: https://docs.usekestrel.ai/quickstart/incident-response Set up AI-powered incident detection and response Kestrel monitors your Kubernetes clusters and cloud infrastructure 24/7, detecting incidents as they happen. When something goes wrong, AI agents automatically investigate the root cause, generate production-ready YAML or Terraform fixes, identify the causal PR, and create postmortems — with optional auto-remediation. ## What You Get * **24/7 Monitoring** — continuous detection across K8s and cloud resources * **Automatic Root Cause Analysis** — AI traces the full causal chain, not just symptoms * **Production-Ready Fixes** — YAML manifests and Terraform changes you can apply directly or merge as PRs * **Causal PR Detection** — identifies the code change that introduced the issue * **Postmortem Generation** — structured incident summaries for your team * **Auto-Remediation** — optionally apply fixes automatically (disabled by default) ## Setup Kestrel needs visibility into your infrastructure to detect incidents. **For Kubernetes:** Install the Kestrel Operator via Helm. See the [Kubernetes integration guide](/integrations/kubernetes) for full instructions. ```bash theme={null} helm install kestrel-operator \ oci://ghcr.io/kestrelai/charts/kestrel-operator \ --version 1.0.0 \ --namespace kestrel-ai --create-namespace \ -f kestrel-ai-operator-values.yaml ``` **For AWS:** Connect your account with a read-only IAM role. See the [AWS integration guide](/integrations/aws). **For OCI:** Connect your Oracle Cloud tenancy. See the [OCI integration guide](/integrations/oci). You can connect multiple clusters and cloud accounts. Kestrel monitors all of them from a single dashboard. Adding observability context improves root cause analysis by giving Kestrel access to metrics, traces, and monitor alerts. * **[Datadog](/integrations/datadog)** — pull metrics, monitors, and APM traces into incident investigations * **[OpenTelemetry](/integrations/opentelemetry)** — ingest traces and metrics from your OTel pipeline Observability integrations are optional. Kestrel performs RCA using Kubernetes and cloud signals alone, but metrics context produces more precise root causes. Route incident alerts to the tools your team already uses. * **[Slack](/integrations/slack)** — receive incident notifications in a channel, ask follow-up questions in threads, and get @mentioned as a namespace owner * **[PagerDuty](/integrations/pagerduty)** — trigger on-call alerts with full RCA context and automatic resolution when the incident clears Linking your repositories enables causal PR detection and GitOps-based remediation. * **[GitHub](/integrations/github)** — detect which PR introduced the issue; create fix PRs automatically * **[GitLab](/integrations/gitlab)** — same capabilities via merge requests When Kestrel generates a fix, it can open a PR against your repository so the change goes through your existing review and deployment pipeline. Knowledge sources give Kestrel historical and organizational context to produce better root cause analysis and more relevant fixes. * **[Confluence](/integrations/confluence)** — runbooks, architecture docs, past incident writeups * **[Jira](/integrations/jira)** — related tickets and known issues * **[Glean](/integrations/glean)** — enterprise knowledge search across all your tools * **[Slack history](/integrations/slack)** — past conversations and tribal knowledge See [Knowledge Sources](/integrations/knowledge-sources) for setup details. ## Your First Incident Once your integrations are connected, Kestrel begins monitoring immediately. Here's what happens when an incident is detected: 1. **Detection** — Kestrel identifies an anomaly (pod crash, deployment failure, cloud resource issue, etc.) 2. **Investigation** — AI agents gather context from your cluster, cloud APIs, metrics, logs, and knowledge sources 3. **Root Cause Analysis** — a structured RCA is generated with the full causal chain 4. **Fix Generation** — production-ready YAML or Terraform changes are proposed 5. **Causal PR** — if a recent code change caused the issue, the specific PR is identified 6. **Notification** — alerts are sent to Slack, PagerDuty, or both — with the RCA, fix, and causal PR attached 7. **Remediation** — apply the fix from the dashboard, merge the generated PR, or let auto-remediation handle it Auto-remediation is disabled by default. Enable it per-cluster from **Settings → Incident Response** only after you've reviewed and trusted Kestrel's fixes in your environment. ## Build Custom Incident Response Workflows For more complex incident response scenarios — escalation chains, multi-step remediation, cross-team coordination — use [Workflows](/quickstart/workflows) to build custom automation that triggers on Kestrel incidents. **Example:** When a critical incident is detected in production, run RCA, post to Slack, page the on-call engineer via PagerDuty, create a Jira ticket, and if unacknowledged after 15 minutes, escalate to the engineering manager. # Workflows Quickstart Source: https://docs.usekestrel.ai/quickstart/workflows Build your first workflow automation in seconds Kestrel Workflows let you automate incident response, cloud resource provisioning, CI/CD pipelines, developer requests, and more — with AI-powered workflows that build themselves. Chain actions across 15+ integrations, triggered by Kubernetes signals, AWS cloud signals, Slack `/kestrel-workflow` requests, PagerDuty incidents, PostHog events, Vercel deployment events, custom webhooks, or on-demand runs. Build workflows with the Workflow Agent using natural language, a drag-and-drop canvas, the CLI, SDK, or MCP. ## Prerequisites * A Kestrel account at [platform.usekestrel.ai](https://platform.usekestrel.ai) * At least one integration connected (see [Integrations](/integrations/kubernetes)) ## Create Your First Workflow From the Kestrel dashboard, go to **Workflows → New Workflow**. You can start from a blank canvas or choose a template. Templates cover common use cases like incident response, cloud provisioning, and CI/CD automation. Type a plain-English description of what you want automated. The **Workflow Agent** generates the workflow steps for you. **Example:** > When a pod crashes in the payments namespace, trigger root cause analysis, send a Slack message with the root cause to #incidents, and create a Jira ticket assigned to the payments team. The Workflow Agent assembles the trigger, actions, and connections. Review the generated workflow on the canvas and adjust as needed. You can also build workflows entirely with the drag-and-drop canvas — add trigger and action nodes, connect them, and configure each step manually. Every workflow starts with a trigger. Select the event source that kicks off the workflow: | Trigger Type | Trigger Blocks | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Kubernetes Signals** | Deployment Replicas Failing, Nodes Unavailable, Pod CrashLoopBackOff, Pod ImagePullBackOff, Pod OOMKilled, Pods Failing, Pods Restarting, StatefulSet Replicas Failing, Node Memory Pressure, Node Disk Pressure, DaemonSet Failing, Any Kubernetes Incident | | **AWS Cloud Signals** | IAM Security Event, Root Account Activity, KMS Key Change, Secrets Manager Event, Security Hub Finding, S3 Bucket Change, EC2 Instance Issue, Lambda Function Issue, ECS/EKS Container Issue, RDS Database Issue, DynamoDB Issue, VPC/Network Change, CloudWatch Alarm, Application Log Errors, Config Rule Non-Compliant, AWS Service Health Event, Any AWS Cloud Incident, Cost Anomaly Detected, Budget Threshold Exceeded | | **Slack Requests** | Create K8s Resource, Edit K8s Resource, General Kubernetes Request, Create AWS Resource, Edit AWS Resource, General Cloud Request, Any Slack `/kestrel-workflow` Request | | **PagerDuty** | Incident Triggered, Incident Acknowledged, Incident Resolved, Any PagerDuty Incident, High Urgency Incident | | **PostHog** | Session Error/Exception, Console Error, Rage Click, Any PostHog Event, Log Error Alert | | **Vercel** | Deployment Failed, Deployment Succeeded, Deployment Created, Error Anomaly, Usage Anomaly, Domain Issue, Firewall Attack Detected, Deployment Checks Failed, Production Rollback | | **Jenkins** | Build Failed, Build Unstable, Build Succeeded, Build Completed, Build Started | | **CircleCI** | Workflow Failed, Workflow Succeeded, Workflow Completed, Job Failed | | **Terraform Cloud** | Run Created, Run Planning, Run Needs Attention, Run Applying, Run Completed, Run Errored, Drift Detected, Assessment Check Failed | | **Pulumi Cloud** | Update Succeeded, Update Failed, Preview Failed, Destroy Succeeded, Deployment Started, Deployment Succeeded, Deployment Failed, Drift Detected, Drift Run Failed, Policy Violation, Stack Created, Stack Deleted | | **HashiCorp Vault** | Vault Sealed, Vault Unsealed, Health Degraded, Secret Version Created, Secret Stale, Policy Created/Deleted, Auth Method Enabled/Disabled | | **Infisical** | Secret Created, Secret Updated, Secret Deleted, Approval Requested, Secret Sync Failed, Identity Created | | **Custom Webhook** | Any HTTP POST to your workflow's unique webhook endpoint with HMAC signature verification | Configure trigger-specific filters to control when the workflow fires — for example, limit a K8s trigger to a specific namespace, cluster, or set of AWS accounts. Add action steps to your workflow. Each action targets an integration and performs a specific operation — sending a Slack message, creating a Jira ticket, running an RCA, applying a Kubernetes manifest, and so on. Use the **variable picker** to insert dynamic values from the trigger or from previous action outputs into any field. Variables use the `{{variable_name}}` syntax and are type-checked at build time. Actions execute sequentially by default. Use parallel branches on the canvas to run actions concurrently. Configure a **cooldown period** to prevent the same trigger from firing the workflow repeatedly in a short window. For example, set a 5-minute cooldown so a pod crash-looping every 30 seconds doesn't generate dozens of duplicate tickets. You can also select "No cooldown" for workflows that should fire on every event. Click **Save** to activate the workflow. Use the **Test** button to simulate a trigger event and trace the workflow execution step by step. Each action shows its input, output, and execution time. You can also trigger a test run from the CLI with `kestrel workflow test `. ## Example Workflows | Use Case | Trigger | Actions | | ---------------------------- | --------------------------------- | ------------------------------------------------------------------------------- | | Self-healing K8s incident | Pod CrashLoopBackOff | RCA → Slack alert → Apply fix → Jira ticket | | Cloud provisioning via Slack | Slack `/kestrel-workflow` request | Parse request → Provision AWS resource → Confirm in thread | | CI/CD automation | GitHub PR merged | ArgoCD sync → Run tests → Notify Slack on failure | | Cost anomaly response | AWS cost spike detected | Investigate resources → Slack alert to #finops → Linear ticket | | Vercel deployment failure | Vercel Deployment Failed | Get build logs → AI investigation → Slack notification | | Jenkins build failure RCA | Jenkins Build Failed | Get console log → AI investigation → Slack alert to #ci → Jira ticket | | CircleCI flaky-build retry | CircleCI Workflow Failed | Rerun from failed → Wait for pipeline → PagerDuty page if it fails again | | Vault stale-secret rotation | Vault Secret Stale | Get secret metadata → Slack approval → Rotate static role → Confirm new version | | Infisical sync recovery | Infisical Secret Sync Failed | Slack approval → Trigger secret sync → Notify #secrets on result | | Terraform apply approval | Terraform Run Needs Attention | Get run plan summary → Slack approval → Apply or Discard run | | Pulumi update failure retry | Pulumi Update Failed | AI investigation → Slack approval → Run deployment (update) → Wait for result | | PostHog exception alert | PostHog Session Error | Session summary → Slack alert → Jira ticket if critical | ## What's Next * [Create Custom Integrations](/workflows/custom-integrations) — build custom HTTP action blocks and webhook triggers for any API or service * [Workflow Observability](/workflows/observability) — monitor execution history, latency, and failure rates * [Set Up Integrations](/workflows/setup-integrations) — connect the tools your workflows need # CLI Source: https://docs.usekestrel.ai/workflows/cli Create, manage, and trigger workflows from the command line The Kestrel CLI lets you manage workflows from your terminal. Use it for scripting, CI/CD integration, and day-to-day workflow management. ## Installation ### Homebrew (macOS / Linux) ```bash theme={null} brew install KestrelAI/tap/kestrel ``` ### Binary Download Download the latest release from [GitHub Releases](https://github.com/KestrelAI/kestrel-cli/releases): ```bash theme={null} # macOS (Apple Silicon) curl -L https://github.com/KestrelAI/kestrel-cli/releases/latest/download/kestrel_darwin_arm64.tar.gz | tar xz sudo mv kestrel /usr/local/bin/ # macOS (Intel) curl -L https://github.com/KestrelAI/kestrel-cli/releases/latest/download/kestrel_darwin_amd64.tar.gz | tar xz sudo mv kestrel /usr/local/bin/ # Linux (x64) curl -L https://github.com/KestrelAI/kestrel-cli/releases/latest/download/kestrel_linux_amd64.tar.gz | tar xz sudo mv kestrel /usr/local/bin/ ``` ## Authentication ### Create an Account New to Kestrel? You can sign up directly from the terminal: ```bash theme={null} kestrel register ``` You'll be prompted for a work email and password (8–30 characters, hidden input). If email verification is enabled on the server, a 6-character code is sent to your inbox — enter it at the prompt and you're logged in automatically. You can also verify later: ```bash theme={null} kestrel verify-email ``` Non-interactive usage (e.g. from a coding agent or script): ```bash theme={null} kestrel register --email you@company.com --password '...' --server https://platform.usekestrel.ai kestrel verify-email ABC123 ``` After verification the CLI saves your session to `~/.kestrel/config.json` and prints suggested next steps (`kestrel integrations connect ...`). ### API Key (recommended) Create an API key in the Kestrel platform under **Workflows > API Keys**, then: ```bash theme={null} kestrel auth kestrel_sk_... ``` ### Email / Password ```bash theme={null} kestrel login ``` You'll be prompted for your server URL, email, and password. Credentials are stored locally at `~/.kestrel/config.json`. ```bash theme={null} kestrel login --server https://platform.usekestrel.ai --email you@company.com ``` ### Check Status ```bash theme={null} kestrel status ``` ### Log Out ```bash theme={null} kestrel logout ``` ## Workflow Commands ### List workflows ```bash theme={null} kestrel workflows list kestrel workflows list --status active ``` Aliases: `kestrel wf list` ### Get workflow details Shows the workflow definition, trigger config, and an ASCII diagram of the workflow DAG: ```bash theme={null} kestrel workflows get ``` ### Create a workflow from file ```bash theme={null} kestrel workflows create --file workflow.json ``` ### Generate from natural language Use the AI agent to generate a workflow from a description: ```bash theme={null} kestrel workflows generate "When a pod crashloops, run RCA, create a Jira ticket, and notify #incidents on Slack" ``` Add `--save` to immediately save as a draft: ```bash theme={null} kestrel workflows generate --save "When a PagerDuty alert fires, investigate the cluster and post findings to Slack" ``` ### Edit a workflow ```bash theme={null} kestrel workflows edit --name "New Name" kestrel workflows edit --description "Updated description" ``` ### Delete a workflow ```bash theme={null} kestrel workflows delete ``` ### Activate / Pause ```bash theme={null} kestrel workflows activate kestrel workflows pause ``` ### Duplicate ```bash theme={null} kestrel workflows duplicate kestrel workflows duplicate --name "Copy of my workflow" ``` ### Test a workflow Run a dry-run execution: ```bash theme={null} kestrel workflows test ``` ### Request a workflow Submit a natural language request to trigger a matching workflow. This is the CLI equivalent of `/kestrel-workflow` in Slack or the "Make a Request" page in the Kestrel Platform UI: ```bash theme={null} kestrel workflows request "provision an MSK cluster with 3 brokers in us-east-1" kestrel workflows request "scale the payments deployment to 5 replicas" kestrel workflows request "create a DNS record for api.example.com in Cloudflare" kestrel workflows request "add a new PagerDuty escalation policy for the payments team" ``` If the matched workflow requires additional parameters, you'll be prompted interactively: ``` $ kestrel workflows request "create a configmap" Routing request: create a configmap ! Matched workflow: On-Demand K8s Resource Provision Extracted: resource_type=ConfigMap Please provide the following details: cluster: prod-cluster namespace: payments name: app-config Submitting with parameters... ✓ Matched workflow: On-Demand K8s Resource Provision ✓ Workflow is now executing. ``` If no workflow matches your request, the CLI asks whether you want to send it to your platform team as a workflow request (instead of filing one automatically): ``` $ kestrel workflows request "set up a Redis cluster for the cache layer" Routing request: set up a Redis cluster for the cache layer ✗ No matching workflow found Category: general Send this to your platform team as a workflow request? They can create a workflow to handle this type of request. [y/N]: y ✓ Request sent to your platform team. Request ID: 4f8a... ``` Answering `n` (or pressing Enter) dismisses the request — it won't appear in the platform's Workflow Requests page. ## Execution Commands ```bash theme={null} kestrel workflows executions kestrel workflows executions --page 2 --page-size 10 ``` ### Statistics ```bash theme={null} kestrel workflows stats ``` ## Approval Commands ```bash theme={null} kestrel approvals list kestrel approvals approve kestrel approvals approve --justification "Emergency fix" kestrel approvals reject ``` ## Version History & Rollback ```bash theme={null} kestrel workflows versions # List version history kestrel workflows rollback --version 3 # Roll back to version 3 ``` ## Execution Replay Replay failed executions from the beginning or from the failed step: ```bash theme={null} kestrel workflows replay # Replay from beginning kestrel workflows replay --from-failed # Replay from failed step ``` ## Request Commands ```bash theme={null} kestrel requests list kestrel requests approve kestrel requests reject ``` ## API Key Commands ```bash theme={null} kestrel apikeys list kestrel apikeys create kestrel apikeys create --scopes "workflows:read,catalog:read" --expires-in 90d kestrel apikeys revoke kestrel apikeys delete ``` ## Catalog and Integrations ```bash theme={null} kestrel workflows catalog # Available triggers and actions kestrel workflows integrations # Integration connection status kestrel workflows suggestions # AI-suggested workflows ``` ## Integration Commands Connect, test, and disconnect integrations (cloud providers, CI/CD, databases, alerting, knowledge sources, and more) directly from the terminal. Aliases: `kestrel integration`, `kestrel int`. Your API key needs the `integrations:read` and `integrations:manage` scopes to manage integrations. The key created during onboarding includes them by default. ### List integrations Shows every integration with its type and current connection status: ```bash theme={null} kestrel integrations list ``` ### Connect an integration Each integration has its own `connect` subcommand: ```bash theme={null} kestrel integrations connect ``` You can pass credentials as flags, but you don't have to — running the bare command walks you through setup interactively: * The CLI prints setup instructions for the integration (where to create the API token, which permissions to grant, etc.), mirroring the platform UI. * It prompts for each required value. **Secrets are read with hidden input** — they never appear on the command line or in your shell history. ```bash theme={null} $ kestrel integrations connect cloudflare Connecting Cloudflare API token: Cloudflare dashboard -> My Profile -> API Tokens -> Create Token -> ... Account ID: in the dashboard URL — dash.cloudflare.com//home. API token (hidden): Cloudflare account ID: 023e105f4ecef8ad9ca31a8372d0c353 ✓ Cloudflare connected ``` To see the flags and setup instructions for any integration: ```bash theme={null} kestrel integrations connect --help ``` Non-secret values can be passed as flags; file-based credentials support a `---file` variant: ```bash theme={null} kestrel integrations connect terraform --organization my-org kestrel integrations connect nebius --credentials-file authorized-key.json kestrel integrations connect confluence --base-url https://acme.atlassian.net --email me@acme.com ``` ### Connection types | Type | Integrations | How it works | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Token | Cloudflare, PagerDuty, Datadog, ArgoCD, Jenkins, CircleCI, Terraform Cloud, Pulumi Cloud, Vercel, Railway, Fly.io, Nebius, Beam, Daytona, Supabase, PlanetScale, Neon, ClickHouse, PostHog, HashiCorp Vault, Infisical, and more | Prompts for API tokens/credentials (hidden input) and connects immediately | | OAuth | GitHub, GitLab, Slack | Prints an authorization/install URL to open in your browser | | Knowledge | Jira, Linear, Confluence, Notion, Glean | Connects and immediately runs a connection test | | Cluster | Kubernetes | Mints an operator token, writes a Helm values file, and prints the `helm install` command | | Cloud | AWS, OCI | AWS is a two-step IAM role flow (bootstrap, then verify with `--role-arn`); OCI prompts for API key details | ### Test a connection ```bash theme={null} kestrel integrations test ``` Supported for token integrations and knowledge sources. ### Disconnect an integration ```bash theme={null} kestrel integrations disconnect ``` OAuth, cluster, and cloud integrations must be disconnected in the Kestrel UI. # Create Workflows Source: https://docs.usekestrel.ai/workflows/create-workflows Build automated workflows using natural language, drag-and-drop, CLI, SDK, or MCP Kestrel workflows let you chain AI-powered actions across 15+ integrations into automated pipelines. Workflows are triggered by events — Kubernetes incidents, cloud alerts, Slack messages, PagerDuty alerts, and more — and execute a sequence of steps that investigate, fix, and notify. ## Creation Methods You can create workflows five ways: | Method | Best For | | ------------------------ | ----------------------------------------------------------------------------------------------- | | **Workflow Agent** | Fastest way to build — describe what you want in natural language, the Workflow Agent builds it | | **Drag-and-Drop Canvas** | Visual construction with full control over step configuration | | **CLI** | Scripting and automation — see [CLI reference](/workflows/cli) | | **SDK** | Programmatic workflow management — see [SDK reference](/workflows/sdk) | | **MCP** | AI agent orchestration (Cursor, Claude, etc.) — see [MCP reference](/workflows/mcp) | *** ## Workflow Agent Describe the workflow you want in plain English. The Workflow Agent interprets your intent, selects the right triggers and actions, wires up template variables, and generates a complete workflow on the canvas. Navigate to **Workflows → All Workflows** in the Kestrel dashboard and click **Create Workflow**. Type a natural language description of what you want automated. Be specific about the trigger, the actions, and where results should go. The Workflow Agent generates the workflow on the canvas. Review each step and iterate the prompt to adjust configuration, add, or remove steps as needed. Click **Save** to activate the workflow. It will begin responding to its configured trigger immediately. **Example prompts:** * *"When a K8s incident is detected, run root cause analysis, send the RCA to #incidents in Slack, and create a Jira ticket with the details."* * *"When someone requests a new service in Slack, generate the K8s manifests, open a GitOps PR in GitHub, and sync with ArgoCD."* * *"When a Vercel deployment fails, pull the build logs, run an AI investigation, and notify the team in Slack."* * *"When a Railway service crashes, fetch the logs, investigate the root cause, roll back to the previous deployment, and alert #platform in Slack."* * *"When AWS costs spike, run a cost anomaly analysis and post a report to #finops in Slack."* The more specific your description, the better the generated workflow. Include the trigger source, desired actions, and output destinations. *** ## Drag-and-Drop Canvas The visual canvas gives you full control over workflow construction. Add triggers and actions from the sidebar, connect them into a pipeline, and configure each step individually. ### Adding a Trigger Every workflow starts with a trigger. Drag a trigger block from the sidebar onto the canvas and configure it: 1. Select the trigger type (e.g., Pod CrashLoopBackOff, IAM Security Event, Slack Request) 2. Configure trigger-specific settings (clusters, namespaces, AWS accounts, severity thresholds) 3. The trigger's output variables become available to all downstream steps ### Adding Actions Drag action blocks onto the canvas and connect them to the trigger or to previous actions: 1. Select the action type (e.g., Kestrel RCA, Slack Message, GitHub PR) 2. Configure action parameters — use the **variable picker** to reference data from previous steps 3. Connect the action's output port to the next step's input port ### Connecting Steps Click an output port and drag to an input port to create a connection. Data flows top-to-bottom through the connections you define. *** ## Trigger Types ### Kubernetes Signals Triggered when Kestrel detects a K8s incident in a connected cluster. Configure filters by cluster, namespace, or incident type. | Trigger Block | Description | | ---------------------------- | --------------------------------------------------------- | | Deployment Replicas Failing | Triggers when a Deployment has unavailable replicas | | Nodes Unavailable | Triggers when a node enters NotReady or unreachable state | | Pod CrashLoopBackOff | Triggers when a pod enters CrashLoopBackOff | | Pod ImagePullBackOff | Triggers when a pod cannot pull its container image | | Pod OOMKilled | Triggers when a container is killed due to out-of-memory | | Pods Failing | Triggers when a pod exits with a non-zero code | | Pods Restarting | Triggers when pod restart count exceeds threshold | | StatefulSet Replicas Failing | Triggers when a StatefulSet has unavailable replicas | | Node Memory Pressure | Triggers when a node reports memory pressure | | Node Disk Pressure | Triggers when a node reports disk pressure | | DaemonSet Failing | Triggers when a DaemonSet has failing replicas | | Any Kubernetes Incident | Triggers on any Kubernetes incident signal | ### AWS Cloud Signals Triggered when Kestrel detects a cloud infrastructure incident. Filter by AWS account, region, or incident type. | Trigger Block | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | IAM Security Event | Triggers on IAM compromise, privilege escalation, or unauthorized access | | Root Account Activity | Triggers when the AWS root account is used | | KMS Key Change | Triggers on KMS key deletion, disabling, or scheduled deletion | | Secrets Manager Event | Triggers on secret deletion, policy changes, or rotation failures | | Security Hub Finding | Triggers on high or critical findings from Security Hub | | S3 Bucket Change | Triggers on S3 public access changes, policy modifications, or encryption changes | | EC2 Instance Issue | Triggers on EC2 instance termination, state changes, or status check failures | | Lambda Function Issue | Triggers on Lambda function errors, configuration issues, or throttling | | ECS/EKS Container Issue | Triggers on container service task failures or deployment issues | | RDS Database Issue | Triggers on RDS database deletion, snapshot issues, or availability problems | | DynamoDB Issue | Triggers on DynamoDB table deletion, throttling, or capacity issues | | VPC/Network Change | Triggers on VPC, security group, NACL, or route table modifications | | CloudWatch Alarm | Triggers when a CloudWatch alarm enters ALARM state | | Application Log Errors | Triggers when application errors are detected in CloudWatch Logs | | Config Rule Non-Compliant | Triggers when an AWS Config rule finds a non-compliant resource | | AWS Service Health Event | Triggers on AWS service health events affecting your resources | | Any AWS Cloud Incident | Triggers on any AWS cloud incident | | Cost Anomaly Detected | Triggers when AWS Cost Anomaly Detection finds an anomaly | | Budget Threshold Exceeded | Triggers when an AWS Budget threshold is breached | | Forecast Exceeds Budget | Triggers when the AWS cost forecast projects a budget overrun before it happens | | Spend Spike Detected | Triggers when day-over-day spend rises beyond a configurable percentage threshold | | Idle Resource Detected | Triggers when a daily scan finds idle resources (unattached EBS volumes, unassociated Elastic IPs, low-CPU instances, old snapshots) | ### Schedule Triggered on a recurring cadence rather than by an external event. Ideal for periodic cost reports, cleanup scans, and audits. Configure the interval (hourly, daily, weekly, or monthly), the time of day (UTC), and — for weekly/monthly schedules — the day of week or day of month. | Trigger Block | Description | | ------------------ | -------------------------------------------------------------------- | | Recurring Schedule | Triggers on a recurring schedule (hourly, daily, weekly, or monthly) | Scheduled triggers expose `{{signal.schedule_interval}}`, `{{signal.fired_at}}`, and `{{signal.fired_date}}` to downstream steps. ### Slack Requests Triggered when a user sends a `/kestrel-workflow` command in Slack. The user's message is available as `{{request.prompt}}`. | Trigger Block | Description | | ----------------------------------- | ----------------------------------------------------------------- | | Create K8s Resource | Triggered when someone requests a new Kubernetes resource | | Edit K8s Resource | Triggered when someone requests changes to a Kubernetes resource | | General Kubernetes Request | Triggered when someone asks a Kubernetes-related question or task | | Create AWS Resource | Triggered when someone requests a new AWS resource | | Edit AWS Resource | Triggered when someone requests changes to an AWS resource | | General Cloud Request | Triggered when someone asks a cloud-related question or task | | Any Slack /kestrel-workflow Request | Triggers on any /kestrel-workflow request (catch-all) | ### PagerDuty Triggered when a PagerDuty incident is created or updated. Filter by service, urgency, or severity. | Trigger Block | Description | | ---------------------- | -------------------------------------------------- | | Incident Triggered | Triggers when a new PagerDuty incident is created | | Incident Acknowledged | Triggers when a PagerDuty incident is acknowledged | | Incident Resolved | Triggers when a PagerDuty incident is resolved | | Any PagerDuty Incident | Triggers on any PagerDuty incident event | | High Urgency Incident | Triggers only on high-urgency PagerDuty incidents | ### PostHog Triggered by PostHog product analytics events. Filter by event type and properties. | Trigger Block | Description | | ----------------------- | ------------------------------------------------------ | | Session Error/Exception | Triggers when PostHog captures a frontend exception | | Console Error | Triggers when PostHog captures a console error | | Rage Click | Triggers when PostHog detects rage clicks in a session | | Any PostHog Event | Triggers on any PostHog event sent via webhook | | Log Error Alert | Triggers when PostHog Logs detects error patterns | ### Vercel Triggered by Vercel deployment and platform events. Filter by project and event type. | Trigger Block | Description | | ------------------------ | -------------------------------------------------------------- | | Deployment Failed | Triggers when a Vercel deployment build fails | | Deployment Succeeded | Triggers when a Vercel deployment succeeds | | Deployment Created | Triggers when a new Vercel deployment is created | | Error Anomaly | Triggers when Vercel detects a 5xx error rate anomaly | | Usage Anomaly | Triggers when Vercel detects abnormal usage patterns | | Domain Issue | Triggers when a Vercel project domain has configuration issues | | Firewall Attack Detected | Triggers when Vercel's WAF detects an attack | | Deployment Checks Failed | Triggers when post-deployment checks fail | | Production Rollback | Triggers when a production deployment is rolled back | ### Railway Triggered by Railway deployment and platform events. Filter by project, environment, and event type. | Trigger Block | Description | | --------------------- | ----------------------------------------------------------------- | | Deployment Failed | Triggers when a Railway deployment build or deploy fails | | Deployment Crashed | Triggers when a Railway service crashes after a successful deploy | | Deployment Succeeded | Triggers when a Railway deployment succeeds and is live | | Volume Usage Alert | Triggers when a Railway volume approaches its capacity | | CPU/RAM Monitor Alert | Triggers when a Railway service exceeds CPU or memory thresholds | ### Fly.io Triggered by Fly.io machine lifecycle changes. Fly.io has no outbound webhooks, so Kestrel detects these events by polling the Fly Machines API and diffing machine state on a configurable interval. Filter by app, region, and event type, and set the poll cadence (1m / 5m / 15m / 30m). | Trigger Block | Description | | --------------- | ------------------------------------------------------------ | | Machine Crashed | Triggers when a Fly machine crashes (non-zero exit / OOM) | | Machine Stopped | Triggers when a machine transitions to stopped/suspended | | Machine Started | Triggers when a machine transitions to started | | App Down | Triggers when all of an app's machines are stopped/unhealthy | ### Nebius AI Cloud Triggered by Nebius GPU/compute and Managed Kubernetes node changes. Nebius has no outbound webhooks, so Kestrel detects these events by polling the Nebius compute and Managed Kubernetes APIs and diffing instance state and node conditions on a configurable interval. Filter by project, cluster, and event type, and set the poll cadence (1m / 5m / 15m / 30m). | Trigger Block | Description | | --------------------- | ------------------------------------------------------- | | GPU Error | Triggers when a node reports a GPU error condition | | Maintenance Scheduled | Triggers when a node has scheduled maintenance | | Node Not Ready | Triggers when a Kubernetes node leaves the Ready state | | Instance Stopped | Triggers when a compute instance transitions to stopped | ### Jenkins Triggered by Jenkins build lifecycle events, delivered by a webhook notification plugin (or a post-build curl step) authenticated with a per-tenant shared secret. Filter by job name (full path for folder-nested jobs) and build status. | Trigger Block | Description | | ---------------------------- | ------------------------------------------------------------- | | Build Failed | Triggers when a build completes with FAILURE | | Build Unstable | Triggers when a build completes UNSTABLE (e.g. test failures) | | Build Succeeded | Triggers when a build completes with SUCCESS | | Build Completed (any result) | Triggers when a build finishes with any result | | Build Started | Triggers when a build starts | ### CircleCI Triggered by CircleCI's native outbound webhooks (workflow-completed / job-completed), signed with HMAC-SHA256 and verified by Kestrel. Filter by project slug, branch, and status. | Trigger Block | Description | | ------------------------------- | ---------------------------------------------------------------------------------------- | | Workflow Failed | Triggers when a workflow completes with failed or error status | | Workflow Succeeded | Triggers when a workflow completes successfully | | Workflow Completed (any status) | Triggers when a workflow finishes with any status | | Job Failed | Triggers when a job completes with failed status (finer-grained than workflow-completed) | ### Terraform Cloud Triggered by Terraform Cloud (HCP Terraform) run lifecycle and drift events, delivered by per-workspace **notification configurations** (webhooks) signed with HMAC-SHA512 and verified by Kestrel. Filter by workspace, event type, and run status. | Trigger Block | Description | | ----------------------- | ---------------------------------------------------------------------------- | | Run Created | Triggers when a new run is queued on a workspace | | Run Planning | Triggers when a run starts planning | | Run Needs Attention | Triggers when a plan finishes and the run awaits confirmation | | Run Applying | Triggers when a run starts applying | | Run Completed | Triggers when a run applies successfully or finishes as planned-and-finished | | Run Errored | Triggers when a run fails (plan/apply error, policy hard-fail) | | Drift Detected | Triggers when a workspace health assessment detects infrastructure drift | | Assessment Check Failed | Triggers when a workspace health assessment fails to run | ### Pulumi Cloud Triggered by Pulumi Cloud stack, update, deployment, drift, and policy events, delivered by organization or stack **webhooks** signed with HMAC-SHA256 (`Pulumi-Webhook-Signature`) and verified by Kestrel. Filter by stack (project/stack reference), project, and event type. | Trigger Block | Description | | -------------------- | -------------------------------------------------------------- | | Update Succeeded | Triggers when a stack update (`pulumi up`) succeeds | | Update Failed | Triggers when a stack update, refresh, or destroy fails | | Preview Failed | Triggers when a stack preview fails | | Destroy Succeeded | Triggers when a stack destroy succeeds | | Deployment Started | Triggers when a Pulumi Deployments run starts executing | | Deployment Succeeded | Triggers when a Pulumi Deployments run succeeds | | Deployment Failed | Triggers when a Pulumi Deployments run fails | | Drift Detected | Triggers when a drift detection run finds infrastructure drift | | Drift Run Failed | Triggers when a drift detection or remediation run fails | | Policy Violation | Triggers when a CrossGuard policy violation is detected | | Stack Created | Triggers when a new stack is created in the organization | | Stack Deleted | Triggers when a stack is deleted from the organization | ### HashiCorp Vault Triggered by changes detected by **polling** the Vault API (Vault has no outbound control-plane webhooks). Pollers read only KV metadata — never secret values. Filter secret triggers by mount and secret-path prefix; the poll cadence is configurable per trigger (1m–30m). | Trigger Block | Description | | ------------------------------ | ------------------------------------------------------------------------------------------ | | Vault Sealed | Triggers when the cluster seals (secrets become unavailable) | | Vault Unsealed | Triggers when the cluster is unsealed | | Health Degraded | Triggers when Vault becomes unreachable or reports an unhealthy status | | Secret Version Created | Triggers when a KV v2 secret gets a new version (metadata diff) | | Secret Stale | Triggers when a secret's latest version exceeds a rotation age threshold (default 90 days) | | Policy Created / Deleted | Triggers when ACL policies change | | Auth Method Enabled / Disabled | Triggers when authentication methods change | ### Infisical Triggered by changes detected by **polling** the Infisical audit log and project APIs. Pollers never read secret values. Filter secret triggers by project, environment slug, and folder-path prefix; the poll cadence is configurable per trigger (1m–30m, default 5m). | Trigger Block | Description | | ---------------------------------- | --------------------------------------------------------------------- | | Secret Created / Updated / Deleted | Trigger when secrets change in a project (derived from the audit log) | | Approval Requested | Triggers when a secret-change approval request is opened | | Secret Sync Failed | Triggers when a secret sync to an external destination fails | | Identity Created | Triggers when a new machine identity appears in the organization | ### Custom Webhook Triggered by an HTTP POST to a unique webhook URL. Accepts any JSON payload with optional HMAC signature verification. See [Custom Integrations](/workflows/custom-integrations) for details. *** ## Action Types Actions are grouped by integration. Each action produces output variables that downstream steps can reference. Trigger K8s RCA & Generate Fix, Apply YAML Fix, Find Causal PRs/MRs, Trigger Cloud RCA & Generate Fix, Generate Runbook, Wait, Generate K8s Manifest, Apply K8s Manifest, Create GitOps PR, Generate Helm Values, Generate Cloud Resource, Execute Cloud CLI, Create IaC PR, Investigate Cloud, Investigate Kubernetes, AI Cost Analysis Create Pull Request, Create Issue, Trigger GitHub Action, Wait for GitHub Action Run, Get GitHub Action Status, Read File, Search Code, AI Code Investigation, AI Code Fix, Wait for PR Approval, Wait for PR Merge Create Merge Request, Create Issue, Trigger Pipeline, Wait for Pipeline, Get Pipeline Status, Wait for MR Approval, Wait for MR Merge Send Message, Update Message, Request Justification Publish RCA, Publish Postmortem, Publish Runbook Entry, Update Page Create Ticket, Add Comment, Transition Ticket Create Issue, Add Comment, Update Issue, Search Issues Create Alert, Acknowledge Alert, Add Note to Alert, Resolve Alert, Escalate Trigger ArgoCD Sync, Wait for ArgoCD Sync, Get ArgoCD App Status Trigger Build, Wait for Build, Get Build Status, Stop Build, Get Console Log, Investigate Jenkins Trigger Pipeline, Wait for Pipeline, Get Workflow Status, Rerun Workflow, Cancel Workflow, Approve On-Hold Job, Get Job Test Results, Investigate CircleCI Query Metrics, Create Monitor, Send Event, Mute Monitor Query Cost Explorer, Get Cost Anomalies, Get Cost Forecast, Get Budget Status, Get Rightsizing Recommendations, Get Savings Plans Recommendations, Get Reservation Recommendations, Get Commitment Utilization, Compare Cost Periods, Find Idle Resources, Get Compute Optimizer Recommendations, Get Trusted Advisor Cost Checks, Stop EC2 Instances, Delete Unattached EBS Volumes, Release Elastic IPs, Delete Old Snapshots Get Session Summary, Get Session Recording, Query Events, List Session Recordings, Get Error Issue Get Deployment, Get Build Logs, Rollback Production, Promote to Production, List Deployments, Investigate Vercel Get Deployment, Get Deployment Logs, Rollback, Redeploy, Restart, List Deployments, Set Variables, Investigate Railway Restart Machine, Start Machine, Stop Machine, Suspend Machine, Cordon Machine, Uncordon Machine, Get Machine, Get Machine Events, List Machines, Set Secrets, Investigate Fly Get Instance, Start Instance, Stop Instance, Restart Instance, List Instances, List Clusters, List Node Groups, Scale Node Group, Investigate Nebius List/Get Workspace, Lock/Unlock/Force-Unlock Workspace, List/Get Run, Create Run (Plan), Create Destroy Run, Apply Run, Discard Run, Cancel Run, Wait for Run, Get State Outputs, List Variables, Set Variable, Get Drift Assessment, Investigate Terraform List/Get Stack, List/Get Update, Run Deployment, Get/Wait for/Cancel Deployment, Pause/Resume Deployments, Get Stack Outputs, Get Drift Status, Set/Delete Stack Tag, Investigate Pulumi Read/Write/Delete Secret, List Secrets, Get Secret Metadata, Rotate Static Role, List Mounts, List/Read/Write Policy, List Auth Methods, List/Revoke/Renew Lease, Get Health, List/Revoke Token Accessor, Investigate Vault Get/Create/Update/Delete Secret, List Secrets, List Projects/Environments/Folders, Create Folder, List/Trigger Secret Sync, List Approval Requests, Get Audit Logs, List Identities, Investigate Infisical Wait for Manual Approval, Wait for Slack Approval, Wait for PR/MR Approval, Refine RCA with Feedback Create custom action blocks to call any API endpoint, and custom webhook triggers to start workflows from external systems. See [Custom Integrations](/workflows/custom-integrations). The Cloud Cost remediation blocks (**Stop EC2 Instances**, **Delete Unattached EBS Volumes**, **Release Elastic IPs**, **Delete Old Snapshots**) mutate real infrastructure. Always place them behind an [Approval Gate](#approval-gates). Resources tagged `kestrel:protected` are always skipped, every remediation block is pinned to an explicit region, and safety caps limit how many resources a single run can touch. *** ## Template Variables Template variables let you pass data between workflow steps. Use double curly braces to reference outputs from triggers and previous actions. ### Variable Syntax ``` {{incident.title}} — Data from the trigger event {{rca_result.root_cause}} — Promoted field from RCA actions {{step_outputs.action-1.summary}} — Output from a specific action step {{request.prompt}} — User input (for Slack request triggers) ``` ### Common Variables | Variable | Source | Description | | ----------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------- | | `{{incident.title}}` | K8s/Cloud trigger | Incident title | | `{{incident.namespace}}` | K8s trigger | Kubernetes namespace | | `{{incident.resource_name}}` | K8s/Cloud trigger | Affected resource | | `{{rca_result.root_cause}}` | Kestrel RCA action | Root cause summary | | `{{rca_result.recommended_fix}}` | Kestrel RCA action | Recommended remediation | | `{{fix.description}}` | Kestrel RCA action | Generated fix description | | `{{step_outputs.action-1.summary}}` | Any action step | Summary output of the step | | `{{request.prompt}}` | Slack request trigger | The user's original message | | `{{request.requester}}` | Slack request trigger | The requesting user | | `{{signal.deployment_id}}` | Vercel trigger | Vercel deployment ID | | `{{signal.deployment_id}}` | Railway trigger | Railway deployment ID | | `{{signal.service_id}}` | Railway trigger | Railway service ID | | `{{signal.environment_id}}` | Railway trigger | Railway environment ID | | `{{signal.app_name}}` | Fly.io trigger | Fly app name | | `{{signal.machine_id}}` | Fly.io trigger | Fly machine ID | | `{{signal.region}}` | Fly.io trigger | Fly region (e.g. iad) | | `{{signal.exit_code}}` | Fly.io trigger | Machine exit code (crash events) | | `{{signal.project_id}}` | Nebius trigger | Nebius project ID | | `{{signal.instance_id}}` | Nebius trigger | Nebius compute instance ID | | `{{signal.cluster_id}}` | Nebius trigger | Nebius Managed Kubernetes cluster ID | | `{{signal.node_group_id}}` | Nebius trigger | Nebius node group ID | | `{{signal.node_name}}` | Nebius trigger | Kubernetes node name | | `{{signal.condition}}` | Nebius trigger | Node condition that fired (e.g. NebiusGPUError) | | `{{signal.job_name}}` | Jenkins / CircleCI trigger | Jenkins job full name, or CircleCI job name (job-completed events) | | `{{signal.build_number}}` | Jenkins trigger | Number of the build | | `{{signal.build_status}}` | Jenkins trigger | Build result (SUCCESS, FAILURE, UNSTABLE, ABORTED) | | `{{signal.build_url}}` | Jenkins trigger | Link to the build | | `{{signal.project_slug}}` | CircleCI trigger | Project slug (e.g. gh/org/repo) | | `{{signal.pipeline_id}}` | CircleCI trigger | ID of the pipeline | | `{{signal.workflow_id}}` | CircleCI trigger | ID of the workflow | | `{{signal.workflow_name}}` | CircleCI trigger | Name of the workflow | | `{{signal.job_number}}` | CircleCI trigger | Number of the job (job-completed events) | | `{{signal.status}}` | CircleCI trigger | Workflow/job status (success, failed, error, canceled) | | `{{signal.branch}}` | CircleCI trigger | VCS branch the pipeline ran on | | `{{signal.mount}}` | Vault trigger | Secrets engine mount (secret events, e.g. `secret/`) | | `{{signal.secret_path}}` | Vault / Infisical trigger | Secret path (Vault) or folder path (Infisical); values are never included | | `{{signal.secret_version}}` | Vault trigger | New version number (secret.version\_created events) | | `{{signal.secret_age_days}}` | Vault trigger | Days since the last version (secret.stale events) | | `{{signal.policy_name}}` | Vault trigger | ACL policy name (policy events) | | `{{signal.auth_method}}` | Vault trigger | Auth method path (auth\_method events) | | `{{signal.project_id}}` | Infisical trigger | Infisical project the event is about | | `{{signal.environment}}` | Infisical trigger | Environment slug (secret events, e.g. `prod`) | | `{{signal.secret_key}}` | Infisical trigger | Secret key name (secret events; values are never included) | | `{{signal.sync_id}}` | Infisical trigger | Secret sync (sync.failed events) | | `{{signal.actor}}` | Infisical trigger | Who performed the action (user email or identity name) | | `{{signal.session_id}}` | PostHog trigger | PostHog session ID | | `{{signal.workspace}}` | Terraform Cloud trigger | Workspace name | | `{{signal.run_id}}` | Terraform Cloud trigger | Run ID (run-...) | | `{{signal.run_status}}` | Terraform Cloud trigger | Run status at notification time | | `{{signal.run_url}}` | Terraform Cloud trigger | Link to the run in Terraform Cloud | | `{{signal.organization}}` | Terraform Cloud trigger | Terraform Cloud organization | | `{{signal.session_id}}` | PostHog trigger | PostHog session ID | | `{{signal.stack}}` | Pulumi Cloud trigger | Stack reference (project/stack) | | `{{signal.project}}` | Pulumi Cloud trigger | Pulumi project name | | `{{signal.operation}}` | Pulumi Cloud trigger | Operation kind (update, preview, refresh, destroy, detect-drift, remediate-drift) | | `{{signal.result}}` | Pulumi Cloud trigger | Operation result (succeeded, failed) | | `{{signal.update_version}}` | Pulumi Cloud trigger | Stack update version number | | `{{signal.deployment_id}}` | Pulumi Cloud trigger | Pulumi Deployments run ID (deployment events) | | `{{signal.update_url}}` | Pulumi Cloud trigger | Link to the update in the Pulumi Cloud console | | `{{signal.user}}` | Pulumi Cloud trigger | User who initiated the operation | | `{{signal.aws_account_id}}` | AWS cost trigger | AWS account ID for cost anomaly, budget, forecast, spend spike, and idle resource signals | | `{{signal.schedule_interval}}` | Schedule trigger | Configured cadence (hourly, daily, weekly, monthly) | | `{{signal.fired_at}}` | Schedule trigger | UTC timestamp (RFC3339) when the schedule fired | | `{{signal.fired_date}}` | Schedule trigger | UTC date (YYYY-MM-DD) when the schedule fired | ### Variable Picker When configuring an action step, click the **Variables** button on any text field to open the variable picker. It shows all available variables from the trigger and preceding steps, grouped by source. The picker is DAG-aware — it only shows variables that will be available at runtime based on the step's position in the workflow. The variable picker only shows variables that are available at the current step's position in the workflow. Move a step earlier or later to change which variables it can access. *** ## Cooldown Configuration Cooldowns prevent a workflow from firing repeatedly for the same event. Configure the cooldown period per workflow. | Preset | Duration | Default For | | ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | No cooldown | 0 | — | | 5 minutes | 5m | — | | 10 minutes | 10m | Slack, PostHog, Vercel, Railway, Fly.io, Nebius, Jenkins, CircleCI, Terraform Cloud, Pulumi Cloud, Vault, Infisical, custom webhook triggers | | 30 minutes | 30m | — | | 1 hour | 1h | — | | 3 hours | 3h | — | | 12 hours | 12h | — | | 24 hours | 24h | K8s incidents, cloud incidents, PagerDuty alerts | Cooldowns are scoped to the specific trigger event. For K8s incidents, the cooldown applies per workload. For Slack requests, it applies per request. *** ## Approval Gates Insert approval gates between steps to require human sign-off before the workflow continues. Three approval methods are available: ### Wait for Manual Approval The workflow pauses and shows a pending approval in the Kestrel dashboard. An authorized user clicks **Approve** or **Reject**. ### Wait for Slack Approval The workflow posts an approval request to a configured Slack channel with **Approve** and **Reject** buttons. Any member of the channel can respond. ### Wait for PR/MR Approval The workflow waits for the associated pull request or merge request to be approved or merged (webhook-driven). Merging the PR approves the step; closing it rejects it. ### Refine RCA with Feedback (human-in-the-loop) A self-looping review gate for incident-response workflows. Place it after a **Trigger K8s RCA** or **Trigger Cloud RCA** step. It presents the root cause and proposed fix for review with three outcomes: * **Approve** — continues on the approved branch (e.g. apply the fix, publish the runbook). * **Reject** — continues on the rejected branch. * **Request changes** — you provide free-text guidance ("the real cause is the readiness probe timeout, not the image tag"), and the RCA agent **re-runs with your feedback** and comes back for review again. Feedback accumulates across rounds, so each pass builds on the last. The loop repeats until you approve or reject, capped by **Max Refine Rounds** (default 5). When the cap is hit, the workflow advances on the approved branch. Choose **Kestrel UI** or **Slack** as the review channel in the block config. *** ## Poll Until (Loops) The **Poll Until** block (Flow Control) repeatedly runs a single catalog action at a fixed interval and evaluates an exit condition against each iteration's output. It's the building block for "wait for X to reach state Y" automation — waiting for a sandbox to stop, a deployment to become healthy, a pipeline to finish, or a cloud resource to reach a target state. Drag **Poll Until** from the sidebar (next to Conditions) onto the canvas, then configure: * **Action to Poll** — any catalog action (searchable dropdown). Its config fields appear below and behave exactly like the standalone action block, including dynamic dropdowns (e.g. Cluster → Namespace scoping) and template variables. * **Output Field Path** — which output field of the polled action to test, chosen from the action's known outputs. * **Operator + Value** — `equals`, `not_equals`, `contains`, `not_contains`, `exists`, `not_exists`. Where the output has known discrete values (states, statuses) or refers to live resources (cluster IDs), the Value field is a dropdown. You can select **multiple values**: `equals`/`contains` are met when the output matches ANY selected value; `not_equals`/`not_contains` are met only when it matches NONE. * **Poll Interval** — how often to re-run the action (default 60s, minimum 30s). * **Timeout** — how long to keep polling (default 60 minutes). The block has two outgoing branches: * **met** — the exit condition held on some iteration; downstream steps see the final iteration's output. * **timeout** — the timeout elapsed before the condition was met. Polling is durable. The loop's schedule is persisted server-side, so a server restart mid-interval resumes the countdown where it left off instead of restarting the interval — and in-flight loops survive redeployments. *** ## For Each (Fan-Out) The **For Each** block (Flow Control) resolves a list from an upstream step's output at runtime and runs a single catalog action once per element, in order. It's the building block for "do X for every Y" automation — creating a Jira ticket for each new audit finding, sending a message per failed service, or opening an issue for every flagged resource — without knowing the number of items in advance. Drag **For Each** from the sidebar (Flow Control) onto the canvas, then configure: * **Items to Iterate Over** — a template that resolves to a list, e.g. `{{step_outputs.action-1.outputs.new_findings}}` from an Execute Script step that wrote an array to its outputs. JSON-encoded array strings and comma-separated strings also work. * **Action per Item** — any catalog action (searchable dropdown). Its config fields appear below and behave exactly like the standalone action block. Inside these fields, three extra template variables are available: * `{{item}}` — the current element (rendered as JSON when it's an object) * `{{item.}}` — a field of the current element, e.g. `{{item.title}}` * `{{item_index}}` — the zero-based index of the current element * **Max Items** — caps the fan-out (default 25, hard cap 100). Extra items are skipped and the aggregated output is marked `truncated`. * **Continue on Error** — when enabled, a failed item doesn't fail the whole block; remaining items still run. The block emits **one aggregated output**: `items_total`, `items_processed`, `succeeded`, `failed`, per-item `results`, and a flat `summary` string usable in downstream templates. Its outgoing edges fire after **all** items have been processed. For Each runs items once each, immediately and sequentially. To repeatedly re-run an action until a condition holds, use **Poll Until** instead. *** ## Generating Runbooks The **Generate Runbook** action (Kestrel) distills a completed RCA and its fixes into a reusable, generalized runbook for that *class* of incident — symptom, diagnosis steps, remediation steps, verification, and rollback. It requires an upstream RCA step. The generated runbook is available to downstream steps as `{{runbook_html}}` / `{{runbook_markdown}}`, and **Publish Runbook Entry** (Confluence) consumes `runbook_html` automatically. *** ## Testing Workflows Before relying on a workflow in production, test it with a simulated trigger: 1. Open the workflow on the canvas 2. Click **Test Workflow** 3. Kestrel generates a sample trigger payload based on the trigger type 4. Click **Run Test** to execute the workflow with the sample data 5. Inspect the output of each step in the execution detail view Test executions are marked as test runs in the [Observability dashboard](/workflows/observability) and do not count toward cooldowns. *** ## Example Workflows ### K8s Incident → RCA → Slack → Jira **Workflow Agent prompt:** *"When a Kubernetes incident is detected in any cluster, run root cause analysis, send the root cause and recommended fix to #incidents in Slack, and create a Jira ticket in the INFRA project with the full investigation summary."* | Step | Type | Configuration | | ------------ | ------------------------------ | ------------------------------------------------------------------------------------------- | | **Trigger** | Any Kubernetes Incident | All clusters | | **Action 1** | Trigger K8s RCA & Generate Fix | Include metrics and logs | | **Action 2** | Slack Send Message | Channel: `#incidents`, message: `{{incident.title}}` — `{{rca_result.root_cause}}` | | **Action 3** | Jira Create Ticket | Project: INFRA, summary: `{{incident.title}}`, body: `{{rca_result.investigation_summary}}` | ### K8s Incident → RCA → Refine with Feedback → Apply → Publish Runbook A full human-in-the-loop loop: detect the incident, run RCA, draft a reusable runbook, let an engineer iteratively refine the analysis, then apply the fix and publish the runbook. **Workflow Agent prompt:** *"When a Kubernetes pod crashes, run root cause analysis, generate a runbook, then let me review and refine the analysis before applying — when I approve, apply the YAML fix and publish the runbook to Confluence; if I reject, notify #incidents."* | Step | Type | Configuration | | -------------------------- | ---------------------------------- | -------------------------------------------------------- | | **Trigger** | Any Kubernetes Incident | All clusters | | **Action 1** | Trigger K8s RCA & Generate Fix | Include metrics and logs | | **Action 2** | Generate Runbook | Audience: SRE, include rollback | | **Approval** | Refine RCA with Feedback | Channel: Kestrel UI, Max Refine Rounds: 3 | | **On Approved → Action 3** | Apply YAML Fix | Auto-detected from RCA | | **On Approved → Action 4** | Publish Runbook Entry (Confluence) | Uses `{{runbook_html}}` | | **On Rejected → Action 5** | Slack Send Message | Channel: `#incidents`, message: RCA rejected by reviewer | On **Request changes**, the reviewer's free-text guidance re-runs the RCA agent and the gate re-opens — repeating until approve/reject or the round cap. ### Slack Request → K8s Manifest → GitOps PR → ArgoCD Sync **Workflow Agent prompt:** *"When someone requests a new Kubernetes resource via /kestrel-workflow in Slack, generate the K8s manifests based on their request, create a pull request in acme/k8s-manifests on GitHub, wait for the PR to be merged, then trigger an ArgoCD sync."* | Step | Type | Configuration | | ------------ | ----------------------------------- | ---------------------------- | | **Trigger** | Any Slack /kestrel-workflow Request | Catch-all | | **Action 1** | Generate K8s Manifest | Prompt: `{{request.prompt}}` | | **Action 2** | Create GitOps PR | Repo: `acme/k8s-manifests` | | **Approval** | Wait for PR/MR Approval | Wait for PR merge | | **Action 3** | Trigger ArgoCD Sync | Auto-detected from PR | ### Vercel Deployment Failure → Build Logs → Investigation → Slack **Workflow Agent prompt:** *"When a Vercel deployment fails, fetch the build logs, run an AI investigation to identify the root cause of the build failure, and send the findings to #deploys in Slack."* | Step | Type | Configuration | | ------------ | ------------------------ | ------------------------------------------------------------------ | | **Trigger** | Vercel Deployment Failed | All projects | | **Action 1** | Get Build Logs | Deployment: `{{signal.deployment_id}}` | | **Action 2** | Investigate Vercel | Query: analyze build logs from `{{step_outputs.action-1.summary}}` | | **Action 3** | Slack Send Message | Channel: `#deploys`, message: `{{step_outputs.action-2.summary}}` | ### Railway Deployment Crash → Logs → Investigation → Rollback → Slack **Workflow Agent prompt:** *"When a Railway service crashes, fetch the deployment logs, run an AI investigation to find the root cause, roll the service back to its previous deployment, and post a summary to #platform in Slack."* | Step | Type | Configuration | | ------------ | -------------------------- | -------------------------------------------------------------------------- | | **Trigger** | Railway Deployment Crashed | All projects | | **Action 1** | Get Deployment Logs | Deployment: `{{signal.deployment_id}}` | | **Action 2** | Investigate Railway | Query: analyze crash logs from `{{step_outputs.action-1.summary}}` | | **Action 3** | Rollback | Service: `{{signal.service_id}}`, Environment: `{{signal.environment_id}}` | | **Action 4** | Slack Send Message | Channel: `#platform`, message: `{{step_outputs.action-2.summary}}` | ### Fly.io Machine Crash → Events → Investigation → Restart → Slack **Workflow Agent prompt:** *"When a Fly.io machine crashes, pull its recent events, run an AI investigation to find the root cause, restart the machine, and post a summary to #platform in Slack."* | Step | Type | Configuration | | ------------ | ---------------------- | ------------------------------------------------------------------------------------------------- | | **Trigger** | Fly.io Machine Crashed | All apps, poll every 1m | | **Action 1** | Get Machine Events | App: `{{signal.app_name}}`, Machine: `{{signal.machine_id}}` | | **Action 2** | Investigate Fly | Query: find why machine `{{signal.machine_id}}` crashed using `{{step_outputs.action-1.summary}}` | | **Action 3** | Restart Machine | App: `{{signal.app_name}}`, Machine: `{{signal.machine_id}}` | | **Action 4** | Slack Send Message | Channel: `#platform`, message: `{{step_outputs.action-2.summary}}` | ### Nebius GPU Error → Investigation → Scale Node Group → Slack **Workflow Agent prompt:** *"When a Nebius GPU node reports an error, run an AI investigation to find the root cause, add a node to the affected node group, and post a summary to #ml-infra in Slack."* | Step | Type | Configuration | | ------------ | ------------------ | ----------------------------------------------------------------------------------- | | **Trigger** | Nebius GPU Error | All projects, poll every 5m | | **Action 1** | Investigate Nebius | Query: find why node `{{signal.node_name}}` reported a GPU error | | **Action 2** | Scale Node Group | Cluster: `{{signal.cluster_id}}`, Node Group: `{{signal.node_group_id}}`, Size: `3` | | **Action 3** | Slack Send Message | Channel: `#ml-infra`, message: `{{step_outputs.action-1.summary}}` | ### Jenkins Build Failure → Console Log → Investigation → Slack + Jira **Workflow Agent prompt:** *"When a Jenkins build fails, fetch the console log, run an AI investigation to identify the root cause of the failure, post the findings to #ci in Slack, and create a Jira ticket in the INFRA project."* | Step | Type | Configuration | | ------------ | -------------------- | ----------------------------------------------------------------------- | | **Trigger** | Jenkins Build Failed | All jobs | | **Action 1** | Get Console Log | Job: `{{signal.job_name}}`, Build: `{{signal.build_number}}` | | **Action 2** | Investigate Jenkins | Query: analyze the console log from `{{step_outputs.action-1.summary}}` | | **Action 3** | Slack Send Message | Channel: `#ci`, message: `{{step_outputs.action-2.summary}}` | | **Action 4** | Jira Create Ticket | Project: `INFRA`, description: `{{step_outputs.action-2.answer}}` | ### CircleCI Workflow Failure → Rerun From Failed → Wait → PagerDuty **Workflow Agent prompt:** *"When a CircleCI workflow fails on main, rerun only the failed jobs, wait for the pipeline to finish, and page on-call via PagerDuty with an AI investigation summary if the rerun also fails."* | Step | Type | Configuration | | ------------ | ------------------------ | ----------------------------------------------------------------------------------- | | **Trigger** | CircleCI Workflow Failed | Projects: all, branch: `main` | | **Action 1** | Rerun Workflow | Workflow: `{{signal.workflow_id}}`, from failed jobs only | | **Action 2** | Wait for Pipeline | Project: `{{signal.project_slug}}`, Pipeline: `{{signal.pipeline_id}}`, timeout 30m | | **Action 3** | Investigate CircleCI | Query: find why `{{signal.workflow_name}}` keeps failing (only on failure branch) | | **Action 4** | PagerDuty Create Alert | Summary: `{{step_outputs.action-3.summary}}` | ### PostHog Exception → Investigation → Slack ### Vault Stale Secret → Slack Approval → Rotate → Confirm **Workflow Agent prompt:** *"When a secret in secret/app/prod hasn't been rotated in 60 days, post its metadata to #secrets in Slack and ask for approval, then rotate the database static role and confirm the rotation."* | Step | Type | Configuration | | -------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------- | | **Trigger** | Vault Secret Stale | Mount: `secret/`, paths: `app/prod`, max age: 60 days | | **Action 1** | Get Secret Metadata | Mount: `{{signal.mount}}`, Path: `{{signal.secret_path}}` | | **Action 2** | Wait for Slack Approval | Channel: `#secrets`, message includes `{{signal.secret_path}}` age `{{signal.secret_age_days}}` days | | **On Approved → Action 3** | Rotate Static Role | Mount: `database/`, Role: `app-db` | | **Action 4** | Slack Send Message | Channel: `#secrets`, message: `{{step_outputs.action-3.summary}}` | ### Infisical Sync Failure → Slack Approval → Retry Sync **Workflow Agent prompt:** *"When an Infisical secret sync fails, ask for approval in #secrets, then re-trigger the sync and report the result."* | Step | Type | Configuration | | -------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------- | | **Trigger** | Infisical Secret Sync Failed | All projects | | **Action 1** | Wait for Slack Approval | Channel: `#secrets`, message includes `{{signal.sync_name}}` in `{{signal.project_name}}` | | **On Approved → Action 2** | Trigger Secret Sync | Sync: `{{signal.sync_id}}` | | **Action 3** | Slack Send Message | Channel: `#secrets`, message: `{{step_outputs.action-2.summary}}` | **Workflow Agent prompt:** *"When PostHog captures a frontend exception, generate an AI summary of the user's session including what they were doing before the error, and send the summary with the replay URL to #product-alerts in Slack."* | Step | Type | Configuration | | ------------ | ------------------------------- | ---------------------------------------------------------------------------------------- | | **Trigger** | PostHog Session Error/Exception | All events | | **Action 1** | Get Session Summary | Session: `{{signal.session_id}}` | | **Action 2** | Slack Send Message | Channel: `#product-alerts`, message: `{{step_outputs.action-1.summary}}` with replay URL | ### Terraform Run Needs Attention → Plan Summary → Slack Approval → Apply **Workflow Agent prompt:** *"When a Terraform Cloud run on a production workspace finishes planning and needs confirmation, fetch the planned changes, ask for approval in #infra-approvals, and apply the run if approved or discard it if rejected."* | Step | Type | Configuration | | --------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------- | | **Trigger** | Terraform Run Needs Attention | Workspaces: `prod-*` | | **Action 1** | Get Run | Workspace: `{{signal.workspace}}`, Run: `{{signal.run_id}}` | | **Action 2** | Wait for Slack Approval | Channel: `#infra-approvals`, message includes `{{step_outputs.action-1.summary}}` and `{{signal.run_url}}` | | **On Approved** | Apply Run | Workspace: `{{signal.workspace}}`, Run: `{{signal.run_id}}` | | **On Rejected** | Discard Run | Workspace: `{{signal.workspace}}`, Run: `{{signal.run_id}}` | ### Pulumi Update Failure → Investigation → Slack Approval → Retry **Workflow Agent prompt:** *"When a Pulumi Cloud update fails on a production stack, investigate the failure, post the findings to #infra-approvals in Slack and ask for approval, then retry the update if approved."* | Step | Type | Configuration | | --------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------- | | **Trigger** | Pulumi Update Failed | Stacks: `prod-*` | | **Action 1** | Investigate Pulumi | Query: why did the update fail?, Stack: `{{signal.stack}}` | | **Action 2** | Wait for Slack Approval | Channel: `#infra-approvals`, message includes `{{step_outputs.action-1.summary}}` and `{{signal.update_url}}` | | **On Approved** | Run Deployment | Stack: `{{signal.stack}}`, Operation: `update` | | **Action 4** | Wait for Deployment | Stack: `{{signal.stack}}`, Deployment: `{{step_outputs.action-3.deployment_id}}` | ### Cloud Cost Anomaly → AI Analysis → Slack Report **Workflow Agent prompt:** *"When an AWS cost anomaly is detected, query the cost breakdown for the affected account over the last 7 days, run an AI cost analysis to identify the root cause and recommend optimizations, and post the analysis to #finops in Slack."* | Step | Type | Configuration | | ------------ | --------------------- | ---------------------------------------------------------------- | | **Trigger** | Cost Anomaly Detected | All AWS accounts | | **Action 1** | Query Cost Explorer | Account: `{{signal.aws_account_id}}`, time range: 7 days | | **Action 2** | AI Cost Analysis | Type: anomaly | | **Action 3** | Slack Send Message | Channel: `#finops`, message: `{{step_outputs.action-2.summary}}` | ### Weekly Schedule → Cost Report → Savings Recommendations → Slack **Workflow Agent prompt:** *"Every Monday at 9am UTC, query last week's AWS costs grouped by service, get savings plans recommendations, run an AI cost analysis, and post an executive summary to #finops in Slack."* | Step | Type | Configuration | | ------------ | --------------------------------- | -------------------------------------------------------------------------------- | | **Trigger** | Recurring Schedule | Weekly, Monday, 09:00 UTC | | **Action 1** | Query Cost Explorer | Time range: 7 days, group by: Service | | **Action 2** | Get Savings Plans Recommendations | Term: 1 year, payment: no upfront | | **Action 3** | AI Cost Analysis | Type: summary | | **Action 4** | Slack Send Message | Channel: `#finops`, message: `{{step_outputs.action-3.formatted_slack_message}}` | ### Idle Resource Scan → Approval → Cleanup → Slack **Workflow Agent prompt:** *"Every day at 8am UTC, scan for idle resources (unattached EBS volumes and unassociated Elastic IPs), ask for approval in #finops, and if approved delete the unattached volumes (snapshot first) and release the idle Elastic IPs."* | Step | Type | Configuration | | -------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Trigger** | Recurring Schedule | Daily, 08:00 UTC | | **Action 1** | Find Idle Resources | Resource types: EBS volumes, Elastic IPs; all enabled regions | | **Action 2** | Wait for Slack Approval | Channel: `#finops`, message includes `{{step_outputs.action-1.idle_count}}` idle resources, est. savings `{{step_outputs.action-1.estimated_monthly_savings}}` | | **On Approved → Action 3** | Delete Unattached EBS Volumes | Volume IDs: `{{step_outputs.action-1.idle_volume_ids}}`, snapshot first | | **On Approved → Action 4** | Release Elastic IPs | Allocation IDs: `{{step_outputs.action-1.idle_eip_allocation_ids}}` | | **Action 5** | Slack Send Message | Channel: `#finops`, message: cleanup summary | ### Forecast Overrun → Compare Periods → Rightsizing → Slack **Workflow Agent prompt:** *"When the AWS cost forecast projects a budget overrun, compare this month's spend to last month to find the biggest movers, get rightsizing recommendations, and post the analysis to #finops in Slack."* | Step | Type | Configuration | | ------------ | ------------------------------- | ---------------------------------------------------------------- | | **Trigger** | Forecast Exceeds Budget | All AWS accounts | | **Action 1** | Compare Cost Periods | This month vs last month, group by: Service | | **Action 2** | Get Rightsizing Recommendations | All recommendation types | | **Action 3** | AI Cost Analysis | Type: optimization | | **Action 4** | Slack Send Message | Channel: `#finops`, message: `{{step_outputs.action-3.summary}}` | *** ## Next Steps * [Custom Integrations](/workflows/custom-integrations) — Build custom HTTP action blocks and webhook triggers for services not covered by built-in integrations * [Workflow Integrations Setup](/workflows/setup-integrations) — Connect the integrations your workflows need * [Workflow Observability](/workflows/observability) — Monitor execution metrics and debug failed runs # Custom Integrations Source: https://docs.usekestrel.ai/workflows/custom-integrations Create custom HTTP action blocks and webhook trigger blocks Kestrel's built-in integrations cover the most common tools in the cloud operations stack. For everything else — internal APIs, niche SaaS products, or any service with an HTTP API or webhooks — you can build custom action blocks and webhook trigger blocks. Custom blocks are created from the workflow builder sidebar by clicking **Create Custom Workflow Block**. *** ## Custom Action Blocks (HTTP) A custom HTTP action block lets you call any API endpoint as a workflow step. Define the request template once, then reuse it across workflows with dynamic parameters. ### Quickest Method: Paste a cURL Command The fastest way to create a custom action block is to paste a cURL command. Kestrel automatically extracts the method, URL, headers, body, and authentication. Credentials are detected and stored securely. In the workflow builder sidebar, click **Create Custom Workflow Block** and select **Action Block**. Paste a cURL command from the API you want to call. Kestrel's AI agent automatically extracts: * HTTP method and URL * Headers (including authentication headers) * Request body * Authentication type (API key, bearer token, or basic auth) * Response mapping suggestions * Parameter definitions for dynamic values Review the auto-detected configuration. The AI agent chooses the right authentication method and suggests which values should be parameterized. Adjust names, descriptions, and defaults as needed. Click **Create** to save the custom action block. It appears in the workflow sidebar under your custom integration group and can be used in any workflow. You can also click **Skip — build from scratch** to manually configure every field if you prefer full control. ### Manual Configuration (Build from Scratch) If you prefer to configure everything manually: **Request Configuration:** | Field | Description | Example | | ----------- | ------------------------------------------ | -------------------------------------------------------- | | **Method** | HTTP method | `POST`, `GET`, `PUT`, `PATCH`, `DELETE` | | **URL** | Endpoint URL (supports template variables) | `https://api.example.com/v1/alerts/{{trigger.alert_id}}` | | **Headers** | Request headers as key-value pairs | `Content-Type: application/json` | | **Body** | Request body (JSON) | `{"message": "{{rca_result.root_cause}}"}` | **Authentication:** Send an API key as a header. * **Header Name**: The header to set (e.g., `X-API-Key`) * **API Key**: The key value (stored encrypted) Send a bearer token in the `Authorization` header. * **Token**: The bearer token value (stored encrypted) Send HTTP Basic Authentication credentials. * **Username**: The username * **Password**: The password (stored encrypted) Credentials are stored encrypted and never exposed in workflow logs or execution history. **Response Mapping:** Define output variables by mapping JSON paths from the API response: | Output Variable Name | JSON Path | Description | | -------------------- | ----------- | ----------------------- | | `status` | `$.status` | Response status field | | `result_id` | `$.data.id` | ID from nested response | | `message` | `$.message` | Response message | Mapped variables become available as `{{step_outputs..}}` in downstream steps. **Parameters:** Define named parameters that appear as configurable fields when the custom action is added to a workflow canvas. | Field | Type | Description | | --------------- | ------ | ----------------------------------------------- | | **Name** | string | Parameter identifier used in URL/body templates | | **Label** | string | Display label in the UI | | **Description** | string | Help text shown to the user | | **Default** | any | Default value (optional) | *** ## Custom Trigger Blocks (Webhooks) A custom webhook trigger lets you start a workflow from any external service that can send HTTP POST requests. ### Quickest Method: Paste an Example Payload The fastest way to create a webhook trigger is to paste a sample JSON payload from the external service. Kestrel automatically detects the event type field and extracts useful fields for your workflow. In the workflow builder sidebar, click **Create Custom Workflow Block** and select **Trigger Block**. Paste a sample JSON payload from the external service. Kestrel automatically: * Detects the event type field (e.g., `type`, `event`, `event_type`) * Extracts useful fields as template variables * Generates a payload mapping with sensible variable names * Suggests a name and description for the trigger block Review the auto-detected field mapping. Rename variables, adjust which fields are extracted, and configure event type filtering if needed. Enable HMAC signature verification and enter the signing secret from the external service. Click **Create** to save the trigger block. Copy the generated webhook URL and configure it in the external service as the webhook destination. You can also click **Skip — build from scratch** to manually define every field mapping and configuration option. ### Webhook URL Each custom webhook trigger gets a unique URL: ``` https://platform.usekestrel.ai/api/webhooks/custom/ ``` Configure this URL in the external service as the webhook destination. ### HMAC Signature Verification Enable HMAC verification to ensure webhook payloads are authentic: 1. Enter the **Signing Secret** provided by the external service 2. Specify the **Signature Header** name (e.g., `X-Hub-Signature-256`, `Stripe-Signature`) Kestrel validates the HMAC signature on every incoming request and rejects payloads that don't match. Without HMAC verification, any HTTP client that knows the webhook URL can trigger the workflow. Enable verification for production workflows. ### Event Type Filtering If the external service sends different event types to the same webhook, configure the trigger to only fire for specific event types. The event type path is auto-detected when you paste a sample payload, or you can configure it manually. ### Payload Field Mapping When you paste an example payload, Kestrel parses the JSON and generates a field map: ```json theme={null} { "type": "email.delivered", "data": { "email_id": "em_abc123", "to": "user@example.com", "subject": "Your invoice" } } ``` Auto-detected variables become available as `{{signal.}}` in the workflow: | Variable | Mapped From | Description | | ----------------------- | --------------- | ----------------------- | | `{{signal.event_type}}` | `type` | The webhook event type | | `{{signal.email_id}}` | `data.email_id` | Extracted email ID | | `{{signal.to}}` | `data.to` | Recipient email address | | `{{signal.subject}}` | `data.subject` | Email subject line | *** ## Examples ### Resend Email Events **Workflow Agent prompt:** *"When a Resend email bounces, send a notification to #email-ops in Slack with the bounce details and create a Jira ticket for the email team to investigate."* **Trigger**: Custom webhook trigger with Resend's `email.bounced` event type. **Example payload**: ```json theme={null} { "type": "email.bounced", "data": { "email_id": "em_abc123", "to": "user@example.com", "subject": "Your invoice", "bounce_type": "hard" } } ``` **Workflow**: On bounce → Slack notification to `#email-ops` with bounce details → Jira ticket for investigation. ### Stripe Payment Webhooks **Workflow Agent prompt:** *"When a Stripe payment fails, investigate the customer's recent activity, send an alert to #payments in Slack with the failure details, and create a PagerDuty alert if the amount is over \$1000."* **Trigger**: Custom webhook filtered to `payment_intent.payment_failed`. HMAC enabled with Stripe's webhook signing secret, header `Stripe-Signature`. **Workflow**: On failed payment → Kestrel Investigation with customer context → Slack alert to `#payments` → conditional PagerDuty alert. ### Internal Service Events **Workflow Agent prompt:** *"When our deployment service sends a deploy event, generate the K8s manifests based on the deployment spec, create a GitOps PR in acme/k8s-manifests, and trigger an ArgoCD sync after the PR is merged."* **Trigger**: Custom webhook with your application's deployment event schema. **Workflow**: On deployment event → Generate K8s manifests → Create GitOps PR → Wait for PR merge → ArgoCD sync. *** ## Next Steps * [Create Workflows](/workflows/create-workflows) — Build workflows with built-in and custom integrations * [Workflow Integrations Setup](/workflows/setup-integrations) — Connect built-in integrations # MCP Source: https://docs.usekestrel.ai/workflows/mcp Manage Kestrel workflows through the Model Context Protocol for AI agent integration The Kestrel MCP server exposes workflow management as tools that AI coding agents (Cursor, Claude Desktop, OpenAI Codex, etc.) can call directly. This lets you create, modify, and monitor workflows through conversational AI. ## Prerequisites Install the [Kestrel CLI](/workflows/cli) and authenticate: ```bash theme={null} brew install KestrelAI/tap/kestrel kestrel auth kestrel_sk_... ``` The MCP server uses your CLI credentials stored at `~/.kestrel/config.json`. ## Setup The MCP server runs as a subcommand of the CLI. Configure your AI tool to run `kestrel mcp`. ### Cursor Add to `~/.cursor/mcp.json`: ```json theme={null} { "mcpServers": { "kestrel": { "command": "kestrel", "args": ["mcp"] } } } ``` ### Claude Desktop Add to `~/Library/Application Support/Claude/claude_desktop_config.json`: ```json theme={null} { "mcpServers": { "kestrel": { "command": "kestrel", "args": ["mcp"] } } } ``` ### OpenAI Codex Add via the Codex CLI: ```bash theme={null} codex mcp add kestrel -- kestrel mcp ``` Or add to `~/.codex/config.toml`: ```toml theme={null} [mcp_servers.kestrel] command = "kestrel" args = ["mcp"] ``` If `kestrel` isn't on your PATH, use the full path (e.g., `/usr/local/bin/kestrel`). ## Available Tools ### Workflow Management | Tool | Description | | -------------------- | ------------------------------------------------------------ | | `list_workflows` | List all workflows, optionally filtered by status | | `get_workflow` | Get workflow details including definition and trigger config | | `create_workflow` | Create a new workflow from a definition | | `update_workflow` | Update a workflow's name, description, or definition | | `delete_workflow` | Delete a workflow | | `activate_workflow` | Activate a workflow | | `pause_workflow` | Pause an active workflow | | `duplicate_workflow` | Copy a workflow as a new draft | ### Workflow Generation | Tool | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `generate_workflow` | Generate a workflow from natural language | | `request_workflow` | Submit a request to trigger a matching workflow. If nothing matches, returns `pending_confirmation` with a `request_id` | | `confirm_workflow_request` | Resolve a `pending_confirmation` request: confirm to send it to the platform team, or dismiss it. The agent asks you before calling this | | `list_suggested_workflows` | Get AI-suggested workflows | ### Execution Management | Tool | Description | | -------------------- | -------------------------------------------- | | `list_executions` | List executions for a workflow | | `get_execution` | Get execution details including step results | | `cancel_execution` | Cancel a running execution | | `get_workflow_stats` | Aggregate execution statistics | ### Approval Management | Tool | Description | | ------------------------ | -------------------------------------------- | | `list_pending_approvals` | List pending approval gates | | `approve_step` | Approve a gate (with optional justification) | | `reject_step` | Reject a gate | ### Request Management | Tool | Description | | -------------------------- | -------------------------------- | | `list_unmatched_requests` | List unmatched workflow requests | | `approve_workflow_request` | Approve a request | | `reject_workflow_request` | Reject a request | ### Catalog and Integrations | Tool | Description | | ------------------------- | ------------------------------------------- | | `get_catalog` | Available signal triggers and action blocks | | `get_integrations_status` | Integration connection status | ### Integration Management | Tool | Description | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------- | | `list_integrations` | List every integration with its type, required credential fields, and connection status | | `connect_integration` | Connect a token or knowledge-source integration, or get the browser URL for OAuth integrations (GitHub, GitLab, Slack) | | `test_integration` | Test a connected integration's credentials | | `disconnect_integration` | Disconnect a token integration or knowledge source | Kubernetes, AWS, and OCI use multi-step flows and must be connected with the [CLI](/workflows/cli#integration-commands) (`kestrel integrations connect `) instead of MCP tools. For credential safety, prefer the CLI's interactive hidden prompts for secrets rather than pasting tokens into an agent chat. ## Example Conversations **Creating a workflow:** > "Create a workflow that triggers when a PagerDuty alert fires, runs K8s RCA, and posts results to #incidents on Slack" The AI agent calls `generate_workflow`, reviews the result, then `create_workflow` and `activate_workflow`. **Checking executions:** > "Show me the last 5 failed workflow executions and what went wrong" The agent calls `list_workflows`, `list_executions`, then `get_execution` for failed ones. **Approving a gate:** > "Are there any pending approvals? Approve the production deployment one" The agent calls `list_pending_approvals`, identifies the right one, and calls `approve_step`. **Connecting an integration:** > "Connect our PagerDuty account to Kestrel" The agent calls `list_integrations` to find the required credential fields, asks you for the values, then calls `connect_integration` and verifies with `test_integration`. **Requesting a change with no matching workflow:** > "Request a Redis cluster for the cache layer" The agent calls `request_workflow`. When no workflow matches, it gets back `pending_confirmation` and asks whether you want to send the request to your platform team; based on your answer, it calls `confirm_workflow_request` to file or dismiss it. # Workflow Observability Source: https://docs.usekestrel.ai/workflows/observability Monitor workflow execution metrics, performance, and health The Workflow Observability dashboard gives you visibility into how your workflows are performing — execution counts, success rates, integration usage, step durations, and full execution history with step-by-step output inspection. Navigate to **Workflows → Observability** in the Kestrel dashboard. ## Dashboard Overview The top of the dashboard shows aggregate metrics across all workflows: | Metric | Description | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Total Executions** | Number of workflow runs in the selected time range | | **Success Rate** | Percentage of executions that completed without errors | | **Active Workflows** | Number of enabled workflows | | **Trigger Sources** | Breakdown of executions by trigger type (K8s, Cloud, Slack, PagerDuty, PostHog, Vercel, Railway, Fly.io, Nebius, Jenkins, CircleCI, Terraform Cloud, Pulumi Cloud, Vault, Infisical, Webhook) | Use the date range picker in the top-right corner to adjust the time window. Defaults to the last 7 days. *** ## Integration Usage The integration usage chart shows which integrations are used most frequently across all workflow executions. Each bar represents an integration (Slack, GitHub, Jira, Linear, Kestrel RCA, etc.) with the total number of action executions. * Paginated — use the navigation controls to browse through all integrations * Sorted by usage count (highest first) * Hover over a bar to see the exact count and percentage of total This chart helps identify which integrations are central to your automation and where to focus reliability efforts. *** ## Execution Duration by Workflow A paginated bar chart showing the average execution duration for each workflow. Identifies slow workflows that may need optimization. * Each bar represents one workflow, labeled by name * Duration is the average end-to-end time from trigger to final step completion * Color-coded: green for fast (under 30s), yellow for moderate (30s–2min), red for slow (over 2min) * Paginated for organizations with many workflows If a workflow's average duration is high, check the Slowest Steps analysis to find the bottleneck. *** ## Slowest Steps A ranked list of the slowest individual steps across all workflows, based on average execution time. Each entry shows: | Field | Description | | ---------------- | ------------------------------------------------------------- | | **Step Name** | The action name and type (e.g., "RCA Analysis — Kestrel RCA") | | **Workflow** | The parent workflow | | **Avg Duration** | Average execution time | | **P95 Duration** | 95th percentile execution time | | **Executions** | Total number of times this step has run | Use this to identify steps that slow down your workflows — whether it's a slow external API, a long-running AI analysis, or a blocking approval gate. *** ## Execution History The execution history table lists every workflow run with filtering and search. ### Filters | Filter | Options | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Workflow** | Select a specific workflow or view all | | **Status** | Success, Failed, Running, Pending Approval, Timed Out | | **Trigger Source** | K8s Incident, Cloud Signal, Slack, PagerDuty, PostHog, Vercel, Railway, Fly.io, Nebius, Jenkins, CircleCI, Terraform Cloud, Pulumi Cloud, Vault, Infisical, Webhook | | **Date Range** | Custom start and end dates | ### Table Columns | Column | Description | | ------------ | -------------------------------------------------- | | **Workflow** | Workflow name | | **Trigger** | What triggered the execution (with source details) | | **Status** | Execution status with color-coded badge | | **Duration** | End-to-end execution time | | **Steps** | Number of completed steps / total steps | | **Started** | Timestamp of execution start | Click any row to open the execution detail view. *** ## Execution Detail View The detail view shows the full step-by-step breakdown of a single workflow execution. Each step displays: * **Status**: Success, failed, skipped, or pending * **Duration**: How long the step took * **Input**: The resolved template variables passed to the step * **Output**: The full output returned by the step (API responses, AI analysis results, created resource IDs) * **Error**: If the step failed, the error message and details Step outputs are retained for 30 days. After that, execution metadata (status, duration, timestamps) is kept but outputs are purged. ### Debugging Failed Executions When a workflow fails: 1. Open the execution detail view 2. Find the step with a **Failed** status badge 3. Expand the step to see the error message 4. Check the **Input** section to verify the template variables resolved correctly 5. Check the **Output** section for the raw error response from the integration Common failure causes: * **Authentication expired** — Reconnect the integration on the Integrations page * **Rate limiting** — The external API returned a 429. Increase the cooldown period or reduce workflow frequency * **Invalid template variable** — A referenced variable was null or malformed. Check the upstream step's output * **Approval timeout** — An approval gate was not responded to within the configured timeout *** ## Next Steps * [Create Workflows](/workflows/create-workflows) — Build new workflows or optimize existing ones * [Custom Integrations](/workflows/custom-integrations) — Add HTTP actions and webhook triggers # Python SDK Source: https://docs.usekestrel.ai/workflows/sdk Programmatically create, manage, and monitor workflows with the Kestrel Python SDK The Kestrel Python SDK provides a typed, fluent API for building and managing workflows programmatically. It supports both synchronous and asynchronous usage with API key authentication. ## Installation ```bash theme={null} pip install kestrel-workflows ``` ## Authentication Create an API key in the Kestrel platform under **Workflows > API Keys**, then use it to authenticate: ```python theme={null} from kestrel import KestrelClient client = KestrelClient(api_key="kestrel_sk_...") ``` For async usage: ```python theme={null} from kestrel import AsyncKestrelClient async with AsyncKestrelClient(api_key="kestrel_sk_...") as client: workflows = await client.workflows.list() ``` If you're logged in via the [CLI](/workflows/cli), the SDK can reuse your session: ```python theme={null} client = KestrelClient.from_config() ``` ## Workflow Builder Build workflows using typed classes with full IDE autocompletion: ```python theme={null} from kestrel import KestrelClient from kestrel.workflows import Workflow, Trigger, Action, Condition, Approval, PollUntil wf = ( Workflow("Pod Crash RCA + Jira") .description("Run RCA on pod crash, create Jira ticket") .trigger( Trigger.k8s_pod_status() .reasons("CrashLoopBackOff") .cluster("my-cluster-id") .namespace("production") ) .cooldown(hours=24) .then(Action.kestrel_trigger_rca().label("Run RCA")) .then(Action.jira_create_ticket() .project("KAN") .title("{{incident.title}}") .priority("High") .assignee("Raman Varma") .label("Create Jira Ticket") ) .then(Action.confluence_publish_rca() .space_key("SD") .title("RCA: {{incident.title}}") .label("Publish RCA") ) .alert_on_failure(channel="alerts-channel-id") ) client = KestrelClient(api_key="kestrel_sk_...") created = client.workflows.deploy(wf, activate=True) print(f"Deployed: {created.id} ({created.status})") ``` ### Triggers Create triggers for different signal sources: ```python theme={null} # Kubernetes — Pod Status Trigger.k8s_pod_status().reasons("CrashLoopBackOff") Trigger.k8s_pod_status().reasons("ImagePullBackOff", "ErrImagePull") Trigger.k8s_pod_status().reasons("OOMKilled") # uses terminated_reasons Trigger.k8s_pod_status().reasons("Error") # pod exits non-zero Trigger.k8s_pod_status().restart_threshold(5) # pods restarting excessively # Kubernetes — Workload Rollout Trigger.k8s_rollout_status().reasons("ProgressDeadlineExceeded").resource_kinds("Deployment") Trigger.k8s_rollout_status().reasons("ReplicasNotReady").resource_kinds("StatefulSet") Trigger.k8s_rollout_status().reasons("PodsUnavailable").resource_kinds("DaemonSet") # Kubernetes — Node Conditions Trigger.k8s_node_condition().conditions("NotReady") Trigger.k8s_node_condition().conditions("MemoryPressure") Trigger.k8s_node_condition().conditions("DiskPressure") # Kubernetes — Catch-all Trigger.k8s_any() # AWS — CloudTrail (infrastructure events) Trigger.aws_cloudtrail().service_names("iam", "sts") Trigger.aws_cloudtrail().service_names("s3") Trigger.aws_cloudtrail().service_names("ec2") Trigger.aws_cloudtrail().service_names("rds") Trigger.aws_cloudtrail().service_names("lambda") Trigger.aws_cloudtrail().service_names("dynamodb") Trigger.aws_cloudtrail().service_names("kms") Trigger.aws_cloudtrail().service_names("secretsmanager") Trigger.aws_cloudtrail().incident_types("RootAccountActivity") # AWS — Application Errors Trigger.aws_cloudwatch_log().incident_types("ApplicationLogErrors").service_names("ec2") # AWS — CloudWatch Alarms Trigger.aws_cloudwatch_metric() # AWS — Security & Compliance Trigger.aws_security_hub().severities("CRITICAL", "HIGH") Trigger.aws_config_rule() Trigger.aws_service_health() # AWS — Cost & Budget Trigger.aws_cost_anomaly() Trigger.aws_budget_alert() Trigger.aws_forecast_overrun() # forecast projects a budget overrun Trigger.aws_spend_spike() # day-over-day spend spike Trigger.aws_idle_resource() # daily idle-resource scan hit # AWS — Catch-all Trigger.aws_any() # Schedule (recurring workflows — no external event needed) Trigger.schedule_hourly() # fires every hour Trigger.schedule_daily("09:00") # daily at 09:00 UTC Trigger.schedule_weekly(day_of_week=1, time_utc="09:00") # Mondays 09:00 UTC (0=Sunday) Trigger.schedule_monthly(day_of_month=1, time_utc="06:00") # 1st of month 06:00 UTC # PagerDuty Trigger.pagerduty_triggered() Trigger.pagerduty_acknowledged() Trigger.pagerduty_resolved() Trigger.pagerduty_any() Trigger.pagerduty_triggered().urgency("high") # GitHub Trigger.github_pr_opened() Trigger.github_pr_merged() Trigger.github_pr_approved() Trigger.github_action_completed() Trigger.github_action_failed() Trigger.github_push() # New commits pushed to branch # PostHog Trigger.posthog_exception() Trigger.posthog_rage_click() Trigger.posthog_log_alert() Trigger.posthog_console_error() Trigger.posthog_any() # Vercel Trigger.vercel_deployment_failed() Trigger.vercel_deployment_ready() Trigger.vercel_deployment_created() Trigger.vercel_checks_failed() Trigger.vercel_rollback() Trigger.vercel_firewall_attack() Trigger.vercel_error_anomaly() Trigger.vercel_usage_anomaly() Trigger.vercel_domain_issue() # Railway Trigger.railway_deployment_failed() Trigger.railway_deployment_crashed() Trigger.railway_deployment_succeeded() Trigger.railway_volume_alert() Trigger.railway_resource_alert() Trigger.railway_any() # Fly.io (poll-based) Trigger.flyio_machine_crashed() Trigger.flyio_machine_stopped() Trigger.flyio_machine_started() Trigger.flyio_app_down() Trigger.flyio_any() # Nebius AI Cloud (poll-based) Trigger.nebius_gpu_error() Trigger.nebius_maintenance_scheduled() Trigger.nebius_node_not_ready() Trigger.nebius_instance_stopped() Trigger.nebius_any() # Jenkins (webhook via notification plugin or post-build step) Trigger.jenkins_build_failed() Trigger.jenkins_build_unstable() # e.g. test failures Trigger.jenkins_build_succeeded() Trigger.jenkins_build_completed() # any result Trigger.jenkins_build_started() Trigger.jenkins_any() # CircleCI (native outbound webhooks, HMAC-signed) Trigger.circleci_workflow_failed() # failed or error Trigger.circleci_workflow_succeeded() Trigger.circleci_workflow_completed() # any status Trigger.circleci_job_failed() Trigger.circleci_any() # Terraform Cloud (webhook via workspace notification configurations) Trigger.terraform_run_created() Trigger.terraform_run_needs_attention() # plan finished, awaiting confirmation Trigger.terraform_run_completed() Trigger.terraform_run_errored() Trigger.terraform_drift_detected() # health assessment found drift Trigger.terraform_check_failed() Trigger.terraform_any() # Pulumi Cloud (webhook via organization or stack webhooks) Trigger.pulumi_update_succeeded() Trigger.pulumi_update_failed() Trigger.pulumi_preview_failed() Trigger.pulumi_destroy_succeeded() Trigger.pulumi_deployment_started() Trigger.pulumi_deployment_succeeded() Trigger.pulumi_deployment_failed() Trigger.pulumi_drift_detected() # scheduled drift run found drift Trigger.pulumi_policy_violation() # mandatory policy pack blocked an update Trigger.pulumi_stack_created() Trigger.pulumi_stack_deleted() Trigger.pulumi_any() # HashiCorp Vault (poll-based; metadata only — secret values are never read) Trigger.vault_sealed() Trigger.vault_unsealed() Trigger.vault_health_degraded() Trigger.vault_secret_version_created() # KV v2 metadata diff Trigger.vault_secret_stale() # age threshold via .vault_secret_max_age_days() Trigger.vault_policy_created() Trigger.vault_policy_deleted() Trigger.vault_auth_method_enabled() Trigger.vault_auth_method_disabled() Trigger.vault_any() # Infisical (poll-based via audit log; secret values are never read) Trigger.infisical_secret_created() Trigger.infisical_secret_updated() Trigger.infisical_secret_deleted() Trigger.infisical_approval_requested() Trigger.infisical_secret_sync_failed() Trigger.infisical_identity_created() Trigger.infisical_any() # Developer Requests (on-demand via Slack /kestrel-workflow, Kestrel Platform UI, or CLI) Trigger.request_kubernetes() # Kubernetes operations Trigger.request_cloud() # AWS / Cloud operations Trigger.request_cloudflare() # Cloudflare operations Trigger.request_pagerduty() # PagerDuty operations Trigger.request_datadog() # Datadog operations Trigger.request_argocd() # ArgoCD operations Trigger.request_github() # GitHub operations Trigger.request_gitlab() # GitLab operations Trigger.request_helm() # Helm operations Trigger.request_vercel() # Vercel operations Trigger.request_jenkins() # Jenkins operations Trigger.request_circleci() # CircleCI operations Trigger.request_terraform() # Terraform Cloud operations Trigger.request_pulumi() # Pulumi Cloud operations Trigger.request_vault() # HashiCorp Vault operations Trigger.request_infisical() # Infisical operations Trigger.request_general() # General (catch-all) ``` Each request trigger can be scoped further: ```python theme={null} from kestrel.workflows import Trigger # Kubernetes requests — only create operations from Slack Trigger.request_kubernetes().config( request_sources=["slack"], request_types=["create"], ) # Cloud requests — only from Platform UI and CLI Trigger.request_cloud().config( request_sources=["platform", "cli"], ) ``` ### Submitting Requests Programmatically ```python theme={null} # Submit a request (like /kestrel-workflow in Slack) result = client.workflows.request("create a configmap for my-app in production") # If clarification is needed: if result.status == "awaiting_params": reply = client.workflows.request_reply(result.id, "use the prod-cluster, namespace=payments") # If no workflow matches, the request is held as pending_confirmation. # Confirm to send it to the platform team, or dismiss it: if result.status == "pending_confirmation": client.workflows.request_confirm(result.id, True) # file for the platform team # client.workflows.request_confirm(result.id, False) # dismiss ``` The async client mirrors this via `client.requests.confirm(request_id, confirm)`. # Custom Webhook Trigger.custom\_webhook("my-event-type") # Datadog Monitor Trigger.datadog\_monitor\_alert() Trigger.datadog\_monitor\_warn() Trigger.datadog\_monitor\_recovered() Trigger.datadog\_monitor\_no\_data() Trigger.datadog\_monitor\_any() Trigger.datadog\_monitor\_alert().monitor\_names("High CPU").datadog\_tags("env:production") ```` All triggers support filter chaining: ```python trigger = ( Trigger.k8s_pod_status() .reasons("CrashLoopBackOff") .cluster("cluster-id-1", "cluster-id-2") .namespace("production", "staging") .workload("api-server", "worker") ) ```` Fly.io triggers are poll-based — set the apps, regions, and poll cadence to control how often Kestrel checks the Fly Machines API: ```python theme={null} trigger = ( Trigger.flyio_machine_crashed() .fly_apps("my-app", "another-app") # omit for all apps the token can see .fly_regions("iad", "sjc") .fly_poll_interval("1m") # 1m | 5m | 15m | 30m ) ``` Nebius triggers are also poll-based — scope to projects/clusters and set the poll cadence to control how often Kestrel checks the Nebius compute and Managed Kubernetes APIs: ```python theme={null} trigger = ( Trigger.nebius_gpu_error() .nebius_projects("project-abc") # omit for all projects the credentials can see .nebius_clusters("cluster-xyz") # optional: restrict to specific clusters .nebius_poll_interval("5m") # 1m | 5m | 15m | 30m ) ``` Jenkins and CircleCI triggers are webhook-based — scope to jobs/projects and optionally statuses: ```python theme={null} trigger = ( Trigger.jenkins_build_failed() .jenkins_jobs("platform/deploy-api") # full path for folder-nested jobs; omit for all .jenkins_build_statuses("FAILURE") # optional: SUCCESS | FAILURE | UNSTABLE | ABORTED ) trigger = ( Trigger.circleci_workflow_failed() .circleci_projects("gh/acme/api") # project slugs; omit for all followed projects .circleci_branches("main") # optional: filter by branch .circleci_statuses("failed", "error") # optional: filter by status ) ``` Terraform Cloud triggers are webhook-based — scope to workspaces and optionally run statuses: ```python theme={null} trigger = ( Trigger.terraform_run_errored() .terraform_workspaces("prod-vpc", "prod-eks") # omit or use "*" for all workspaces .terraform_run_statuses("errored") # optional: filter by run status ) ``` Pulumi Cloud triggers are webhook-based — scope to stacks (project/stack) and optionally projects: ```python theme={null} trigger = ( Trigger.pulumi_update_failed() .pulumi_stacks("my-project/prod") # project/stack refs or bare stack names; omit or "*" for all .pulumi_projects("my-project") # optional: filter by Pulumi project name ) ``` Vault and Infisical triggers are poll-based (no outbound webhooks; only metadata and audit events are read — never secret values). Scope secret triggers and tune the poll cadence: ```python theme={null} trigger = ( Trigger.vault_secret_stale() .vault_mounts("secret/") # KV v2 mounts; omit or "*" for all .vault_secret_paths("app/prod") # path prefixes within the mount .vault_secret_max_age_days(60) # rotation age threshold (default 90) .vault_poll_interval("15m") # 1m | 5m | 15m | 30m ) trigger = ( Trigger.infisical_secret_updated() .infisical_projects("proj-id") # project IDs; omit or "*" for all .infisical_environments("prod") # environment slugs .infisical_secret_paths("/backend") # folder path prefixes .infisical_poll_interval("5m") # 1m | 5m | 15m | 30m ) ``` ### Actions Every integration has typed factory methods: ```python theme={null} # Kestrel — RCA & Investigation Action.kestrel_trigger_rca() Action.kestrel_trigger_cloud_rca() Action.kestrel_generate_runbook() # Distill RCA + fixes into a reusable runbook (requires an upstream RCA step) Action.kestrel_investigate_k8s().query("Why is {{incident.workload_name}} crashing?") Action.kestrel_investigate_cloud().query("Investigate IAM event") Action.kestrel_find_causal_prs() Action.kestrel_find_service_deps().cluster_id("...").ns("default").workload_name("api") Action.kestrel_analyze_costs().analysis_type("anomaly") # Kestrel — K8s Operations Action.kestrel_apply_yaml_fix() Action.kestrel_restart_workload().workload_name("{{incident.workload_name}}").namespace("{{incident.namespace}}") Action.kestrel_scale_workload().workload_name("{{incident.workload_name}}").namespace("{{incident.namespace}}").replicas(0) Action.kubectl_execute("get pods -n production") # Run any kubectl command Action.generate_kubectl_command("Find pods in CrashLoopBackOff in production") # AI generates a kubectl command Action.kestrel_generate_k8s_manifest().resource_type("Deployment").query("Create a deployment for nginx") Action.kestrel_generate_helm_values().chart_name("nginx-ingress") Action.kestrel_apply_k8s_manifest() Action.kestrel_create_gitops_pr().repo("org/k8s-manifests") # Kestrel — Platform Engineering Action.kestrel_generate_cloud_resource().cloud_service("rds").output_format("terraform") Action.kestrel_execute_cloud_cli() Action.kestrel_create_iac_pr().repo("org/infra-repo") # Kestrel — Flow Control Action.kestrel_wait().duration(5).unit("minutes") PollUntil(Action.daytona_get_sandbox().sandbox_id("{{signal.sandbox_id}}"), Condition.equals("sandbox_state", "stopped")).every(seconds=60).timeout(minutes=30) # Slack Action.slack_send_message().channel("incidents").message("{{incident.title}}").include_rca(True) Action.slack_update_message() Action.slack_request_justification().channel("approvals") # Jira Action.jira_create_ticket().project("KAN").title("{{incident.title}}").priority("High") Action.jira_add_comment().ticket_key("KAN-123").comment("Update: {{rca_result.root_cause}}") Action.jira_transition_ticket().ticket_key("KAN-123").status("Done") # Linear Action.linear_create_issue().team("ENG").title("{{incident.title}}").priority("High") Action.linear_add_comment().issue_identifier("ENG-123").body("Update: {{rca_result.root_cause}}") Action.linear_update_issue().issue_identifier("ENG-123").status("Done") Action.linear_search_issues().query("{{incident.title}}").team("ENG") # GitHub Action.github_create_pr().repo("org/repo").title("Fix: {{incident.title}}") Action.github_create_issue().repo("org/repo").title("Bug: {{incident.title}}") Action.github_investigate_code().query("{{rca_result.root_cause}}").repo("org/repo") Action.github_analyze_push().repo("org/repo").head_sha("{{signal.commit_sha}}") # Determine affected services from push Action.github_generate_code_fix().context("{{rca_result.root_cause}}").repo("org/repo") Action.github_trigger_workflow().repo("org/repo").workflow_file("deploy.yml") Action.github_wait_for_run().repo("org/repo").workflow_file("deploy.yml") Action.github_get_run_status().repo("org/repo").workflow_file("ci.yml") Action.github_read_file().repo("org/repo").path("src/config.yaml") Action.github_search_code().repo("org/repo").query("handleError") Action.github_wait_pr_approval() Action.github_wait_pr_merge() Action.github_investigate_action_failure().repo("org/repo").run_id("{{signal.run_id}}") # GitLab Action.gitlab_create_mr().repo("org/repo").title("Fix: {{incident.title}}") Action.gitlab_create_issue().repo("org/repo").title("Bug: {{incident.title}}") Action.gitlab_trigger_pipeline().repo("org/repo").ref("main") Action.gitlab_wait_for_pipeline().repo("org/repo") Action.gitlab_get_pipeline_status().repo("org/repo") Action.gitlab_wait_mr_approval() Action.gitlab_wait_mr_merge() # Confluence Action.confluence_publish_rca().space_key("ENG") Action.confluence_publish_postmortem().space_key("ENG") Action.confluence_publish_runbook().space_key("ENG") Action.confluence_update_page().page_id("123456") # PagerDuty Action.pagerduty_create_alert().service("service-id").severity("critical") Action.pagerduty_acknowledge_alert() Action.pagerduty_resolve_alert() Action.pagerduty_add_note().note("RCA: {{rca_result.root_cause}}") Action.pagerduty_escalate() # Datadog Action.datadog_query_metrics().cluster_id("...").query("avg:kubernetes.cpu.usage.total{*}") Action.datadog_create_monitor().cluster_id("...").name("High CPU") Action.datadog_send_event().cluster_id("...").title("Incident resolved") Action.datadog_mute_monitor().cluster_id("...").monitor_id("12345") # ArgoCD Action.argocd_sync().cluster_id("...").app_name("my-app") Action.argocd_wait_sync().cluster_id("...").app_name("my-app") Action.argocd_get_status().cluster_id("...").app_name("my-app") Action.argocd_rollback().cluster_id("...").app_name("my-app") Action.argocd_find_app().cluster_id("...").workload_name("{{incident.workload_name}}").namespace("{{incident.namespace}}") # Helm Action.helm_upgrade().cluster_id("...").release_name("my-app").namespace("production").chart("oci://ghcr.io/org/app").values_set("image.tag={{signal.commit_sha}}") Action.helm_install().cluster_id("...").release_name("my-app").namespace("production").chart("bitnami/nginx").create_namespace(True) Action.helm_rollback().cluster_id("...").release_name("my-app").namespace("production") Action.helm_uninstall().cluster_id("...").release_name("my-app").namespace("production") Action.helm_status().cluster_id("...").release_name("my-app").namespace("production") # AWS Cost — insights (read-only) Action.aws_query_cost_explorer().aws_account("123456789012").time_range("30d") Action.aws_get_cost_anomalies().aws_account("123456789012") Action.aws_get_cost_forecast().aws_account("123456789012") Action.aws_get_budget_status().aws_account("123456789012") Action.aws_get_rightsizing_recommendations().aws_account("123456789012") Action.aws_get_savings_plans_recommendations().aws_account("123456789012") Action.aws_get_reservation_recommendations().aws_account("123456789012") Action.aws_get_commitment_utilization().aws_account("123456789012") Action.aws_compare_cost_periods().aws_account("123456789012") Action.aws_find_idle_resources().aws_account("123456789012") Action.aws_get_compute_optimizer_recommendations().aws_account("123456789012") Action.aws_get_trusted_advisor_cost_checks().aws_account("123456789012") # AWS Cost — remediation (destructive; place behind an Approval block. # Resources tagged kestrel:protected are always skipped.) Action.aws_stop_ec2_instances().aws_account("123456789012") \ .config("instance_ids", "{{step_outputs.scan.idle_instance_ids}}").config("region", "us-east-1") Action.aws_delete_unattached_ebs_volumes().aws_account("123456789012") \ .config("volume_ids", "{{step_outputs.scan.idle_volume_ids}}").config("region", "us-east-1") \ .config("snapshot_first", True) Action.aws_release_elastic_ips().aws_account("123456789012") \ .config("allocation_ids", "{{step_outputs.scan.idle_eip_allocation_ids}}").config("region", "us-east-1") Action.aws_delete_old_snapshots().aws_account("123456789012") \ .config("region", "us-east-1").config("age_days", 90) # PostHog Action.posthog_get_session_summary().session_ids("{{signal.session_id}}") Action.posthog_get_session_recording().session_id("{{signal.session_id}}") Action.posthog_query_events().query("SELECT event FROM events WHERE ...") Action.posthog_list_session_recordings().limit(10) Action.posthog_get_error_issue().issue_id("{{signal.properties.$exception_issue_id}}") # Vercel Action.vercel_get_deployment().deployment_id("{{signal.deployment_id}}") Action.vercel_get_build_logs().deployment_id("{{signal.deployment_id}}") Action.vercel_rollback().project_id("{{signal.project_id}}") Action.vercel_promote().deployment_id("{{signal.deployment_id}}") Action.vercel_list_deployments().project_id("{{signal.project_id}}") Action.vercel_investigate().query("Why did the build fail?") # Railway Action.railway_get_deployment().deployment_id("{{signal.deployment_id}}") Action.railway_get_deployment_logs().deployment_id("{{signal.deployment_id}}") Action.railway_rollback().config("service_id", "{{signal.service_id}}").config("environment_id", "{{signal.environment_id}}") Action.railway_redeploy().deployment_id("{{signal.deployment_id}}") Action.railway_restart().deployment_id("{{signal.deployment_id}}") Action.railway_list_deployments().config("service_id", "{{signal.service_id}}").config("status", "FAILED") Action.railway_set_variables().config("service_id", "{{signal.service_id}}").config("environment_id", "{{signal.environment_id}}").config("variables", {"LOG_LEVEL": "debug"}) Action.railway_investigate().config("query", "Why did the service crash?") # Fly.io (machine-scoped: require app_name + machine_id) Action.flyio_restart_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_start_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_stop_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_suspend_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_cordon_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_uncordon_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_get_machine().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_get_machine_events().app_name("{{signal.app_name}}").machine_id("{{signal.machine_id}}") Action.flyio_list_machines().app_name("{{signal.app_name}}").config("state", "started") Action.flyio_set_secrets().app_name("{{signal.app_name}}").config("secrets", {"API_KEY": "..."}) Action.flyio_investigate().config("query", "Why did the machine crash?") # Nebius AI Cloud (compute: project_id + instance_id; mk8s: cluster_id + node_group_id) Action.nebius_get_instance().project_id("{{signal.project_id}}").instance_id("{{signal.instance_id}}") Action.nebius_start_instance().project_id("{{signal.project_id}}").instance_id("{{signal.instance_id}}") Action.nebius_stop_instance().project_id("{{signal.project_id}}").instance_id("{{signal.instance_id}}") Action.nebius_restart_instance().project_id("{{signal.project_id}}").instance_id("{{signal.instance_id}}") Action.nebius_list_instances().project_id("{{signal.project_id}}") Action.nebius_list_clusters().project_id("{{signal.project_id}}") Action.nebius_list_node_groups().cluster_id("{{signal.cluster_id}}") Action.nebius_scale_node_group().cluster_id("{{signal.cluster_id}}").node_group_id("{{signal.node_group_id}}").size(3) Action.nebius_investigate().config("query", "Why did the GPU node error?") # Jenkins (job-scoped: .job(); build-scoped: .build_number()) Action.jenkins_trigger_build().job("platform/deploy-api").parameters("ENV=staging\nVERSION=1.2.3") Action.jenkins_wait_for_build().job("{{signal.job_name}}").build_number("{{signal.build_number}}").timeout_minutes(30) Action.jenkins_get_build_status().job("{{signal.job_name}}").build_number("{{signal.build_number}}") Action.jenkins_stop_build().job("{{signal.job_name}}").build_number("{{signal.build_number}}") # gate behind approval Action.jenkins_get_console_log().job("{{signal.job_name}}").build_number("{{signal.build_number}}").max_lines(200) Action.jenkins_investigate().query("Why did the nightly build fail?").job("{{signal.job_name}}") # CircleCI (project-scoped: .project_slug(); workflow-scoped: .workflow_id()) Action.circleci_trigger_pipeline().project_slug("gh/acme/api").branch("main").parameters("deploy_env=staging") Action.circleci_wait_for_pipeline().project_slug("{{signal.project_slug}}").pipeline_id("{{signal.pipeline_id}}").timeout_minutes(30) Action.circleci_get_workflow_status().project_slug("{{signal.project_slug}}").pipeline_id("{{signal.pipeline_id}}") Action.circleci_rerun_workflow().workflow_id("{{signal.workflow_id}}").from_failed(True) Action.circleci_cancel_workflow().workflow_id("{{signal.workflow_id}}") Action.circleci_approve_job().workflow_id("{{signal.workflow_id}}").job_name("hold-production") # gate behind approval Action.circleci_get_job_tests().project_slug("{{signal.project_slug}}").job_number("{{signal.job_number}}") Action.circleci_investigate().query("Why did the main-branch workflow fail?").project_slug("{{signal.project_slug}}") # Terraform Cloud (workspace-scoped: .workspace(); run-scoped: .run_id()) Action.terraform_list_workspaces() Action.terraform_get_workspace().workspace("{{signal.workspace}}") Action.terraform_lock_workspace().workspace("{{signal.workspace}}").reason("Incident freeze") Action.terraform_unlock_workspace().workspace("{{signal.workspace}}") Action.terraform_force_unlock_workspace().workspace("{{signal.workspace}}") # gate behind approval Action.terraform_list_runs().workspace("{{signal.workspace}}").limit(10) Action.terraform_get_run().workspace("{{signal.workspace}}").run_id("{{signal.run_id}}") Action.terraform_create_run().workspace("prod-vpc").run_message("Queued by Kestrel").auto_apply(False) Action.terraform_create_destroy_run().workspace("staging-sandbox") # gate behind approval Action.terraform_apply_run().workspace("{{signal.workspace}}").run_id("{{signal.run_id}}").comment("Approved via Kestrel") # gate behind approval Action.terraform_discard_run().workspace("{{signal.workspace}}").run_id("{{signal.run_id}}") Action.terraform_cancel_run().workspace("{{signal.workspace}}").run_id("{{signal.run_id}}") Action.terraform_wait_for_run().workspace("{{signal.workspace}}").run_id("{{signal.run_id}}").timeout_minutes(30) Action.terraform_get_state_outputs().workspace("prod-vpc") Action.terraform_list_variables().workspace("prod-vpc") Action.terraform_set_variable().workspace("prod-vpc").key("instance_count").value("3").category("terraform") # gate behind approval Action.terraform_get_drift().workspace("prod-vpc") Action.terraform_investigate().query("Why did the latest run fail?").workspace("{{signal.workspace}}") # Pulumi Cloud (stack-scoped: .stack() in project/stack format; deployment-scoped: .deployment_id()) Action.pulumi_list_stacks() Action.pulumi_get_stack().stack("{{signal.stack}}") Action.pulumi_list_updates().stack("{{signal.stack}}").limit(10) Action.pulumi_get_update().stack("{{signal.stack}}").update_version("{{signal.update_version}}") Action.pulumi_run_deployment().stack("my-project/prod").operation("update") # gate destroy/remediate-drift behind approval Action.pulumi_get_deployment().stack("{{signal.stack}}").deployment_id("{{signal.deployment_id}}") Action.pulumi_wait_for_deployment().stack("{{signal.stack}}").deployment_id("{{signal.deployment_id}}").timeout_minutes(30) Action.pulumi_cancel_deployment().stack("{{signal.stack}}").deployment_id("{{signal.deployment_id}}") Action.pulumi_pause_deployments().stack("my-project/prod") # incident/maintenance freeze Action.pulumi_resume_deployments().stack("my-project/prod") Action.pulumi_get_stack_outputs().stack("my-project/prod") Action.pulumi_get_drift().stack("my-project/prod") Action.pulumi_set_stack_tag().stack("{{signal.stack}}").tag_name("kestrel:quarantined").tag_value("true") Action.pulumi_delete_stack_tag().stack("{{signal.stack}}").tag_name("kestrel:quarantined") Action.pulumi_investigate().query("Why did the update fail?").stack("{{signal.stack}}") # HashiCorp Vault (KV-scoped: .mount() + .path(); read-secret values are sensitive # — redacted in run history but usable in downstream templates) Action.vault_read_secret().mount("secret/").path("app/prod/db").secret_key("password") Action.vault_write_secret().mount("secret/").path("app/prod/db").secret_data('{"password": "{{step_outputs.action-1.value}}"}') Action.vault_delete_secret().mount("secret/").path("app/old").destroy() # gate behind approval Action.vault_list_secrets().mount("secret/").path("app/prod") Action.vault_get_secret_metadata().mount("{{signal.mount}}").path("{{signal.secret_path}}") Action.vault_rotate_static_role().mount("database/").role("app-db") Action.vault_list_mounts() Action.vault_list_policies() Action.vault_read_policy().name("{{signal.policy_name}}") Action.vault_write_policy().name("app-read").policy_hcl('path "secret/data/app/*" { capabilities = ["read"] }') Action.vault_list_auth_methods() Action.vault_list_leases().lease_prefix("database/creds") Action.vault_revoke_lease().lease_id("{{step_outputs.action-1.keys}}") # credential-leak response Action.vault_renew_lease().lease_id("database/creds/app/abc").increment(3600) Action.vault_get_health() Action.vault_list_token_accessors() Action.vault_revoke_token_accessor().accessor("hmac.abc123") # gate behind approval Action.vault_investigate().query("Why is this secret stale?").mount("{{signal.mount}}") # Infisical (project-scoped: .infisical_project() + .environment() + .secret_path(); # get-secret values are sensitive — redacted in run history but usable in templates) Action.infisical_get_secret().infisical_project("backend").environment("prod").secret_path("/api").secret_key("DB_PASSWORD") Action.infisical_create_secret().infisical_project("backend").environment("prod").secret_key("API_KEY").secret_value("{{step_outputs.action-1.value}}") Action.infisical_update_secret().infisical_project("backend").environment("prod").secret_key("API_KEY").secret_value("...") Action.infisical_delete_secret().infisical_project("backend").environment("prod").secret_key("OLD_KEY") # gate behind approval Action.infisical_list_secrets().infisical_project("backend").environment("prod") # key names only, never values Action.infisical_list_projects() Action.infisical_list_environments().infisical_project("backend") Action.infisical_list_folders().infisical_project("backend").environment("prod").path("/") Action.infisical_create_folder().infisical_project("backend").environment("prod").path("/").name("payments") Action.infisical_list_secret_syncs().infisical_project("{{signal.project_id}}") Action.infisical_trigger_secret_sync().sync_id("{{signal.sync_id}}") # retry a failed sync Action.infisical_list_approval_requests().infisical_project("{{signal.project_id}}") Action.infisical_get_audit_logs().infisical_project("{{signal.project_id}}").event_type("delete-secret").limit(100) Action.infisical_list_identities() Action.infisical_investigate().query("Which syncs are failing and why?") # Approval Gates Action.approval_manual() Action.approval_slack().channel("approvals") Action.approval_pr() ``` ### Conditions (Branching) ```python theme={null} wf = ( Workflow("Conditional RCA Response") .trigger(Trigger.k8s_pod_status().reasons("CrashLoopBackOff")) .then(Action.kestrel_trigger_rca()) .then(Condition.equals("rca_result.is_application_level_failure", "true")) .on_true(Action.github_create_pr().repo("org/repo").label("Create Fix PR")) .on_false(Action.slack_send_message().channel("infra").label("Notify Infra Team")) ) ``` Available operators: `equals`, `not_equals`, `contains`, `not_contains`, `exists`, `not_exists`. The value-comparing factories accept multiple candidate values — `equals`/`contains` are met when the field matches **any** value, `not_equals`/`not_contains` only when it matches **none**: ```python theme={null} Condition.equals("sandbox_state", "stopped", "error") # met if stopped OR error Condition.not_equals("phase", "Failed", "Degraded") # met only if neither ``` ### Poll Until (Loops) `PollUntil` builds a self-looping node that repeatedly executes ONE embedded catalog action at a fixed interval and exits on the **met** branch when the condition holds against that iteration's output, or on the **timeout** branch when the timeout elapses. Wire the branches with `.on_met()` / `.on_timeout()`: ```python theme={null} wf = ( Workflow("Snapshot stopped sandbox") .trigger(Trigger.custom_webhook("sandbox.watch")) .then( PollUntil( Action.daytona_get_sandbox().sandbox_id("{{signal.sandbox_id}}"), Condition.equals("sandbox_state", "stopped", "error"), ) .every(seconds=60) .timeout(minutes=30) .label("Poll until sandbox stops") ) .on_met(Action.daytona_create_snapshot().name("post-stop-{{signal.sandbox_id}}")) .on_timeout(Action.slack_send_message().channel("ops").message("Sandbox never stopped")) ) ``` * `.every(seconds=..., minutes=...)` — polling interval (default 60s, minimum 30s). * `.timeout(minutes=..., hours=...)` — loop timeout (default 60 minutes). * Downstream steps on the **met** branch can reference the final iteration's output fields via `{{step_outputs..}}`. Polling is durable across server restarts: an in-flight interval resumes its countdown rather than restarting. ### For Each (Fan-Out) `ForEach` builds a fan-out node that resolves a list from an upstream step's output at runtime and executes ONE embedded catalog action per element, sequentially. Per-item config templates may reference `{{item}}` (the current element), `{{item.}}` (a field of an object element), and `{{item_index}}` (the zero-based index): ```python theme={null} from kestrel.workflows import Workflow, Trigger, Action, ForEach wf = ( Workflow("Ticket per audit finding") .trigger(Trigger.schedule_weekly()) # Script writes an array of finding objects to outputs.json: .then(Action("kestrel", "kestrel-execute-script") .config("script", AUDIT_SCRIPT) .label("Audit cluster")) .then( ForEach( "{{step_outputs.action-1.outputs.new_findings}}", Action.jira_create_ticket() .project("SEC") .title("[Audit] {{item.title}}") .body("{{item.description}}"), ) .max_items(50) .continue_on_error() .label("Create ticket per finding") ) # Runs once, after ALL items have been processed: .then(Action.slack_send_message().channel("security") .message("Filed {{step_outputs.foreach-1.succeeded}} tickets")) ) ``` * The items expression must resolve to a list: an array output, a JSON-encoded array string, or a comma-separated string. * `.max_items(n)` — caps the fan-out (server default 25, hard cap 100). * `.continue_on_error()` — a failed item doesn't fail the node; remaining items still run. * The node emits one aggregated output (`items_total`, `items_processed`, `succeeded`, `failed`, per-item `results`, and a flat `summary`), referenced as `{{step_outputs..}}`. * `ForEach` runs items once each, immediately. To repeatedly re-run an action until a condition holds, use `PollUntil` instead. ### Approvals ```python theme={null} wf = ( Workflow("Approved Deployment Fix") .trigger(Trigger.k8s_rollout_status()) .then(Action.kestrel_trigger_rca()) .then(Approval.slack("#approvals").message("Apply the generated fix?")) .on_approved(Action.kestrel_apply_yaml_fix()) .on_rejected(Action.slack_send_message().channel("ops").message("Fix rejected")) ) ``` Types: `Approval.manual()`, `Approval.slack(channel)`, `Approval.pr_approval()`, `Approval.pr_merge()`. ### Iterative RCA Refinement (human-in-the-loop) `Approval.refine()` builds a single self-looping gate that presents the upstream RCA + fixes for review. When the approver **requests changes** with free-text guidance, the *same* upstream RCA agent re-runs with that feedback (accumulated across rounds) and re-requests approval — looping until approved/rejected or `max_rounds` is reached. It must be placed after a `kestrel_trigger_rca()` or `kestrel_trigger_cloud_rca()` step. ```python theme={null} wf = ( Workflow("RCA with human refinement") .trigger(Trigger.k8s_rollout_status()) .then(Action.kestrel_trigger_rca().label("Run RCA")) .then(Action.kestrel_generate_runbook().label("Draft runbook")) # Reviewer can iteratively refine the RCA before it's applied: .then(Approval.refine().max_rounds(3).message("Review the RCA + fix")) .on_approved(Action.kestrel_apply_yaml_fix()) .on_approved(Action.confluence_publish_runbook()) # publishes runbook_html .on_rejected(Action.slack_send_message().channel("ops").message("RCA rejected")) ) ``` Use `Approval.refine("slack")` to request the refinement approval in Slack instead of the Kestrel UI. `max_rounds` defaults to 5; once exhausted the loop advances on the **approved** branch. ## Workflow Management ### List, Get, Update, Delete ```python theme={null} # List all workflows workflows = client.workflows.list() active = client.workflows.list(status="active") # Get a specific workflow wf = client.workflows.get("workflow-id") # Activate / Pause client.workflows.activate("workflow-id") client.workflows.pause("workflow-id") # Delete client.workflows.delete("workflow-id") ``` ### Generate from Natural Language ```python theme={null} result = client.workflows.generate( "When a PagerDuty alert fires, run K8s RCA, post to Slack, and create a Jira ticket" ) print(result.name, result.explanation) ``` ## Execution Management ### List and Inspect Executions ```python theme={null} execs = client.workflows.executions("workflow-id", page=1, page_size=10) for ex in execs.executions: print(f"{ex.id[:8]}... {ex.status}") # Get a specific execution execution = client.executions.get("execution-id") ``` ### Test and Wait Trigger a test execution and wait for it to complete: ```python theme={null} execution = client.workflows.test("workflow-id") result = client.executions.wait(execution.id, timeout=120.0) print(f"Final status: {result.status}") ``` ## Async SDK The `AsyncKestrelClient` provides the same API surface with `async/await`: ```python theme={null} import asyncio from kestrel import AsyncKestrelClient from kestrel.workflows import Workflow, Trigger, Action async def main(): async with AsyncKestrelClient(api_key="kestrel_sk_...") as client: # List workflows workflows = await client.workflows.list() # Deploy a workflow wf = ( Workflow("Async Example") .trigger(Trigger.k8s_pod_status().reasons("OOMKilled")) .then(Action.kestrel_trigger_rca()) .then(Action.slack_send_message().channel("incidents")) ) created = await client.workflows.deploy(wf, activate=True) # Test and wait for result execution = await client.workflows.test(created.id) result = await client.executions.wait(execution.id, timeout=60.0) print(f"Result: {result.status}") asyncio.run(main()) ``` ## Approvals ```python theme={null} # List pending approvals pending = client.approvals.list_pending() # Approve or reject client.approvals.approve("approval-id", justification="Looks good") client.approvals.reject("approval-id") ``` ## Version History & Rollback ```python theme={null} # List version history versions = client.workflows.list_versions("workflow-id") for v in versions["versions"]: print(f"v{v['version_number']} — {v['change_summary']} ({v['created_at']})") # Roll back to a previous version client.workflows.rollback("workflow-id", version=3) ``` ## Execution Replay Replay failed executions from the beginning or from the failed step: ```python theme={null} # Replay from beginning (re-triggers with same signal data) new_exec = client.executions.replay("execution-id", mode="full") # Replay from failed step (preserves outputs from prior steps) new_exec = client.executions.replay("execution-id", mode="from_failed") # Wait for replay to complete result = client.executions.wait(new_exec.id) print(f"Replay status: {result.status}") ``` ## Error Handling ```python theme={null} from kestrel import KestrelError, AuthError, NotFoundError try: wf = client.workflows.get("nonexistent") except NotFoundError: print("Workflow not found") except AuthError: print("Authentication failed — check your API key") except KestrelError as e: print(f"API error: {e}") ``` ## API Key Scopes API keys can be scoped to specific permissions: | Scope | Description | | -------------------- | ----------------------------------------- | | `*` | Full access (all permissions) | | `workflows:read` | List and view workflows | | `workflows:write` | Create, update, test, duplicate workflows | | `workflows:activate` | Activate and pause workflows | | `workflows:delete` | Delete workflows | | `executions:read` | View execution history | | `executions:cancel` | Cancel running executions | | `approvals:read` | View pending approvals | | `approvals:manage` | Approve or reject gates | | `requests:read` | View workflow requests | | `requests:manage` | Approve or reject requests | | `catalog:read` | View signals, actions, and integrations | Create scoped keys in the platform under **Workflows > API Keys**. # Workflow Integrations Setup Source: https://docs.usekestrel.ai/workflows/setup-integrations Connect the integrations your workflows need Workflows chain actions across multiple integrations. Before using an integration in a workflow, connect it on the **Integrations** page in the Kestrel dashboard. This page links to the setup guide for each supported integration. You only need to connect an integration once. All workflows in your organization share the same integration connections. Prefer the terminal? Every integration can also be connected with the [Kestrel CLI](/workflows/cli#integration-commands) (`kestrel integrations connect ` — it walks you through setup and prompts for secrets with hidden input) or by an AI coding agent through [MCP](/workflows/mcp). ## Infrastructure **Triggers:** Deployment Replicas Failing, Pod CrashLoopBackOff, Pod ImagePullBackOff, Pod OOMKilled, Pods Failing, Pods Restarting, StatefulSet Replicas Failing, Node Memory/Disk Pressure, DaemonSet Failing, and more. **Actions:** Trigger K8s RCA & Generate Fix, Apply YAML Fix, Generate K8s Manifest, Apply K8s Manifest, Investigate Kubernetes. **Triggers:** Node Provisioning Failed, Node Interrupted, NodePool Limit Reached. **Actions:** List NodePools, Get NodePool Status, List NodeClaims, Scale NodePool Limits, Set Disruption Policy, Apply NodePool, Delete NodeClaim. **Triggers:** Policy Violation, Admission Blocked (auto-detected in-cluster, streamed from PolicyReports). **Actions:** List Policies, Get Policy, List Violations, Set Enforcement Mode, Apply Policy, Delete Policy. **Triggers:** IAM Security Event, Root Account Activity, S3 Bucket Change, EC2/Lambda/RDS/DynamoDB Issues, VPC/Network Change, CloudWatch Alarm, Cost Anomaly, Budget Threshold, Forecast Overrun, Spend Spike, Idle Resource, and more. **Actions:** Trigger Cloud RCA & Generate Fix, Generate Cloud Resource (Terraform/CloudFormation/CLI), Execute Cloud CLI, Query Cost Explorer, Get Cost Anomalies/Forecast/Budget Status, Rightsizing/Savings Plans/Reservation Recommendations, Find Idle Resources, Cost Remediation (Stop EC2, Delete EBS Volumes, Release EIPs, Delete Snapshots), Investigate Cloud. Connect Oracle Cloud Infrastructure for resource inventory and incident detection. **Triggers (webhook):** Run Created, Run Planning, Run Needs Attention, Run Applying, Run Completed, Run Errored, Drift Detected, Assessment Check Failed. **Actions:** List/Get Workspace, Lock/Unlock/Force-Unlock Workspace, List/Get Run, Create Run (Plan), Create Destroy Run, Apply/Discard/Cancel Run, Wait for Run, Get State Outputs, List Variables, Set Variable, Get Drift Assessment, Investigate Terraform. **Triggers (webhook):** Update Succeeded/Failed, Preview Failed, Destroy Succeeded, Deployment Started/Succeeded/Failed, Drift Detected, Drift Run Failed, Policy Violation, Stack Created/Deleted. **Actions:** List/Get Stack, List/Get Update, Run Deployment (update/preview/refresh/destroy/drift), Get/Wait for/Cancel Deployment, Pause/Resume Deployments, Get Stack Outputs, Get Drift Status, Set/Delete Stack Tag, Investigate Pulumi. ## Source Control & CI/CD **Actions:** Create Pull Request, Create Issue, Trigger GitHub Action, Wait for GitHub Action Run, Get GitHub Action Status, Read File, Search Code, AI Code Investigation, AI Code Fix, Wait for PR Approval, Wait for PR Merge. **Actions:** Create Merge Request, Create Issue, Trigger Pipeline, Wait for Pipeline, Get Pipeline Status, Wait for MR Approval, Wait for MR Merge. **Actions:** Trigger ArgoCD Sync, Wait for ArgoCD Sync, Get ArgoCD App Status. **Triggers (webhook):** Build Failed, Build Unstable, Build Succeeded, Build Completed, Build Started. **Actions:** Trigger Build, Wait for Build, Get Build Status, Stop Build, Get Console Log, Investigate Jenkins. **Triggers (webhook):** Workflow Failed, Workflow Succeeded, Workflow Completed, Job Failed. **Actions:** Trigger Pipeline, Wait for Pipeline, Get Workflow Status, Rerun Workflow, Cancel Workflow, Approve On-Hold Job, Get Job Test Results, Investigate CircleCI. ## Notifications & Chat **Triggers:** Create K8s Resource, Edit K8s Resource, General Kubernetes Request, Create AWS Resource, Edit AWS Resource, General Cloud Request, Any /kestrel-workflow Request. **Actions:** Send Message, Update Message, Request Justification. **Triggers:** Incident Triggered, Incident Acknowledged, Incident Resolved, Any PagerDuty Incident, High Urgency Incident. **Actions:** Create Alert, Acknowledge Alert, Add Note to Alert, Resolve Alert, Escalate. ## Project Management & Documentation **Actions:** Create Ticket, Add Comment, Transition Ticket. **Actions:** Create Issue, Add Comment, Update Issue, Search Issues. **Actions:** Publish RCA, Publish Postmortem, Publish Runbook Entry, Update Page. ## Monitoring **Actions:** Query Metrics, Create Monitor, Send Event, Mute Monitor. Also provides metrics context for AI RCA and copilot investigations. Collects traces and metrics for infrastructure observability. Provides metrics context for AI RCA and copilot investigations. ## Product Analytics **Triggers:** Session Error/Exception, Console Error, Rage Click, Any PostHog Event, Log Error Alert. **Actions:** Get Session Summary, Get Session Recording, Query Events, List Session Recordings, Get Error Issue. ## Deployment **Triggers:** Deployment Failed, Deployment Succeeded, Deployment Created, Error Anomaly, Usage Anomaly, Domain Issue, Firewall Attack Detected, Deployment Checks Failed, Production Rollback. **Actions:** Get Deployment, Get Build Logs, Rollback Production, Promote to Production, List Deployments, Investigate Vercel. **Triggers:** Deployment Failed, Deployment Crashed, Deployment Succeeded, Volume Usage Alert, CPU/RAM Monitor Alert. **Actions:** Get Deployment, Get Deployment Logs, Rollback, Redeploy, Restart, List Deployments, Set Variables, Investigate Railway. **Triggers (poll-based):** Machine Crashed, Machine Stopped, Machine Started, App Down. **Actions:** Restart Machine, Start Machine, Stop Machine, Suspend Machine, Cordon Machine, Uncordon Machine, Get Machine, Get Machine Events, List Machines, Set Secrets, Investigate Fly. **Triggers (poll-based):** GPU Error, Maintenance Scheduled, Node Not Ready, Instance Stopped. **Actions:** Get Instance, Start Instance, Stop Instance, Restart Instance, Create Instance, Delete Instance, List Instances, List Clusters, List Node Groups, Scale Node Group, Create Node Group, Delete Node Group, Investigate Nebius. **Triggers (webhook):** Sandbox Created, Sandbox Stopped, Sandbox Error, Sandbox Archived, Sandbox Execution Failed, Snapshot Build Failed, Volume Error. **Actions:** List/Create/Get/Start/Stop/Archive/Delete Sandbox, Run Command, Set Auto-Stop, List/Create/Delete Snapshot, List/Get/Create/Delete Volume, Investigate Daytona. **Triggers (poll-based + webhook):** Project Health Degraded, Backup Failed, Read Replica Unhealthy, Usage/Quota Threshold, Branch Created, Branch Migration Failed (poll-based), Database Row Event (Database Webhook). **Actions:** List/Get Project, Get Project Health, Create/List/Get/Merge/Reset/Delete Branch, List Backups, Create Restore Point, Restore Backup (PITR), Setup/Remove Read Replica, Pause/Restore Project, Get/Update Network Restrictions, List API Keys, Investigate Supabase. **Triggers (poll-based):** Branch Created, Branch Ready, Operation Failed, Compute Suspended, Compute Active, Usage/Quota Threshold. **Actions:** List/Get Project, List/Get/Create/Delete/Reset Branch, Restore Branch (Point-in-Time), List/Create/Delete Compute Endpoint, Suspend/Start Compute, Set Autoscaling Limits, Rotate Role Credentials, Get Connection URI, Investigate Neon. **Triggers (webhook + poll-based):** Deploy Request Opened/Queued/In Progress/Schema Applied/Errored/Reverted/Closed, Branch Ready, Branch Anomaly, Branch Primary Promoted, Branch Sleeping, Storage Threshold (webhook), Backup Completed/Failed (poll-based). **Actions:** List/Get Database, List/Get/Create/Delete Branch, Promote Branch, Set Safe Migrations, List/Get/Create/Deploy/Revert/Close/Approve Deploy Request, List/Create Backup, List/Create/Delete Branch Password, Investigate PlanetScale. **Triggers (poll-based):** Service State Changed, Service Idle, Service Scaled, Backup Completed/Failed, Query Error Spike, ClickPipe Failed, Version Changed, Usage/Spend Threshold, Too Many Parts, High Query Concurrency. **Actions:** List/Get/Create/Start/Stop/Delete Service, Update Autoscaling, Update IP Access List, List/Get Backup, Update Backup Configuration, Restore Backup, List API Keys, Get Service Metrics, Get Usage Costs, List/Get/Start/Stop/Resync/Scale ClickPipe, Get/Set/Clear Autoscaling Schedule, Get/Set/Clear Upgrade Window, Get/Update/Reset Setting, Get/Upsert/Delete Query Endpoint, List Members, Remove Member, List Roles, List Activity, Investigate ClickHouse. ## Secret Management **Triggers (poll-based):** Vault Sealed/Unsealed, Health Degraded, Secret Version Created, Secret Stale, Policy Created/Deleted, Auth Method Enabled/Disabled. **Actions:** Read/Write/Delete Secret, List Secrets, Get Secret Metadata, Rotate Static Role, List Mounts, List/Read/Write Policy, List Auth Methods, List/Revoke/Renew Lease, Get Health, List/Revoke Token Accessor, Investigate Vault. **Triggers (poll-based):** Secret Created/Updated/Deleted, Approval Requested, Secret Sync Failed, Identity Created. **Actions:** Get/Create/Update/Delete Secret, List Secrets, List Projects/Environments/Folders, Create Folder, List/Trigger Secret Sync, List Approval Requests, Get Audit Logs, List Identities, Investigate Infisical. ## Platform Security **Triggers (webhook):** Analysis Completed (with quality gate result), Analysis Failed. **Actions:** Get Quality Gate Status, List Issues, List Security Hotspots, Get Project Metrics, Transition Issue, Assign Issue, Comment on Issue, Review Hotspot. **Triggers (poll-based via the System Log):** User Locked Out, Suspicious Activity Reported, Admin Privilege Granted, MFA Factor Changed, User Created/Deactivated, Group Membership Changed, App Assignment Changed. **Actions:** Get User, List Users, Query System Log, List User's Groups, List Group Members, Suspend/Unsuspend User, Unlock User, Deactivate User, Clear Sessions, Expire Password, Reset MFA, Add/Remove User to/from Group. ## Cost Management AWS Cost features are available through the [AWS integration](/integrations/aws). No separate setup is required — cost management actions are enabled when you connect an AWS account. * **Insights (read-only):** Query Cost Explorer, Get Cost Anomalies, Get Cost Forecast, Get Budget Status, Get Rightsizing Recommendations, Get Savings Plans Recommendations, Get Reservation Recommendations, Get Commitment Utilization, Compare Cost Periods, Find Idle Resources, Get Compute Optimizer Recommendations, Get Trusted Advisor Cost Checks, AI Cost Analysis * **Remediation (destructive — place behind an Approval block):** Stop EC2 Instances, Delete Unattached EBS Volumes, Release Elastic IPs, Delete Old Snapshots * **Cost triggers:** Cost Anomaly Detected, Budget Threshold Exceeded, Forecast Exceeds Budget, Spend Spike Detected, Idle Resource Detected * **Scheduled workflows:** combine the [Recurring Schedule trigger](/workflows/create-workflows#schedule) with cost actions for periodic reports and cleanup scans Accounts connected before the cost-optimization release need a CloudFormation stack update to grant the new Cost Explorer, Budgets, Compute Optimizer, and remediation IAM permissions. ## Knowledge Sources During AI-powered steps like RCA and investigation, Kestrel's AI agents search your team's knowledge bases for historical context — past incidents, runbooks, and documentation. Connect knowledge sources on the **Knowledge Sources** page in the dashboard. Runbooks and postmortems Past incident tickets Past issues and bugs Historical conversations For details on how knowledge sources enhance AI actions, see [Knowledge Sources](/integrations/knowledge-sources). *** ## Next Steps * [Create Workflows](/workflows/create-workflows) — Build your first workflow after connecting integrations * [Custom Integrations](/workflows/custom-integrations) — Connect services not listed here using HTTP actions and webhooks