Join our community of software engineering leaders and aspirational developers. Always
stay in-the-know by getting the most important news and exclusive content delivered
fresh to your inbox to learn more about at-scale software development.
REQUIRED
It seems that you've previously unsubscribed from our newsletter
in the past. Click the button below to open the re-subscribe form
in a new tab. When you're done, simply close that tab and continue
with this form to complete your subscription.
The New Stack does not sell your information or share it with
unaffiliated third parties. By continuing, you agree to our
Terms of Use and
Privacy Policy.
Welcome and thank you for joining The New Stack community!
Please answer a few simple questions to help us deliver the news and resources you are interested in.
REQUIRED
REQUIRED
REQUIRED
REQUIRED
REQUIRED
Great to meet you!
Tell us a bit about your job so we can cover the topics you find most relevant.
REQUIRED
REQUIRED
REQUIRED
REQUIRED
REQUIRED
Welcome!
We’re so glad you’re here. You can expect all the best TNS content to arrive
Monday through Friday to keep you on top of the news and at the top of your game.
What’s next?
Check your inbox for a confirmation email where you can adjust your preferences
and even join additional groups.
Follow TNS on your favorite social media networks.
Building robust, scalable applications in Kubernetes often requires fine-tuned automation — and that’s where Kubernetes operators come into play. Operators extend Kubernetes’ capabilities by automating management of custom resources and stateful applications. Think of operators as software that knows how to operate your apps, ensuring everything runs smoothly, even in complex environments.
I will guide you through the steps of creating a Kubernetes operator from scratch, a crucial skill for anyone working in modern cloud native ecosystems. We’ll walk through building a “self-healing diagnostics” operator, designed to boost observability in Kubernetes clusters. With this operator, crash information is automatically annotated to pods, and remediation actions can be applied on demand.
Why Is This Important to You?
For anyone managing Kubernetes clusters, operators streamline workflows, reduce manual intervention and allow proactive infrastructure management.
This tutorial cuts through the theory and gives you practical, real-world experience, covering:
Full local workflow: Learn how to run and test everything locally using Kind, a tool for testing Kubernetes locally, and container manager Docker.
Real-world operator patterns: You’ll create Custom Resource Definitions (CRDs), implement reconciliation loops and leverage the Kubernetes client API.
Immediate feedback: You’ll then test your operator and watch it automatically annotate crash information to pods in real time.
Mastering Kubernetes operators sets you up to manage complex systems with precision and scalability. Ready to level up? Let’s get started!
Prerequisites
Start by installing all the necessary tools to build and run your Kubernetes operator.
Installing prerequisites on macOS
docker --version
kubectl version --client
kind version
go version
kubebuilder version
Set Up a Cluster With Kind
# Create a Kubernetes cluster using Kind
kind create cluster --name self-healing-lab
# Verify the cluster is running
kubectl cluster-info
If you need to reset your environment at any point:
# Reset your cluster if you need a fresh start
kind delete cluster --name self-healing-lab
kind create cluster --name self-healing-lab
Create a Test Application
Let’s begin by creating a deliberately fragile application in Golang that will help you test your operator.
Create a Go Application
mkdir -p fragile-app
cd fragile-app
go mod init local.io/fragile-app
Create the following files:
main.go:
package main
import (
"fmt"
"net/http"
"os"
"os/signal"
"sync/atomic"
"syscall"
)
var requestCount int32
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
count := atomic.AddInt32(&requestCount, 1)
fmt.Fprintf(w, "Hello from the self-destructing app! %d\n", count)
if count > 5 {
fmt.Println("Request limit exceeded. Shutting down...")
go func() {
pid := os.Getpid()
syscall.Kill(pid, syscall.SIGTERM)
}()
}
})
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, "OK")
})
go func() {
fmt.Println("Server is running on :80")
if err := http.ListenAndServe(":80", nil); err != nil {
fmt.Printf("Server error: %v\n", err)
}
}()
// Handle termination signals
stop := make(chan os.Signal, 1)
signal.Notify(stop, os.Interrupt, syscall.SIGTERM)
<-stop
fmt.Println("Server stopped")
}
Dockerfile
# Use a small base image
FROM golang:1.24-alpine
# Set working directory
WORKDIR /app
# Copy and build the Go app
COPY . .
RUN go build -o fragile-app
# Expose port 80 and run the app
EXPOSE 80
CMD ["./fragile-app"]
Build and Test Locally
# Build the Docker image
docker build -t fragile-app:latest .
# Run it locally
docker run -p 8080:80 --rm fragile-app:latest
In another terminal, send requests to test the application; it should crash after five requests:
for i in {1..6}; do curl localhost:8080; echo; done
Deploy to Kubernetes
Create a deployment manifest:
fragile-app-deployment.yaml:
# Load the local image into Kind
kind load docker-image fragile-app:latest --name self-healing-lab
# Apply the deployment
kubectl apply -f fragile-app-deployment.yaml
# Check that the pod is running
kubectl get pods
Test the application in the cluster:
# Port forward to the pod
POD_NAME=$(kubectl get pods -l app=fragile-app -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 8080:80
In another terminal, send requests to crash the app:
for i in {1..10}; do curl localhost:8080; echo; done
Watch Kubernetes restart the pod:
kubectl get pods -w
Understanding the Operator Pattern
Before building an operator, it’s good to understand what makes it special.
What Is a Kubernetes Operator?
An operator is a pattern that extends Kubernetes to handle application-specific operational tasks. It works by doing the following:
It defines custom resources (i.e., CRDs) that represent your application’s desired state.
It implements controllers that continuously reconcile the actual state with the desired state.
Think of operators as automated domain experts that can monitor your application’s health, react to changes or problems, and apply specialized knowledge to fix issues.
The Reconciliation Loop
The heart of the operator is the reconciliation loop! Think of it as Kubernetes’ way of constantly moving from the current state to the desired state.
The controller is triggered whenever a pod changes. Then it:
Checks if the pod matches any PodDiagnoser’s target labels.
Examines container restart information.
Adds diagnostic annotations if crashes are detected.
Applies any configured remediation actions.
This cycle repeats continuously, ensuring your pods always have up-to-date diagnostic information!
Building an Operator
Now let’s create an operator that automatically diagnoses pod crashes.
Initialize the Operator Project
# Create a directory for your operator
mkdir -p pod-diagnoser-operator
cd pod-diagnoser-operator
# Initialize a new project with kubebuilder
kubebuilder init --domain=local.io --repo=local.io/pod-diagnoser
# Create the API and controller
kubebuilder create api --group=diagnostics --version=v1 --kind=PodDiagnoser --resource=true --controller=true
Define the CRD
Edit the generated api/v1/poddiagnoser_types.go file:
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// PodDiagnoserSpec defines the desired state of PodDiagnoser
type PodDiagnoserSpec struct {
// TargetLabels specifies which pods to monitor based on labels
// +optional
TargetLabels map[string]string `json:"targetLabels,omitempty"`
// EnableAnnotations determines whether to add diagnostic annotations
// +optional
EnableAnnotations bool `json:"enableAnnotations,omitempty"`
// RemediationAction specifies what action to take when crashes are detected
// +optional
RemediationAction string `json:"remediationAction,omitempty"`
}
// PodDiagnoserStatus defines the observed state of PodDiagnoser
type PodDiagnoserStatus struct {
// LastProcessedPod is the last pod that was processed
// +optional
LastProcessedPod string `json:"lastProcessedPod,omitempty"`
// DiagnosedPods contains the count of pods that have been diagnosed
// +optional
DiagnosedPods int `json:"diagnosedPods,omitempty"`
}
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
// PodDiagnoser is the Schema for the poddiagnosers API
type PodDiagnoser struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec PodDiagnoserSpec `json:"spec,omitempty"`
Status PodDiagnoserStatus `json:"status,omitempty"`
}
//+kubebuilder:object:root=true
// PodDiagnoserList contains a list of PodDiagnoser
type PodDiagnoserList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []PodDiagnoser `json:"items"`
}
func init() {
SchemeBuilder.Register(&PodDiagnoser{}, &PodDiagnoserList{})
}
Run the following command to update the generated code:
package controller
import (
"context"
"fmt"
"time"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/tools/record"
"k8s.io/client-go/util/retry"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log"
diagnosticsv1 "local.io/pod-diagnoser/api/v1"
)
// PodDiagnoserReconciler reconciles a PodDiagnoser object
type PodDiagnoserReconciler struct {
client.Client
Scheme *runtime.Scheme
Recorder record.EventRecorder // Add event recorder
}
//+kubebuilder:rbac:groups=diagnostics.local.io,resources=poddiagnosers,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=diagnostics.local.io,resources=poddiagnosers/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=diagnostics.local.io,resources=poddiagnosers/finalizers,verbs=update
//+kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;update;patch
//+kubebuilder:rbac:groups="",resources=events,verbs=create;patch
// Reconcile is part of the main kubernetes reconciliation loop
func (r *PodDiagnoserReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// Get the pod that triggered reconciliation
pod := &corev1.Pod{}
if err := r.Get(ctx, req.NamespacedName, pod); err != nil {
if errors.IsNotFound(err) {
// Pod was deleted, nothing to do
return ctrl.Result{}, nil
}
logger.Error(err, "Unable to fetch Pod")
return ctrl.Result{}, err
}
// Get all PodDiagnosers
diagnoserList := &diagnosticsv1.PodDiagnoserList{}
if err := r.List(ctx, diagnoserList); err != nil {
logger.Error(err, "Unable to list PodDiagnosers")
return ctrl.Result{}, err
}
// Check if this pod should be diagnosed based on PodDiagnoser rules
var matchingDiagnoser *diagnosticsv1.PodDiagnoser
for i := range diagnoserList.Items {
diagnoser := &diagnoserList.Items[i]
if shouldDiagnosePod(diagnoser, pod) {
matchingDiagnoser = diagnoser
break
}
}
if matchingDiagnoser == nil {
// No matching diagnoser for this pod
return ctrl.Result{}, nil
}
// Check for container restarts and diagnose
updated := false
for _, containerStatus := range pod.Status.ContainerStatuses {
if containerStatus.RestartCount > 0 && containerStatus.LastTerminationState.Terminated != nil {
// Container has restarted at least once
if pod.Annotations == nil {
pod.Annotations = make(map[string]string)
}
// Add diagnostic annotations
restartReason := containerStatus.LastTerminationState.Terminated.Reason
exitCode := containerStatus.LastTerminationState.Terminated.ExitCode
restartTime := containerStatus.LastTerminationState.Terminated.FinishedAt.Time
pod.Annotations["diagnostics.local.io/restart-reason"] =
fmt.Sprintf("%s (Exit Code: %d)", restartReason, exitCode)
pod.Annotations["diagnostics.local.io/restart-time"] =
restartTime.Format(time.RFC3339)
updated = true
logger.Info("Diagnosed pod restart",
"pod", pod.Name,
"container", containerStatus.Name,
"reason", restartReason,
"exitCode", exitCode)
}
}
if updated {
// Update the pod with new annotations using retry for robustness
if err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
// Get the latest version to avoid conflicts
latest := &corev1.Pod{}
if err := r.Get(ctx, req.NamespacedName, latest); err != nil {
return err
}
// Update annotations
if latest.Annotations == nil {
latest.Annotations = make(map[string]string)
}
latest.Annotations["diagnostics.local.io/restart-reason"] =
pod.Annotations["diagnostics.local.io/restart-reason"]
latest.Annotations["diagnostics.local.io/restart-time"] =
pod.Annotations["diagnostics.local.io/restart-time"]
return r.Update(ctx, latest)
}); err != nil {
logger.Error(err, "Failed to update Pod with diagnostic annotations")
return ctrl.Result{}, err
}
// Update the diagnoser status with retry for robustness
if err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
// Get the latest version
latest := &diagnosticsv1.PodDiagnoser{}
if err := r.Get(ctx, client.ObjectKey{
Namespace: matchingDiagnoser.Namespace,
Name: matchingDiagnoser.Name,
}, latest); err != nil {
return err
}
latest.Status.LastProcessedPod = pod.Name
latest.Status.DiagnosedPods++
return r.Status().Update(ctx, latest)
}); err != nil {
logger.Error(err, "Unable to update PodDiagnoser status")
return ctrl.Result{}, err
}
// Apply remediation if configured
if matchingDiagnoser.Spec.RemediationAction != "" {
if err := r.applyRemediation(ctx, matchingDiagnoser, pod); err != nil {
logger.Error(err, "Failed to apply remediation")
return ctrl.Result{}, err
}
}
}
return ctrl.Result{}, nil
}
// shouldDiagnosePod determines if this pod should be diagnosed by the given diagnoser
func shouldDiagnosePod(diagnoser *diagnosticsv1.PodDiagnoser, pod *corev1.Pod) bool {
if !diagnoser.Spec.EnableAnnotations {
return false
}
// If no target labels specified, match all pods
if len(diagnoser.Spec.TargetLabels) == 0 {
return true
}
// Check if pod matches all target labels
for key, value := range diagnoser.Spec.TargetLabels {
if pod.Labels[key] != value {
return false
}
}
return true
}
// applyRemediation applies remediation based on the diagnoser configuration
func (r *PodDiagnoserReconciler) applyRemediation(ctx context.Context,
diagnoser *diagnosticsv1.PodDiagnoser, pod *corev1.Pod) error {
logger := log.FromContext(ctx)
switch diagnoser.Spec.RemediationAction {
case "RestartPod":
logger.Info("Applying remediation: restarting pod", "pod", pod.Name)
// Just delete the pod, the deployment controller will create a new one
return r.Delete(ctx, pod)
case "LogEvent":
// Use the Kubernetes event recorder to log an event
r.Recorder.Event(pod,
corev1.EventTypeWarning,
"PodRestarted",
fmt.Sprintf("Pod %s restarted due to %s (Exit Code: %d)",
pod.Name,
pod.Annotations["diagnostics.local.io/restart-reason"],
pod.Status.ContainerStatuses[0].LastTerminationState.Terminated.ExitCode))
return nil
default:
logger.Info("No remediation action defined or unknown action",
"action", diagnoser.Spec.RemediationAction)
return nil
}
}
// SetupWithManager sets up the controller with the Manager.
func (r *PodDiagnoserReconciler) SetupWithManager(mgr ctrl.Manager) error {
// Initialize the event recorder
r.Recorder = mgr.GetEventRecorderFor("pod-diagnoser")
return ctrl.NewControllerManagedBy(mgr).
For(&corev1.Pod{}).
Complete(r)
}
Update the role-based access control (RBAC) manifests:
Modify the Deployment Manifest in Your Config Directory
Locate your operator’s deployment YAML file, typically in config/manager/manager.yaml
Add imagePullPolicy: Never to the container spec:
yaml
Copy
spec:
template:
spec:
containers:
- name: manager
image: controller:latest
imagePullPolicy: Never # Add this line
Build and Deploy the Operator
# Build the operator image
make docker-build IMG=pod-diagnoser:latest
# Load the image into the Kind cluster
kind load docker-image pod-diagnoser:latest --name self-healing-lab
# Install the CRDs
make install
# Deploy the operator
make deploy IMG=pod-diagnoser:latest
Apply the RBAC permissions:
kubectl apply -f operator-permissions.yaml
Check that the operator is running:
kubectl get pods -n pod-diagnoser-operator-system
If the pod is having issues pulling the image, you might need to restart it:
kubectl delete pod -n pod-diagnoser-operator-system \
$(kubectl get pods -n pod-diagnoser-operator-system -o jsonpath='{.items[0].metadata.name}')
Security alert: While using cluster-admin is convenient for experimentation, it grants excessive privileges. For production, always follow the principle of least privilege by creating custom roles with only the necessary permissions.
If you still see RBAC errors, you can temporarily grant broader permissions for development:
# For development only - not recommended for production
kubectl create clusterrolebinding pod-diagnoser-admin \
--clusterrole=cluster-admin \
--serviceaccount=pod-diagnoser-operator-system:pod-diagnoser-operator-controller-manager
Create a PodDiagnoser Instance
Create a file named config/samples/diagnostics_v1_poddiagnoser.yaml:
# Port forward to the pod
POD_NAME=$(kubectl get pods -l app=fragile-app -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 8080:80
In another terminal, send requests to crash the app:
for i in {1..10}; do curl localhost:8080; echo; done
Wait for the pod to restart, then check if the operator added diagnostic annotations:
kubectl get pod -l app=fragile-app -o jsonpath='{.items[0].metadata.annotations}'
Also, check for the event recorded by the operator:
kubectl get events | grep PodRestarted
You should see events like:
56s Warning PodRestarted pod/fragile-app-6488586b44-n7qg7 Pod fragile-app-6488586b44-n7qg7 restarted due to Completed (Exit Code: 0) (Exit Code: 0)
Clean Up
When you’re done with the tutorial, it’s time to clean up your resources:
# Delete the test application
kubectl delete -f fragile-app-deployment.yaml
# Delete the PodDiagnoser custom resource
kubectl delete -f config/samples/diagnostics_v1_poddiagnoser.yaml
# Uninstall the operator and CRDs
make undeploy
make uninstall
# Delete the cluster when you're done
kind delete cluster --name self-healing-lab
Conclusion
Congratulations! You’ve successfully built a Kubernetes operator that diagnoses pod crashes, applies remediation strategies and integrates seamlessly with Kubernetes events for enhanced visibility. This achievement not only showcases your understanding of the operator pattern but also highlights your ability to extend Kubernetes with domain-specific automation and intelligence.
By leveraging the power of the operator, you’ve enriched Kubernetes’ self-healing capabilities, creating a smarter, more efficient system that proactively addresses application failures with diagnostic insights.
Key takeaways for future success:
CRDs: You now know how to define schemas tailored to your application’s needs!
Controllers and reconciliation: You’ve mastered how to build the control loop that drives your operator.
Robust error handling: You can now implement retry mechanisms, ensuring resilience in future operations.
RBAC and security: You’ve scoped permissions, making your operator production-ready.
This project is a testament to what’s possible when you pair Kubernetes’ power with precise, purpose-built automation. Keep building, iterating and pushing the boundaries of what your Kubernetes environment can do. The future is powered by innovation like this, so get out there and keep making things happen.
Learn how to build a scalable CI/CD pipeline for Kubernetes with Andela’s guide “Make a Scalable CI/CD Pipeline for Kubernetes with GitHub and ArgoCD.”
Andela provides the world’s largest private marketplace for global remote tech talent driven by an AI-powered platform to manage the complete contract hiring lifecycle. Andela helps companies scale teams & deliver projects faster via specialized areas: App Engineering, AI, Cloud, Data & Analytics.