Sunday, 23 August 2026

Zero-Downtime Rolling Updates in Kubernetes: Configuring Readiness Probes and PreStop Hooks

Deploying application updates in Kubernetes without dropping user requests requires more than just specifying strategy: RollingUpdate. While Kubernetes handles container orchestration gracefully, a race condition often occurs during pod termination between endpoint removal and traffic draining.

Understanding the Pod Termination Race Condition

When a Kubernetes deployment rolls out a new replica set, the following events occur concurrently for terminating pods:

  • Endpoint Removal: The endpoints controller removes the terminating pod IP from the Service's Endpoints object. Kube-proxy and ingress controllers update iptables/IPVS or upstream connection routing across nodes.
  • SIGTERM Signal: Kubelet sends a SIGTERM signal directly to the container process inside the pod.

Because network rule propagation across all worker nodes takes time (a few hundred milliseconds to several seconds), ingress controllers or upstream services might forward incoming requests to a pod that has already stopped accepting new connections or begun tearing down.

1. Implementing a PreStop Hook for Traffic Draining

A preStop lifecycle hook executes before the SIGTERM signal is sent to the container. Adding a brief sleep period (e.g., 5 to 15 seconds) ensures the endpoints controller and node iptables have finished propagating the removal before the application initiates shutdown.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

2. Configuring Precise Readiness and Liveness Probes

Readiness probes determine when a container is ready to accept traffic. If a pod fails its readiness check, Kubernetes removes it from the service endpoints before killing it.

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3

3. Setting terminationGracePeriodSeconds

The default terminationGracePeriodSeconds is 30 seconds. Ensure this value accommodates both your preStop sleep duration and the time required for active requests to finish processing.

Production Deployment Manifest Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  labels:
    app: api-service
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      terminationGracePeriodSeconds: 45
      containers:
      - name: api-service
        image: api-service:v1.2.0
        ports:
        - containerPort: 8080
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /healthz/live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
          failureThreshold: 3

Key Takeaways

  • Set maxUnavailable: 0 to prevent capacity drops during rollouts.
  • Use a preStop sleep hook to allow network rule propagation before shutting down the container.
  • Ensure terminationGracePeriodSeconds is greater than the sum of the preStop hook and the maximum in-flight request duration.
  • Implement distinct /healthz/ready and /healthz/live endpoints in your application code.

No comments:

Post a Comment