Deploying application updates in Kubernetes without dropping user requests requires more than just specifying strategy: RollingUpdate. While Kubernetes handles container orchestration gracefully, a race condition often occurs during pod termination between endpoint removal and traffic draining.
Understanding the Pod Termination Race Condition
When a Kubernetes deployment rolls out a new replica set, the following events occur concurrently for terminating pods:
- Endpoint Removal: The endpoints controller removes the terminating pod IP from the Service's Endpoints object. Kube-proxy and ingress controllers update iptables/IPVS or upstream connection routing across nodes.
- SIGTERM Signal: Kubelet sends a
SIGTERMsignal directly to the container process inside the pod.
Because network rule propagation across all worker nodes takes time (a few hundred milliseconds to several seconds), ingress controllers or upstream services might forward incoming requests to a pod that has already stopped accepting new connections or begun tearing down.
1. Implementing a PreStop Hook for Traffic Draining
A preStop lifecycle hook executes before the SIGTERM signal is sent to the container. Adding a brief sleep period (e.g., 5 to 15 seconds) ensures the endpoints controller and node iptables have finished propagating the removal before the application initiates shutdown.
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
2. Configuring Precise Readiness and Liveness Probes
Readiness probes determine when a container is ready to accept traffic. If a pod fails its readiness check, Kubernetes removes it from the service endpoints before killing it.
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
failureThreshold: 3
3. Setting terminationGracePeriodSeconds
The default terminationGracePeriodSeconds is 30 seconds. Ensure this value accommodates both your preStop sleep duration and the time required for active requests to finish processing.
Production Deployment Manifest Example
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-service
labels:
app: api-service
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
selector:
matchLabels:
app: api-service
template:
metadata:
labels:
app: api-service
spec:
terminationGracePeriodSeconds: 45
containers:
- name: api-service
image: api-service:v1.2.0
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
failureThreshold: 3
Key Takeaways
- Set
maxUnavailable: 0to prevent capacity drops during rollouts. - Use a
preStopsleep hook to allow network rule propagation before shutting down the container. - Ensure
terminationGracePeriodSecondsis greater than the sum of the preStop hook and the maximum in-flight request duration. - Implement distinct
/healthz/readyand/healthz/liveendpoints in your application code.
No comments:
Post a Comment