Speed up rolling update of Kubernetes deployment

January 29, 2023

Recently I was updating one of the infrastructure component consist of many pods presented as a Kubernetes deployment. I suddenly realized that this process due to some timing requirements like probe initialDelaySeconds, terminationGracePeriodSeconds is too high to finish rolling deployment updates in a short time. So in this situation it was better for me to somehow speed up the rolling update than waiting for so long to make it happen automatically. Let’s experiment with what would happen during the deployment rolling update when I manually inject pods by setting the same label like as the new deployment. So the scenario would be:

Create new sample deployment
Set a new image tag, which triggers a rolling update. Rolling update is changing one pod at the time and wait 60s for startupProbe.
I will detach selected pod from the “old” deployment

$ kubectl label pods -l pod-template-hash=6676fdc4c7 delete-after=true app=nginx-detached --overwrite

I will “inject” detached pods into the “new” deployment

$ kubectl label pods -l app=nginx-detached pod-template-hash=6bd9b5f4cd delete-after=true app=nginx --overwrite

So the new deployment should have pods with an old and new image, to unify them I will delete pods with “old” image

$ kubectl delete pods -l delete-after=true

What is the effect of this experiment:

$ kubectl  rollout status deployment/nginx | gawk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }'
[2023-01-29 12:19:20] Waiting for deployment "nginx" rollout to finish: 1 out of 8 new replicas have been updated...
[2023-01-29 12:20:19] Waiting for deployment "nginx" rollout to finish: 1 out of 8 new replicas have been updated...
[2023-01-29 12:20:19] Waiting for deployment "nginx" rollout to finish: 1 out of 8 new replicas have been updated...
[2023-01-29 12:20:19] Waiting for deployment "nginx" rollout to finish: 2 out of 8 new replicas have been updated... <- started with the point 4 of my scenario
[2023-01-29 12:20:32] Waiting for deployment "nginx" rollout to finish: 2 out of 8 new replicas have been updated...
[2023-01-29 12:20:32] Waiting for deployment "nginx" rollout to finish: 2 out of 8 new replicas have been updated...
[2023-01-29 12:20:33] Waiting for deployment "nginx" rollout to finish: 2 out of 8 new replicas have been updated...
[2023-01-29 12:20:33] Waiting for deployment "nginx" rollout to finish: 2 out of 8 new replicas have been updated...
[2023-01-29 12:20:41] Waiting for deployment "nginx" rollout to finish: 3 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 3 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 3 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 3 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 4 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 4 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 4 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 4 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 5 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 5 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 5 out of 8 new replicas have been updated...
[2023-01-29 12:20:42] Waiting for deployment "nginx" rollout to finish: 3 out of 8 new replicas have been updated...
[2023-01-29 12:20:43] Waiting for deployment "nginx" rollout to finish: 5 out of 8 new replicas have been updated...
[2023-01-29 12:20:43] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:33] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:33] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:43] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:53] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:53] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:53] Waiting for deployment "nginx" rollout to finish: 6 out of 8 new replicas have been updated...
[2023-01-29 12:21:53] Waiting for deployment "nginx" rollout to finish: 7 out of 8 new replicas have been updated...
[2023-01-29 12:22:54] Waiting for deployment "nginx" rollout to finish: 7 out of 8 new replicas have been updated...
[2023-01-29 12:22:54] Waiting for deployment "nginx" rollout to finish: 7 out of 8 new replicas have been updated...
[2023-01-29 12:22:54] Waiting for deployment "nginx" rollout to finish: 7 of 8 updated replicas are available...
[2023-01-29 12:24:04] deployment "nginx" successfully rolled out

So theoretically the whole rolling update should take ca. 8min, with this mix the overall time took 5min (rollout) + 1min (delete “old” image pods) 6min. Not bad with more pods it should gain more benefits, but still it’s rather an experiment than something to copy-paste ;)