When pods behave badly in your cluster, looping over and over, it is known as CrashLoopBackOff. If the pod contains important data or you just need to edit something on the pod to get it fixed like running some checker on a program installed on the pod or moving things around or whatever the reason is but you need to get into the pod to fix it but there is no way you can just
kubectl -n namespace exec -it name bash to it then what do you do? dang you CrashLoopBackOff!!
- Advantage: the pod runs before the main application and finishes
- Disadvantage: the pod exits and the main application starts so if you are needing to do something interactive this does not work.
One possible way to get going is to run an
initContainer. Since a pod can run 1 or more containers the
initContainer can be easily added to the pod to run. You can have 1 or more
initContainer per pod and they run before the app containers start so if you need to fix a permission or make changes or run some pre-script before your application container then this is perfect.
- Init containers always run to completion
- Each init container must complete successfully before the next one starts
If a Pod’s init container fails, the kubelet repeatedly restarts that init container until it succeeds. However, if the Pod has a restartPolicy of Never, and an init container fails during startup of that Pod, Kubernetes treats the overall Pod as failed. (https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
If you are running via
Deployments you can setup
podTemplate to run your
One useful example is in elasticsearch to install plugins to the container before starting it. you would add
- podTemplate: spec: initContainers: - name: install-plugins command: - sh - -c - | bin/elasticsearch-plugin install --batch repository-gcs
If you are just changing this for a Pod then you can add it to the main
spec: containers: - name: myapp-container image: busybox:1.28 command: ['sh', '-c', 'echo The app is running! && sleep 3600'] initContainers: - name: init-myservice image: busybox:1.28 command: ['sh', '-c', "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"]
- Advantage: The container for the pod starts but the application does not start so you can make interactive changes before the application
- Disadvantage: You would need to reconfigure the pod after making updates to bring it up to normal
This method works by changing the pod template of the ReplicationController/Deployment/DeploymentConfig/Statefulset so that it starts with a blocking, never-failing command like
/bin/cat so that the command blocks the main program from starting then you can
exec into the container to run your fixes.
You will need to give it a
spec.template.spec from something like
spec: template: spec: containers: name: logstash
spec: template: spec: containers: name: logstash command: [ "/bin/cat" ] tty: true stdin: true
Then apply the change. The pod will restart and will run
/bin/cat forever so that you can
kubectl -n namespace exec -it containername bash and perform your work then exit out.
Once you are done you would revert the change and your pod should restart.