Lab: Tip/Code sample Kubernetes Container Lifecycle Events and Hooks

You might encounter cases where you need to instruct Kubernetes to start a pod only when a condition is met, such as dependencies are running, or sidecar containers are ready. Likewise, you might want to execute a command before Kubernetes terminates a pod to release the resources in use and gracefully terminate the application.

You can do so easily with two container lifecycle hooks:

1. PostStart: This hook is executed right after a container is created. However, the hook might be invoked after the container’s ENTRYPOINT is executed. The hook handler must not accept any parameters.
2. PreStop: This hook is executed immediately before a container is terminated due to any reason, such as resource contention, liveness probe failure, etc. You can not pass any parameters to the handler, and the container will be terminated irrespective of the outcome of the handler.

Here’s an illustration of the lifecycle events of a pod comprising two containers starting from the point when you instruct Kubernetes to create it to the point when both of them are running:

There are two types of handlers that you can attach to a lifecycle hook:

1. exec: It executes the specified command in the container’s main process. The command is executed in parallel with the container’s ENTRYPOINT instruction. If the hook takes too long or fails, the kubelet process will restart the container.
2. httpGet or tcpSocket: It sends an HTTP request or establishes a TCP socket connection against a specific endpoint on the container. Unlike the exec, which is executed by the container, this handler is executed by the kubelet process.

The hooks are executed at least once, and for HTTP handlers, the kubelet makes only one request delivery unless the kubelet restarts in the middle of sending the request.

Here’s an example of a deployment comprising a main container running NGINX and a sidecar container running busybox. The main container serves the file index.html from the mounted volume on port 80. The sidecar container writes scheduled logs to the same file, index.html, served by the main container. The sidecar container will start only when the main container is ready.

apiVersion: v1
kind: Pod
metadata:
  name: sidecar-container-demo
spec:
  containers:
    - image: busybox
      command: ["/bin/sh"]
      args:
        [
          "-c",
          "while true; do echo echo $(date -u) 'Written by busybox sidecar container' >> /var/log/index.html; sleep 5;done",
        ]
      name: sidecar-container
      resources: {}
      volumeMounts:
        - name: var-logs
          mountPath: /var/log
      lifecycle:
        postStart:
          httpGet:
            path: /index.html
            port: 80
            host: localhost
            scheme: HTTP
    - image: nginx
      name: main-container
      resources: {}
      ports:
        - containerPort: 80
      volumeMounts:
        - name: var-logs
          mountPath: /usr/share/nginx/html
  dnsPolicy: Default
  volumes:
    - name: var-logs
      emptyDir: {}

Till the hook postStart fails, the sidecar container will keep restarting. You can force the sidecar container to fail by changing the lifecycle check to the following:

lifecycle:
  postStart:
    httpGet:
      path: /index.html
      port: 5000
      host: localhost
      scheme: HTTP

You can view the events generated by the kubelet by running the following command:

kubectl describe pod/sidecar-container-demo

Here is the output of the command:

Let’s implement the next hook, preStop. The following command will print a log message and ensure that pod shuts down gracefully:

💡 Tip: You can direct preStop output to the PID 1 stdout, which ends up in the application logs. We’ll use this trick to track the execution of a pre-stop hook next.

apiVersion: v1
kind: Pod
metadata:
  name: prestop-demo
spec:
  containers:
    - image: nginx
      name: nginx-container
      resources: {}
      ports:
        - containerPort: 80
      lifecycle:
        preStop:
          exec:
            command:
              - sh
              - -c
              - echo "Stopping container now...">/proc/1/fd/1 && nginx -s stop
  dnsPolicy: Default

When the container using the pre-stop hook is terminated, the command nginx -s quit is executed in the container before kubelet sends the SIGTERM signal to the main process.

If you delete the pod while keeping a watch on the logs of the NGINX container, you will see the following output:

Finally, let’s discuss some important details about the lifecycle hooks:

• When you delete a pod object, the pre-stop hook is executed first, followed by the TERM signal to the main process. Next, the kubelet waits for the process to stop within seconds specified in the terminationGracePeriodSeconds property, after which it is killed.
• When you delete a pod object, all its containers are terminated in parallel. You can grant each container a custom grace period in seconds by setting the deletionGracePeriodSeconds property in the container’s specification.
• Handling the TERM signal using a handler is better than shortening the termination or deletion grace period.