This post is again something from a question that popped in #kubernetes-users. A user was updating an immutable secret and yet the value was not propagating. But what is an immutable secret in the first place?

Once a Secret or ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data field. You can only delete and recreate the Secret. Existing Pods maintain a mount point to the deleted Secret – it is recommended to recreate these pods.

So the workflow is: Delete old immutable secret, create a new one, restart the pods that use it.

Let’s create an immutable secret from the command line:

% kubectl create secret generic version -o json --dry-run=client --from-literal=version=1  | jq '. += {"immutable": true}' | kubectl apply -f -
secret/version created

If there’s a way to create an immutable secret from the command line without using jq please tell me.

Now let’s create a deployment that uses it as an environment variable

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        env:
        - name: VERSION
          valueFrom:
            secretKeyRef:
              name: version
              key: version

Now let’s check the value for $VERSION:

% kubectl exec -it nginx-68bbbd9d9f-rwtfm -- env | grep ^VERSION
VERSION=1

The time has come for us to update the secret with a new value. Since it is immutable, we delete and recreate it:

% kubectl delete secrets version
secret "version" deleted
% kubectl create secret generic version -o json --dry-run=client --from-literal=version=2  | jq '. += {"immutable": true}' | kubectl apply -f -
secret/version created

We restart the pod and check the value of $VERSION again:

% kubectl delete pod nginx-68bbbd9d9f-rwtfm
pod "nginx-68bbbd9d9f-rwtfm" deleted
% kubectl exec -it nginx-68bbbd9d9f-x4c74 -- env | grep ^VERSION
VERSION=1

What happened here? Why has the pod the old value still? It seems that there some caching at play here and the new immutable secret is not passed at the pod. But let’s try something different now:

% kubectl delete secrets version
secret "version" deleted
% kubectl create secret generic version -o json --dry-run=client --from-literal=version=3  | jq '. += {"immutable": true}' | kubectl apply -f -
secret/version created
% kubectl scale deployment nginx --replicas 0
deployment.apps/nginx scaled
% kubectl scale deployment nginx --replicas 1
deployment.apps/nginx scaled
% kubectl exec -it nginx-68bbbd9d9f-zkj8h -- env | grep ^VERSION
VERSION=3

From what I understand, if you scale down your deployment and scale it up again this gives enough time for the kubelet to release the old cached value and pass the new, proper one.

Now all the above were tested with Docker Desktop Kubernetes. I did similar tests with a multinode microk8s cluster and when restarting the pods the environment variable was updated properly, but if instead you used a volume mount for the secret, it did not and you needed to scale down to zero first.

In the Kubernetes Slack an interesting question was posed:

Hi, can a pod belong to 2 workloads? For example, can a pod belong both the a workload and to the control plane workload?

My initial reaction was that, while a Pod can belong to two (or three, or more) services, it cannot belong to two workloads (Deployments for example). I put my theory to the test by creating initially a pod with some labels

apiVersion: v1
kind: Pod
metadata:
  name: caddy
  labels:
    apache: ok
    nginx: ok
spec:
  containers:
  - name: caddy
    image: caddy
    ports:
    - name: http
      containerPort: 80
    - name: https
      containerPort: 443

Sure enough the pod was created

% kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
caddy   1/1     Running   0          2s

Next I created a replicaSet whose pods have a label that the above (caddy) pod has also.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      nginx: ok
  template:
    metadata:
      labels:
        app: nginx
        nginx: ok
    spec:
      containers:
      - name: nginx
        image: bitnami/nginx
        ports:
        - containerPort: 8080

Since the original pod and the replicaSet share a common label (nginx: ok), the pod is assimilated in the replicaSet and it launches one additional pod only:

% kubectl get pod
NAME          READY   STATUS    RESTARTS   AGE
caddy         1/1     Running   0          2m52s
nginx-lmmbk   1/1     Running   0          3s

We can now ask Kubernetes to create an identical replicaSet that launches apache instead of nginx and has the apache: ok label set.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: apache
  labels:
    app: apache
spec:
  replicas: 2
  selector:
    matchLabels:
      apache: ok
  template:
    metadata:
      labels:
        app: apache
        apache: ok
    spec:
      containers:
      - name: apache
        image: bitnami/apache
        ports:
        - containerPort: 8080

If a pod can be shared among workloads, then it should start a single apache pod. Does it?

% kubectl get pod
NAME           READY   STATUS    RESTARTS   AGE
apache-8fwdz   1/1     Running   0          4s
apache-9xwhd   1/1     Running   0          4s
caddy          1/1     Running   0          5m17s
nginx-lmmbk    1/1     Running   0          2m28

As you can see, it starts two apache pods and the pods carrying the apache-ok label are three:

 % kubectl get pod -l apache=ok
NAME           READY   STATUS    RESTARTS   AGE
apache-8fwdz   1/1     Running   0          6m20s
apache-9xwhd   1/1     Running   0          6m20s
caddy          1/1     Running   0          11m

% kubectl get rs
NAME     DESIRED   CURRENT   READY   AGE
apache   2         2         2       6m21s
nginx    2         2         2       8m45s

So there you have it, a Pod cannot be shared among workloads.

While creating a Deployment earlier today, I faced a weird situation where a specific environment variable that was held as a Secret, was not being set. I tried deleting and recreating the secret, with no success. Mind you this was a long YAML with volumes, volumeMounts, ConfigMaps as enviroment variables, lots of lines. In the end the issue was pretty simple and I missed it because kubectl silently accepted the submitted YAML. I had two(!) env: blocks defined for the same container and somehow I missed that. It turns out, that when you do so, only the last one gets accepted, and whatever is defined in the previous, is not taken into account. To show this with an example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: box
  name: box
spec:
  replicas: 1
  selector:
    matchLabels:
      app: box
  template:
    metadata:
      labels:
        app: box
    spec:
      containers:
      - image: busybox
        name: busybox
        env:
        - name: FIRST_ENV
          value: "True"
        - name: ANOTHER_FIRST
          value: "True"
        command:
        - sleep
        - infinity
        env:
        - name: SECOND_ENV
          value: "True"

In the above example, when the Pod starts the container, only SECOND_ENV is set. FIRST_ENV and ANOTHER_FIRST are not.

I do not know whether this is a well known YAML fact or not, but it cost me some head-scratching and I hope it won’t cost you too.