For some reason I needed a running pod to be able to access the value of its own container image(s). I initially thought of the Downward API but while it can give you the pod’s name, IP and other stuff, and even access resource limits per container using their name, the image is one of the attributes that is not immediately available. But the Downward API allows for metadata annotations to be available. So one can define a pod like:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    nginx_container: bitnami/nginx:1.23.2
  labels:
    app: nginx
  name: nginx
spec:
  containers:
  - image: bitnami/nginx:1.23.2
    name: nginx
    env:
    - name: IMAGE
      valueFrom:
        fieldRef:
          fieldPath: metadata.annotations['nginx_container']

But this has the drawback of defining the image value in two places in the same file. Unless you can template this out with helm, or you generate the YAML via some other automation, this is an invitation for a bug the time when you will forget to update both places. Is there a better way?

Of course there is, and I might have seen it sooner had I not been fixed on utilizing the Downward API. The answer is YAML anchors and aliases:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  containers:
  - image: &nginx_image bitnami/nginx:1.23.2
    name: nginx
    env:
    - name: IMAGE
      value: *nginx_image

When you run an RKE cluster on AWS, pulling images from an ECR is something to be expected. However, it is not the easiest of things to do that, since the credentials produced by aws ecr get-login expire every few hours and thus, you need something to refresh them. In Kubernetes world this means we should use a CronJob.

What we do in this post is a simple improvement on other work here and here. The biggest difference is using the Bitnami docker images for aws-cli and kubectl to achieve the same result.

So we need a CronJob that will schedule a pod to run every hour and refresh the credentials. This job will:

• run in a specific namespace and refresh the credentials there
• have the ability to delete and re-create the credentials
• do its best not to leak them
• use "well known" images outside the ECR in question

To this end our Pod needs an initContainer that is going to run aws ecr get-login and store it in an ephemeral space (emptyDir) for the main container to pick it up. The main container in turn, will pick up the password generated by the init Container and complete the credential refreshing.

Are we done yet? No, because by default the service account the Pod operates on, does not have the ability to delete and create credentials. So we need to create an appropriate Role and RoleBinding for this.

All of the above is reproduced in the below YAML. If you do not wish to hardcode the AWS region and ECR URL, you can of course make them environment variables.

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dns
  name: ecr-secret-role
rules:
- apiGroups: [""]
  resources:
  - secrets
  - serviceaccounts
  - serviceaccounts/token
  verbs:
  - 'create'
  - 'delete'
  - 'get'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dns
  name: ecr-secret-rolebinding
subjects:
- kind: ServiceAccount
  name: default
  namespace: dns
roleRef:
  kind: Role
  name: ecr-secret-role
  apiGroup: ""
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: ecr-update-login
  namespace: dns
spec:
  schedule: "38 */1 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          initContainers:
          - name: awscli
            image: bitnami/aws-cli
            command:
            - /bin/bash
            - -c
            - |-
              aws ecr get-login --region REGION_HERE | cut -d' ' -f 6 > /ecr/ecr.token
            volumeMounts:
            - mountPath: /ecr
              name: ecr-volume
          containers:
          - name: kubectl
            image: bitnami/kubectl
            command:
            - /bin/bash
            - -c
            - |-
              kubectl -n dns delete secret --ignore-not-found ecr-registry
              kubectl -n dns create secret docker-registry ecr-registry --docker-username=AWS --docker-password=$(cat /ecr/ecr.token) --docker-server=ECR_URL_HERE
            volumeMounts:
            - mountPath: /ecr
              name: ecr-volume
          volumes:
          - name: ecr-volume
            emptyDir: {}

You can now use the ECR secret to pull images by adding to your Pod spec:

 imagePullSecrets:
  - name: ecr-registry

And yes, it is possible to solve the same issue with instance profiles:

”ecr:GetAuthorizationToken”,
“ecr:BatchCheckLayerAvailability”,
“ecr:GetDownloadUrlForLayer”,
“ecr:GetRepositoryPolicy”,
“ecr:DescribeRepositories”,
“ecr:ListImages”,
“ecr:BatchGetImage”

This is a solution when for whatever reason, for when you cannot. Plus it may give you ideas for other helpers using the aws-cli and kubectl. Or for other clouds and registries even.

true is a reserved word in YAML and this can bite you when you least expect it. Consider the following extremely simple Pod and livenessProbe:

apiVersion: v1
kind: Pod
metadata:
  name: true-test
spec:
  containers:
  - name: nginx
    image: nginx
    livenessProbe:
      exec:
        command:
        - true

Let’s create this, shall we?

$ kubectl create -f true.yaml 

Error from server (BadRequest): error when creating "true.yaml": Pod in version "v1" cannot be handled as a Pod: v1.Pod.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.LivenessProbe: v1.Probe.Handler: Exec: v1.ExecAction.Command: []string: ReadString: expects " or n, but found t, error found in #10 byte of ...|ommand":[true]}},"na|..., bigger context ...|age":"nginx","livenessProbe":{"exec":{"command":[true]}},"name":"nginx"}]}}
|...
$

All because in the above specification, true is not quoted, as it should:

    livenessProbe:
      exec:
        command:
        - "true"

A different take, if you do not want to mess with true would be:

   livenessProbe:
      exec:
        command:
        - exit

Beware though, if you want to exit 0 you need to quote again:

   livenessProbe:
      exec:
        command:
        - exit
        - "0"

Oh, the many ways you can waste your time…

Now why would you want to do that? Because sometimes, you have to get data from customers and they whitelist specific IP addresses for you to get their data from. But the whole concept of Kubernetes means that, in general, you do not care where your process runs on (as long as it runs on the Kubernetes cluster) and also you get some advantage from the general elasticity it offers (because you know, an unspoken design principle is that your clusters are on the the cloud, and basically throwaway machines, alas life has different plans).

So assuming that you have a somewhat elastic cluster with some nodes that never get disposed and some nodes added and deleted for elasticity, how would you secure access to the customer’s data from a fixed point? You run a proxy service (like tinyproxy, haproxy, or whatever) on the fixed servers of your cluster (which of course the client has whitelisted). You need to label them somehow in the beginning, like kubectl label nodes fixed-node-1 tinyproxy=true. You now need a docker container to run tinyproxy from. Let’s build one using the Dockerfile below:

FROM centos:7
RUN yum install -y epel-release
RUN yum install -y tinyproxy
# This is needed to allow global access to tinyproxy.
# See the comments in tinyproxy.conf and tweak to your
# needs if you want something different.
RUN sed -i.bak -e s/^Allow/#Allow/ /etc/tinyproxy/tinyproxy.conf
ENTRYPOINT [ "/usr/sbin/tinyproxy", "-d", "-c", "/etc/tinyproxy/tinyproxy.conf" ]

Let’s build and push it to docker hub (obviously you’ll want to push to your own docker repository):

docker build -t yiorgos/tinyproxy .
docker push yiorgos/tinyproxy

We can now attempt to deploy a deployment in our cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tinyproxy
  namespace: adamo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tinyproxy
  template:
    metadata:
      labels:
        app: tinyproxy
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: tinyproxy
                operator: In
                values:
                - worker-4
                - worker-3
      containers:
      - image: yiorgos/tinyproxy
        imagePullPolicy: Always
        name: tinyproxy
        ports:
        - containerPort: 8888
          name: 8888tcp02
          protocol: TCP```

We can apply the above with `kubectl apply -f tinyproxy-deployment.yml`. The trained eye will recognize however that this YAML file is somehow [Rancher](http://rancher.com) related. Indeed it is, I deployed the above container with Rancher2 and got the YAML back with `kubectl -n proxy get deployment tinyproxy -o yaml`.  So what is left now to make it usable within the cluster? A service to expose this to other deployments within Kubernetes:

apiVersion: v1 kind: Service metadata: name: tinyproxy namespace: proxy spec: type: ClusterIP ports:

• name: 8888tcp02 port: 8888 protocol: TCP targetPort: 8888

We apply this with `kubectl apply -f tinyproxy-service.yml` and we are now set. All pods within your cluster can now access the tinyproxy and get access to whatever they need to by connecting to `tinyproxy.proxy.svc.cluster.local:8888`. I am running this in its own namespace. This may come handy if you use [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) and you want to restrict access to the proxy server for certain pods within the cluster.

This is something you can use with [haproxy](https://hub.docker.com/_/haproxy) containers instead of tinyproxy, if you so like.