A stunnel to (Google) LDAP Pod

Sometimes you have an application that needs to speak with an LDAP server. And more specifically to Google’s LDAP service. Google’s LDAP requires certificates to connect to it (which you get from your Workspace console). You can integrate those certificates with your application if possible, or you can run stunnel as a proxy, as per Google’s instructions.

It is not that complex then to expand from those instructions to running a stunnel Pod as a proxy. And it happens that Chainguard maintain a stunnel image that can be of use to us. Thus we can now run an LDAP service which can proxy our queries to our (Google) LDAP service:

apiVersion: v1
kind: ConfigMap
metadata:
  name: stunnel-conf
data:
  stunnel.conf: |
    foreground = yes
    #debug = debug
    output = /dev/stdout
    [ldap]
    client = yes
    accept = 0.0.0.0:1389
    connect = ldap.google.com:636
    cert = /google-ldap/tls.crt
    key = /google-ldap/tls.key
---
apiVersion: v1
kind: Service
metadata:
  name: ldap
spec:
  selector:
    app: ldap
  ports:
    - protocol: TCP
      port: 389
      targetPort: 1389
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ldap
  name: ldap
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ldap
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ldap
    spec:
      volumes:
      - name: google-ldap
        secret:
          secretName: google-ldap
      - name: stunnel-conf
        configMap:
          name: stunnel-conf
      containers:
      - image: chainguard/stunnel
        command:
        - /usr/bin/stunnel
        - /etc/conf/stunnel.conf
        name: stunnel
        ports:
        - containerPort: 1389
        volumeMounts:
        - name: google-ldap
          mountPath: /google-ldap
        - name: stunnel-conf
          mountPath: /etc/conf/stunnel.conf
          subPath: stunnel.conf

What is not defined in the above YAML is the kubernetes TLS secret that contains the certificate key-pair from Google:

$ kubectl create secret tls google-ldap \
--key ./Google_123456.key \
--cert ./Google_123456.crt

The LDAP service is now accessible from within your Kubernetes cluster as ldap://ldap.default.svc.cluster.local:389 (that is ldap:// and NOT ldaps://). If this is an issue for you, you can make it a sidecar container and thus access the stunnel proxy as ldap://127.0.0.1:389 instead.

My GitLab is not my resume

[ Originally posted on LinkedIn ]

My #GitHub is not my resume. It is mostly a repository of forks of things I found interesting enough to keep a local copy.

My #GitLab is not my resume. It is just a place where I used (but not anymore) store some stuff related to classes that I teach. So that I could also have a feel of how GitLab works.

You are not looking for a resume. You are looking for some weird form of reassurance because you see some repos, you know nothing about, there. And you will never audit beyond the latest commit so you don’t even have a clue whether and how I wrote that stuff.

I keep my stuff in sr.ht and mostly private.

How sometimes Docker saves the day

For reasons, I needed to work with a GNUPG version 2.2.0 precicely. This is not what comes with the standard packages of the OS distribution that I work with, so why not compile and install it in a separate tree and be done with it?

Unfortunately, GCC >= 10 cannot compile version 2.2.0 with all sorts of errors popping at different places of preprocessor definitions. I followed through some bug reports after googling around, and was this close to launching a VM with an old OS carrying the gpg. But I remembered that GCC is also on dockerhub. And I thought, In the old days I have used newer versions of GCC to compile older ones for similar reasons. Maybe now I can use a container?

Sure enough using

docker run --rm -it -w $(pwd):/app -w /app gcc:9 bash

and then (and after unzipping the source code) from within the container

make distclean
./configure --prefix=/app/gnupg --disable-libdns
make
make install

and voila! I had the thing installed locally.

Not much of an innovative story here, but I had not written in the blog for days and thought I could share a story.

OpenVPN, LDAP and group membership

While the need for LDAP integration and OpenVPN seems straightforward, it seems to me that the documentation for the auth-ldap plugin is not very easy to locate and find. Take for example the following auth-ldap.conf configuration file

<LDAP>
  URL ldap://ldap.example.com
  Timeout 15
</LDAP>
<Authorization>
  BaseDN "ou=users,dc=example,dc=com"
  SearchFilter "(uid=%u)" # (or choose your own LDAP filter for users)
  RequireGroup false
</Authorization>

This is a very handy starter that would allow any user with a working password under the ou=users part of your tree to be granted access. But what if you would want to restrict access based on group membership? According to fragments of documentation scattered at different bits of forums and StackOverflow / ServerFault, you’d need to set RequireGroup true and then use the BaseDN of the group and the memberUid attribute within a <Group> ... </Group> subsection of Authorization. This never worked for me. What worked was changing the Search filter to include group membership:

<LDAP>
  URL ldaps//ldap.example.com
  Timeout 15
</LDAP>
<Authorization>
  BaseDN "ou=users,dc=example,dc=com"
  SearchFilter "(&(uid=%u)(memberOf=cn=openvpn,ou=groups,dc=example,dc=com))"
  RequireGroup false
</Authorization>

Voila!

I did not come up with this. I found it via random Googling somewhere in SO (I cannot remember and cite that answer anymore).

X-Google-Smtp-Source

After some years (this blog was known as blog.postmaster.gr in the past afterall) I had to do some digging into an email’s headers. I couldn’t find what I was looking for, but there was a header inserted by the Google Workspace system named X-Google-Smtp-Source. It had a base64 encoded value. Decoding it did not produce any immediately meaningful result, so I dug a bit more. Searching for this header brings pages that repeat the following information from Google:

X-Google-Smtp-Source: This header shows the IP address of the computer that sent the email.
https://support.google.com/mail/thread/230509179/need-to-know-email-sent-recieved-times?hl=en

This does not provide much information. I tried to look for more, and even asked ChatGPT and Bard but they too came up with nothing helpful. Until I reached the following mail thread:

:
These headers are typically base64 encoded encrypted serialized protocol
buffers. Without the key, you won’t get anything out of them, and the
keys rotate on a schedule. For what’s actually in them, it’s probably
overkill, but better safe than sorry.

They also don’t contain the IP address.
:
As Grant pointed out, we consider the IP address of the user to be PII and
do not share it in most cases.
https://www.mail-archive.com/mailop@mailop.org/msg08419.html

Notice also the contradiction: the header seems not to carry the originating IP address. It seems that at a point in time this was indeed carrying the source IP, but something (maybe legislation?) changed and the contents of the header changed too, without the documentation being helpfully updated.

deletePad: delete forgotten etherpad pads

Etherpad is a fantastic tool for collaboration and text sharing among colleagues. It’s docker container can be easily deployed in a Kubernetes cluster and you can tie it to a database for some persistence if you like.

However this persistence can prove problematic sometimes, as you may accidentally share stuff that you shouldn’t, or share stuff for longer than you should. For this reason, it is fairly easy to use the Etherpad API and implement a job that deletes old pads, left unedited after some time. And this is what I did:

deletePad.py – the Python script that does the thing
cronjob.yaml – example Kubernetes CronJob to run this periodically
yiorgos/deletepad – ready made docker container if you trust me

Yes, there are Etherpad plugins that do the same, but I did not want to mess around with my deployment and add one more ConfigMap to support the settings for Etherpad and the like.

Originally, and because I link Etherpad to a Postgres, I was expiring old pads using pg_cron, but this was not the cleanest of solutions, because until restaring the Etherpad, the pad remained in the process’s memory and was serviceable, even though not on the database. And that is why I resorted to using the API.

microk8s, nginx ingress and X-Forwarded-For

Sometimes when you run microk8s in AWS, you may want to have an application load balancer in front. Such configurations mess around with the value of the header X-Forwarded-For regardless of whether the append attribute is present on the load balancer. By reading the nginx ingress documentation, you need to edit the ConfigMap resource and add proxy-real-ip-cidr and use-forwarded-headers. You may also set compute-full-forwarded-for.

It only remains to figure out the name of the ConfigMap when ingress is installed with microk8s enable ingress. It is named nginx-load-balancer-microk8s-conf :

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-load-balancer-microk8s-conf
  namespace: ingress
data:
  proxy-real-ip-cidr: 172.31.0.0/16
  use-forwarded-headers: "true"

Can I have a serviceless Ingress

I wanted to implement a 302 redirect the other day. Normally you run a nginx (or other web server) where you setup the server configuration and give a directive like

server {
  listen 0.0.0.0;
  server_name _;
  root /app;
  index index.htm index.html;
  return 301 $scheme://new_web_server_here$request_uri;
}

So I set about doing that, but thought that it would mean I am running an extra pod, without much need to do so. Would it be possible to run it via the Ingress controller directly? Yes it is possible to do so, since if you do not specify a backend in the nginx ingress controller, it falls back to the default backend and you can affect the ingress behavior with entering a code snippet:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: not-nginx
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      return 302 https://new_web_server_here$request_uri;
spec:
  tls:
  - hosts:
    - old_name_here
    secretName: secret_name_here
  rules:
  - host: old_name_here

(Do not copy-paste, check indentation as sometimes WordPress mangles it)

Depending your setup, you may need to run a separate nginx instance, as snippets can create a security issue in a cluster where many users can configure Ingress objects.

I’d rather deploy on a Sunday

Originally posted on LinkedIn, but also saved here in hopes of longer posterity:

When you deploy you change the behavior of the system. You wouldn’t be deploying if you didn’t want to change it in the first place.

I don’t deploy on Fridays, not because I’m afraid of the technical implications -we know for years how to manage these- of a deployment gone wrong, but of the business implications and outsider systems dependencies that operate in degraded mode during the weekend. It is not whether the Ops or Dev on call can handle the things. It is whether the other side can and the faith you have they can. Or that you (or they) have a business person on call.

I’d rather deploy on a Sunday

postgrest: Column of relation does not exist

If you are using postgrest and you are getting an error of the form:

Column 'column_name' of relation 'table_name' does not exist

Restart postgrest. You will notice that it says

Schema cache loaded

Which means that if you changed the table definition after postgrest started, it will not be able to write to it, unless restarted and re-reading the table definition.

I found out about it while trying to send graylog alerts to a Postgres database using HTTP Alert Notifications.

You can do even more interesting stuff if instead of postgrest you use logstash.