Author: adamo

I am a Greek DevOps Engineer ranting about everything, from email to sports; to living in Athens, Greece.

microk8s, nginx ingress and X-Forwarded-For

Sometimes when you run microk8s in AWS, you may want to have an application load balancer in front. Such configurations mess around with the value of the header X-Forwarded-For regardless of whether the append attribute is present on the load balancer. By reading the nginx ingress documentation, you need to edit the ConfigMap resource and add proxy-real-ip-cidr and use-forwarded-headers. You may also set compute-full-forwarded-for.

It only remains to figure out the name of the ConfigMap when ingress is installed with microk8s enable ingress. It is named nginx-load-balancer-microk8s-conf :

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-load-balancer-microk8s-conf
  namespace: ingress
data:
  proxy-real-ip-cidr: 172.31.0.0/16
  use-forwarded-headers: "true"

Can I have a serviceless Ingress

I wanted to implement a 302 redirect the other day. Normally you run a nginx (or other web server) where you setup the server configuration and give a directive like

server {
  listen 0.0.0.0;
  server_name _;
  root /app;
  index index.htm index.html;
  return 301 $scheme://new_web_server_here$request_uri;
}

So I set about doing that, but thought that it would mean I am running an extra pod, without much need to do so. Would it be possible to run it via the Ingress controller directly? Yes it is possible to do so, since if you do not specify a backend in the nginx ingress controller, it falls back to the default backend and you can affect the ingress behavior with entering a code snippet:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: not-nginx
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      return 302 https://new_web_server_here$request_uri;
spec:
  tls:
  - hosts:
    - old_name_here
    secretName: secret_name_here
  rules:
  - host: old_name_here

(Do not copy-paste, check indentation as sometimes WordPress mangles it)

Depending your setup, you may need to run a separate nginx instance, as snippets can create a security issue in a cluster where many users can configure Ingress objects.

I’d rather deploy on a Sunday

Originally posted on LinkedIn, but also saved here in hopes of longer posterity:

When you deploy you change the behavior of the system. You wouldn’t be deploying if you didn’t want to change it in the first place.

I don’t deploy on Fridays, not because I’m afraid of the technical implications -we know for years how to manage these- of a deployment gone wrong, but of the business implications and outsider systems dependencies that operate in degraded mode during the weekend. It is not whether the Ops or Dev on call can handle the things. It is whether the other side can and the faith you have they can. Or that you (or they) have a business person on call.

I’d rather deploy on a Sunday

postgrest: Column of relation does not exist

If you are using postgrest and you are getting an error of the form:

Column 'column_name' of relation 'table_name' does not exist

Restart postgrest. You will notice that it says

Schema cache loaded

Which means that if you changed the table definition after postgrest started, it will not be able to write to it, unless restarted and re-reading the table definition.

I found out about it while trying to send graylog alerts to a Postgres database using HTTP Alert Notifications.

You can do even more interesting stuff if instead of postgrest you use logstash.

TIL: net/http/httptrace

I write a few lines of Go here and there every few months, therefore I am basically a hobbyist (even though serious work with Kubernetes demands more of it). Today a friend had a very valid question:

How can I know the source port of my HTTP request when using net/http?

Pavlos

This is a valid question. If I have a socket, I should know all four aspects of it (src IP, src port, dst IP, dst port) and protocol of course. But it turns out this is not quite possible with net/http and another friend suggested making your custom transport to have control over such unexported information.

I had a flash and I thought “It can’t be such that no-one has written something that traces an HTTP connection in Go!”. It turns I was right and net/http/httptrace is right there and you can get the information needed thanks to the net.Conn struct (pardon the missing error handling):

package main

import (
    "fmt"
    "net/http"
    "net/http/httptrace"
    "io/ioutil"
)

func main() {
    client := http.Client{}
    req, _ := http.NewRequest("GET", "https://jsonip.com", nil)
        
    trace := &httptrace.ClientTrace{
        GotConn: func(connInfo httptrace.GotConnInfo) {
            fmt.Printf("GotConn: %v %v\n", connInfo.Conn.LocalAddr(), connInfo.Conn.RemoteAddr())
        },
    }
  
    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
    res, _ := client.Do(req)
    resBody, _ := ioutil.ReadAll(res.Body)
    fmt.Printf("%s\n", resBody)
}

I get that this can be written in a better manner, but for now I am happy that I learned something new.

I’ve stopped using docker-compose

docker-compose is a very handy tool when you want to run multi container installations. Using a very simple YAML description, you can develop stuff locally and then push upstream whenever you feel something needs to enter the CI/CD cycle.

Sometimes, I’ve even used it on production, when I wanted to coordinate certain containers on a single VM. But lately I’ve stopped doing so. The reason is simple:

All that we do is ultimately going to be deployed in a Kubernetes cluster somewhere.

Given the above, there’s no need to maintain two sets of YAMLs, one for the docker-compose.yaml and one for the Kubernetes / helm manifests. Just go with Kubernetes from the beginning. Run a cluster on your local machine (Docker Desktop, microk8s, or other) and continue from there. Otherwise you risk running into the variation of works on my machine that is phrased like but it works with docker-compose. Well, there’s no docker-compose in production, why should there be on your machine? Plus you’ll get a sense of how things look like in production.

If you’re so much used to working with docker-compose, you can start a very crude transition by assuming that you have a single deployment and every container that you were to deploy is a side-car container to a single Pod. Afterall, just like a Pod, any docker-compose execution cannot escape a single machine (yes I know about Swarm). Then you can break it down to different deployments per container you want to run.

The above occured to me when I was trying to deploy some software locally, before deploying on Kubernetes, and tried to follow the vendor instructions for docker-compose. They failed and I lost quite some time trying to fix the provided YAML, and it dawned me: I do not need it. I need to test in Kubernetes anyway.

So there, stop using docker-compose when you can. Everyone will be happier.

podman completion zsh

I removed Docker Desktop from my machine in favor of podman. Nothing wrong with Docker Desktop, it still rocks, but I own a copy of Podman in Action, so I need to go through with it. As anyone with zsh who follows fashion, I am using oh my zsh!, so I wanted to enable podman’s command line completion to it. It turns out, others had the same issue some years back and the solution is:

mkdir -p $ZSH_CUSTOM/plugins/podman/
podman completion zsh -f $ZSH_CUSTOM/plugins/podman/_podman

In the end, don’t forget to enable the plugin in your ~/.zshrc In my case for example, I enable the following plugins:

plugins=(git fzf direnv brew tmux golang kubectl helm podman)

Eliza

Today’s paper of the week from Fermat’s Library is about Eliza by J. Weizenbaum. Eliza is a very early natural language processing system, that can mimic a Rogerian pshychologist‘s tactics and thus feel like a very sentient program within its context. Like many of my age, I was first introduced to Eliza via a game: a friend had an Amiga and a copy of it and we played around trying to make it tell us stuff based on information we were feeding it. It looked really amazing at the time.

Also like many others, I come across Eliza every few years and have toyed with an implementation. This is why this podcast on Eliza’s lineage is very interesting, because it shows from where you most likely learned to implement Eliza. Do listen to it. It contains a very interesting observation from a person who has actually read Weizenbaum’s paper:

Its name was chosen to emphasize that it nmy be incrementally improved by its users, since its language abilities may be continually improved by a “teacher”.

ELIZA A Computer Program For the Study of Natural Language Communication Between Man And Machine

So what most people overlook is that the original Eliza included a training mode, much like (in an abstract way at least) current chatbots that are all the craze do.

Weizenbaum himself was disturbed by the acceptance of Eliza and the anthropomorphic effect it had on people. He wrote Computer Power and Human Reason (a book that I need to skim through sometime given that I’ve read McCarthy’s refutation) to point out the issues he thought were important and was a fierce critic of AI in his later life, to the point of being marginalized in conferences where he appeared to preach his warnings. 99 Percent Invisible has an episode on him which you may also find interesting.

One goal for an augmented ELIZA program is thus a system which
already has access to a store of information about some aspect of the
real world and which, by means of conversational interaction with
people, can reveal both what it knows, i.e. behave as an information
retrieval system, and where its knowledge ends and needs to be
augmented. Hopefully the augmentation of its knowledge will also be
a direct consequence of its conversational experience. It is precisely
the prospect that such a program will converse with many people and
learn something from each of them which leads to the hope that it
will prove an interesting and even useful conversational partner.

ELIZA A Computer Program For the Study of Natural Language Communication Between Man And Machine

Too bad he didn’t successfully pursue this goal; no one else has. I think
success would have required a better understanding of formalization than
is exhibited in the book.

Defending AI Research

When terraform requires an IP address but what you have is a DNS name

I needed to expose for a bit an ElastiCache via a Network load balancer. To do so at a point in time you need to create a aws_lb_target_group_attachment. In such cases the target_id needs to be an IP address. 

resource "aws_lb_target_group_attachment" "redis" {
  target_group_arn = aws_lb_target_group.redis.arn
  target_id        = aws_elasticache_replication_group.redis.primary_endpoint_address
}

Now the primary_endpoint_address is a DNS name and not an IP, and what’s more, you cannot get by by thinking, OK it is a hostname, but eventually it will resolve into an IP to be used, no it expects an IP address. So we have to have a level of indirection here to figure it out. dns_a_record_set to the rescue:

data "dns_a_record_set" "redis" {
  host = aws_elasticache_replication_group.redis.primary_endpoint_address
}

However, keep in mind that dns_a_record_set returns a list and not a single record, so it still cannot be used, even if the query returns a single record. And you end up with something like this:

resource "aws_lb_target_group_attachment" "redis" {
  target_group_arn = aws_lb_target_group.redis.arn
  target_id        = data.dns_a_record_set.redis.addrs[0]
}