I’ve stopped using docker-compose

docker-compose is a very handy tool when you want to run multi container installations. Using a very simple YAML description, you can develop stuff locally and then push upstream whenever you feel something needs to enter the CI/CD cycle.

Sometimes, I’ve even used it on production, when I wanted to coordinate certain containers on a single VM. But lately I’ve stopped doing so. The reason is simple:

All that we do is ultimately going to be deployed in a Kubernetes cluster somewhere.

Given the above, there’s no need to maintain two sets of YAMLs, one for the docker-compose.yaml and one for the Kubernetes / helm manifests. Just go with Kubernetes from the beginning. Run a cluster on your local machine (Docker Desktop, microk8s, or other) and continue from there. Otherwise you risk running into the variation of works on my machine that is phrased like but it works with docker-compose. Well, there’s no docker-compose in production, why should there be on your machine? Plus you’ll get a sense of how things look like in production.

If you’re so much used to working with docker-compose, you can start a very crude transition by assuming that you have a single deployment and every container that you were to deploy is a side-car container to a single Pod. Afterall, just like a Pod, any docker-compose execution cannot escape a single machine (yes I know about Swarm). Then you can break it down to different deployments per container you want to run.

The above occured to me when I was trying to deploy some software locally, before deploying on Kubernetes, and tried to follow the vendor instructions for docker-compose. They failed and I lost quite some time trying to fix the provided YAML, and it dawned me: I do not need it. I need to test in Kubernetes anyway.

So there, stop using docker-compose when you can. Everyone will be happier.

podman completion zsh

I removed Docker Desktop from my machine in favor of podman. Nothing wrong with Docker Desktop, it still rocks, but I own a copy of Podman in Action, so I need to go through with it. As anyone with zsh who follows fashion, I am using oh my zsh!, so I wanted to enable podman’s command line completion to it. It turns out, others had the same issue some years back and the solution is:

mkdir -p $ZSH_CUSTOM/plugins/podman/
podman completion zsh -f $ZSH_CUSTOM/plugins/podman/_podman

In the end, don’t forget to enable the plugin in your ~/.zshrc In my case for example, I enable the following plugins:

plugins=(git fzf direnv brew tmux golang kubectl helm podman)

Eliza

Today’s paper of the week from Fermat’s Library is about Eliza by J. Weizenbaum. Eliza is a very early natural language processing system, that can mimic a Rogerian pshychologist‘s tactics and thus feel like a very sentient program within its context. Like many of my age, I was first introduced to Eliza via a game: a friend had an Amiga and a copy of it and we played around trying to make it tell us stuff based on information we were feeding it. It looked really amazing at the time.

Also like many others, I come across Eliza every few years and have toyed with an implementation. This is why this podcast on Eliza’s lineage is very interesting, because it shows from where you most likely learned to implement Eliza. Do listen to it. It contains a very interesting observation from a person who has actually read Weizenbaum’s paper:

Its name was chosen to emphasize that it nmy be incrementally improved by its users, since its language abilities may be continually improved by a “teacher”.

ELIZA A Computer Program For the Study of Natural Language Communication Between Man And Machine

So what most people overlook is that the original Eliza included a training mode, much like (in an abstract way at least) current chatbots that are all the craze do.

Weizenbaum himself was disturbed by the acceptance of Eliza and the anthropomorphic effect it had on people. He wrote Computer Power and Human Reason (a book that I need to skim through sometime given that I’ve read McCarthy’s refutation) to point out the issues he thought were important and was a fierce critic of AI in his later life, to the point of being marginalized in conferences where he appeared to preach his warnings. 99 Percent Invisible has an episode on him which you may also find interesting.

One goal for an augmented ELIZA program is thus a system which
already has access to a store of information about some aspect of the
real world and which, by means of conversational interaction with
people, can reveal both what it knows, i.e. behave as an information
retrieval system, and where its knowledge ends and needs to be
augmented. Hopefully the augmentation of its knowledge will also be
a direct consequence of its conversational experience. It is precisely
the prospect that such a program will converse with many people and
learn something from each of them which leads to the hope that it
will prove an interesting and even useful conversational partner.

ELIZA A Computer Program For the Study of Natural Language Communication Between Man And Machine

Too bad he didn’t successfully pursue this goal; no one else has. I think
success would have required a better understanding of formalization than
is exhibited in the book.

Defending AI Research

When terraform requires an IP address but what you have is a DNS name

I needed to expose for a bit an ElastiCache via a Network load balancer. To do so at a point in time you need to create a aws_lb_target_group_attachment. In such cases the target_id needs to be an IP address.

resource "aws_lb_target_group_attachment" "redis" {
  target_group_arn = aws_lb_target_group.redis.arn
  target_id        = aws_elasticache_replication_group.redis.primary_endpoint_address
}

Now the primary_endpoint_address is a DNS name and not an IP, and what’s more, you cannot get by by thinking, OK it is a hostname, but eventually it will resolve into an IP to be used, no it expects an IP address. So we have to have a level of indirection here to figure it out. dns_a_record_set to the rescue:

data "dns_a_record_set" "redis" {
  host = aws_elasticache_replication_group.redis.primary_endpoint_address
}

However, keep in mind that dns_a_record_set returns a list and not a single record, so it still cannot be used, even if the query returns a single record. And you end up with something like this:

resource "aws_lb_target_group_attachment" "redis" {
  target_group_arn = aws_lb_target_group.redis.arn
  target_id        = data.dns_a_record_set.redis.addrs[0]
}

My .wslconfig for Docker Desktop

I was running some tests within my Docker Desktop and it started not responding. I restarted it and it failed to start. So I thought, OK this is a resource problem. I opened the panel to configure the resources. But since it is integrated with WSL2, it said it needed a .wslconfig file. I’d never configured that before and where would I place it? It resides in %PROFILE%\.wslconfig thankfully and my quick googling around revealed a set of options that made it work:

[wsl2]
memory=8GB
processors=4
swap=8GB
pageReporting=false
localhostforwarding=true
nestedVirtualization=false

FastAPI, PyMongo and ‘ObjectId’ object is not iterable.

If you’re working with FastAPI and PyMongo, it is very likely that you are greeted with:

ValueError: [TypeError("'ObjectId' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

As you may know, every document in MongoDB has an _id attribute (a primary key). When you retrieve a document with PyMongo and you return the dictionary, FastAPI cannot serialize _id (which is of type bson.objectid.ObjectId). If you look around there are a number of solutions to this. Check them out, as they might be more appropriate for your case. However, one particular workaround, namely doing a document.pop('_id', None) before returning the dictionary got me thinking. So for my particular case I tried:

:
id = bucket_list.insert_one(document).inserted_id
# document.pop('_id', None)
document["_id"] = f'{id}'
:

which worked find for my use-case.

A first excursion in Q

I think I first heard of Kdb (precursor to kdb+) back in 1998 thanks to my friend PX. It claimed to be the fastest database in the block, but the K language and the fact that it aimed an industry not close to my interests made me download it a couple of times, fire up the 32bit interpreter and then go on with life. Fast forward some years and I learn that KX was aquired and that in the mean time Q was in the system to make it more accessible. In fact Q is written in K and you can read the implementation if you download the evaluation version. My interest was rekindled thanks to this arraycast episode and I ordered Fun Q. While waiting for the book, I watched the nine first videos from “An introduction to Kdb+” and tried to think whether I can write anything fancy with so little study.

Two simple and basic programs came to mind, FizzBuzz and the Collatz conjecture function. Could I do it with this little exposure to the language and some googling around? It turns that there are more articles about Q/Kdb+ than I expected on the net, so it was possible to figure things out. Let’s see:

Here is my take on the FizzBuzz:

{f:0=x mod 3;b:0=x mod 5;fb:f&b;$[1=fb;"fizzbuzz";1=f;"fizz";1=b;"buzz";x]} '[1+til 35]
1
2
"fizz"
4
"buzz"
"fizz"
7
8
"fizz"
"buzz"
11
"fizz"
13
14
"fizzbuzz"
16
17
"fizz"
19
"buzz"
"fizz"
22
..

I wanted to make a version like this in Python print(((x mod 3 == 0) * "fizz") + ((x mod 5 == 0) * "buzz") or x) but you can’t multiply strings with booleans in Q (or you can, but I have not figured it out yet). So I worked it out with a switch statement instead. The difficult part for me was figuring out the “execute the function for each of the numbers from 1 to 100” (the '[1+til 100] part of the expression).

To write the Collatz function, you do something like {$[0 = x mod 2; x%2; (1+3*x)]}[45] which finds the next term in the sequence starting from 45. But we want to make it run recursively. Q has the / and \ operators for this, and thus a run could be like {$[0 = x mod 2; x%2; (1+3*x)]}\[45] which is supposed to feed each run’s output to the next run and also print the intermediate returns so that we see the sequence build. However, since the run does not converge we will never see an ouput and we need something to limit the runs to a finite number, which allows us to see the sequence:

q){$[0 = x mod 2; x%2; (1+3*x)]}\[18;45]
45
136
68f
34f
17f
52f
26f
13f
40f
20f
10f
5f
16f
8f
4f
2f
1f
4f
2f
q)

I do boolean expression comparisons with 0 = x mod 2 instead of x mod = 0 because q has no precedence and evaluates from right to left. So the x mod 2 = 0 is equal to x mod (2 = 0) which is definitely not the comparison I wanted. That is why this gives 0N as a result. In Q you’re supposed to code golf a bit and I believe that is why 0 = x mod 2 is preferred to (x mod 2)=0.

I really do not know the proper idiomatic way to write this stuff yet, but I hope I stay engaged long enough to find out.

Update: After posting this blog, I googled “q language fizzbuzz” and came accross the Kx solution. I read it between the TV commercials and here is what I came up with:

{fb:sum 1 2*0=x mod/:3 5;$[fb=0;x;fb=1;"fizz";fb=2;"buzz";"fizzbuzz"]}'[1+til 17]

Update 2: Assuming we consider reaching to 1 a stop condition for the Collatz function, then using the convergence feature for scanning we can have the following function in order to compute the sequence: -1 _ {$[1=x;1;0=x mod 2;x%2;1+3*x]}\[18]. We use the drop last item (-1 _) because the scan prints the 1 twice (since convergence stops the iteration until two consecutive iterations return the same result; plus or minus some delta I think).

PS: You may also want to read about the new data platform from the reclusive genius of banking IT

Remembering DK; three stories

The untimely death of old colleague DK found me far away and I was not in a position to attend the service. So in order to honor my friend (he _was_ a friend despite that we did not meet as friends do; he cared) I will remember three stories of him and me:

The weird traceroute

This must have happened around 1998 or 1999. DK was working at the UoA NOC and I was working at the NTUA NOC. He observed that traceroutes to http://www.xoom.com were blocked but tracert (from Windows machines) was working. He mentioned this in the mailing list and he sent me down a rabbit hole to learn stuff. I’d not have bothered had he not asked. He was that kind of guy, he asked and you felt the need to help him out.

Sun’s screen

Still before 2000, this is a time when people write greeklish because different operating systems and window systems treat Greek differently and the path of least resistance is Greeklish. To assist this path of least resistance I had modified the code of script.c in order to translate Greek text to greeklish on the terminal. This worked fine with Linux and any BSD system. Dimitris complained that it did not work with Solaris. Back in the day, you had to have a special license to have access to Solaris code. Well, DK had, and he sent me the required files, I sent him back my version and he promptly sent be back a patch to improve this even further.

Goldratt’s simulation

If you’ve read The Goal, you know there’s an interesting simulation described there. There are a few blog posts and papers dealing with this, one of them being A Simulation Based on Goldratt’s Matchstick/Die Game. I had no access and I wanted to read it. DK had and he shared his copy with me.

This is how we reconnected on twitter after some years

Farewell my friend. It was me who was proud to be talking with you from time to time; I learned stuff.

Understanding the terraform HTTP backend

Terraform is one of the de-facto infrastructure-as-code tools out there. It keeps track of the described infrastructure state and tries to help you not shoot yourself in the foot. It offers a wide variety of backends to store the state, the most popular ones being S3 and keeping state locally (as a beginner). Among the options provided is for example a Postgres backend. However lately my friend Evangelos Balaskas reminded me of the more generic HTTP backend, with an excellent blog post that combines GitLab and the HTTP backend.

I wanted to know more about the HTTP backend for some time now and I thought I should try to write a toy implementation to figure it out. I’ve never written any serious code with Golang and it seemed like a good oportunity to do something more than Go by example.

This is the simplest terraform HTTP backend configuration that assumes a web server running locally:

terraform {
  backend "http" {
    address = "http://127.0.0.1:8090/tf"
    lock_address = "http://127.0.0.1:8090/tf"
    unlock_address = "http://127.0.0.1:8090/tf"
  }
}

The simplest web server that we could write in Golang has to be something like

package main

import (
    "fmt"
    "net/http"
)

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "hello\n")
}

func main() {
    http.HandleFunc("/", hello_terraform)
    http.ListenAndServe(":8090", nil)
}

Running terraform init results in the following error message:

 Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: 2 problems:

- Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
- Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

And ever if we modified the terraform_hello() function to return an empty JSON string we would still get an error about the version attribute. Note that we’d get the same errors with a simple nginx instead of our golang HTTP server too (and this is because we send back a 200 response which terraform cannot parse instead of a 404 or other that would make it think this is a new state). Changing however the function just a bit

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "{"version": 0\n")
}

gives:

Error refreshing state: Unsupported state file format: The state file uses JSON syntax but has a version number of zero. There was never a JSON-based state format zero, so this state file is invalid and cannot be processed.

So we change the version number to 1 and:

PS C:\Users\...\terraform-http-backend> .\terraform.exe init
Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

But we can even have the function serve a pre-existing file on disk:

:
switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
        default:
           return
:

This also has the added value that we are not served with an error about the JSON file; if the file to be served does not exist terraform will get a HTTP 404 and it knows how to handle it.

Terraform is now convinced that somwhere there’s an HTTP server storing its state. In fact you could say that we have just implemented a /dev/null terraform backend.

But we sure want more than that. We want our backend to be able to create, store, serve and update the state file whenever we access it. A function that does this would start looking like:

func hello_terraform(w http.ResponseWriter, req *http.Request) {

	fmt.Printf("%v %v\n", req.Method, req.URL.Path)

	switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
	case "POST":
	   body, _ := ioutil.ReadAll(req.Body)
           b1 :=[]byte(string(body))
           os.WriteFile("tf.state", b1, 0644)
        case "LOCK":
            return
        case "UNLOCK":
            return
	default:
	    /* code */
	    return
	}
}

There: 20 lines of code and stuff I learned from Go by example and you have the beginning of a working terraform HTTP backend. You can remove the fmt.Printf() line as it is there just to show you the HTTP dialogue that takes place. A proper application, using a web framework will deal with this better.

You can find a bigger example, based on the above and far from complete here:

https://git.sr.ht/~adamo/terraform-http-backend

You can expand from here and add whatever functionality you like, like database support, some locking mechanism of your liking, versioning via git or other system, RBAC, anything.

–no-title

I am subscribed to the Fermat’s Library newsletter. Their paper of the week is The Economic Organization of a P.O.W. Camp. The author, R. A. Radford, observed the life of the prisoners and their economic interactions. I guess this is a coping mechanism in a way. Use what you know to understand the situation you are in and survive the best way you can.

Radford’s paper and the use of cigarettes as currency reminded me also of the use of rum as currency in 1790 Australia (the most popular form in fact). Which also came with its own set of problems: Farmers had no incentive to produce crops only to be paid in rum. They had to pay workers in rum and with many workers drinking their pay, productivity was lowered.

It reminds me also of the book Games Prisoners Play, where its author Marek M. Kaminsky, while imprisoned in Communist Poland, started mapping the dynamics between groups of prisoners and analysing them using Game Theory.