A first excursion in Q

I think I first heard of Kdb (precursor to kdb+) back in 1998 thanks to my friend PX. It claimed to be the fastest database in the block, but the K language and the fact that it aimed an industry not close to my interests made me download it a couple of times, fire up the 32bit interpreter and then go on with life. Fast forward some years and I learn that KX was aquired and that in the mean time Q was in the system to make it more accessible. In fact Q is written in K and you can read the implementation if you download the evaluation version. My interest was rekindled thanks to this arraycast episode and I ordered Fun Q. While waiting for the book, I watched the nine first videos from “An introduction to Kdb+” and tried to think whether I can write anything fancy with so little study.

Two simple and basic programs came to mind, FizzBuzz and the Collatz conjecture function. Could I do it with this little exposure to the language and some googling around? It turns that there are more articles about Q/Kdb+ than I expected on the net, so it was possible to figure things out. Let’s see:

Here is my take on the FizzBuzz:

{f:0=x mod 3;b:0=x mod 5;fb:f&b;$[1=fb;"fizzbuzz";1=f;"fizz";1=b;"buzz";x]} '[1+til 35]
1
2
"fizz"
4
"buzz"
"fizz"
7
8
"fizz"
"buzz"
11
"fizz"
13
14
"fizzbuzz"
16
17
"fizz"
19
"buzz"
"fizz"
22
..

I wanted to make a version like this in Python print(((x mod 3 == 0) * "fizz") + ((x mod 5 == 0) * "buzz") or x) but you can’t multiply strings with booleans in Q (or you can, but I have not figured it out yet). So I worked it out with a switch statement instead. The difficult part for me was figuring out the “execute the function for each of the numbers from 1 to 100” (the '[1+til 100] part of the expression).

To write the Collatz function, you do something like {$[0 = x mod 2; x%2; (1+3*x)]}[45] which finds the next term in the sequence starting from 45. But we want to make it run recursively. Q has the / and \ operators for this, and thus a run could be like {$[0 = x mod 2; x%2; (1+3*x)]}\[45] which is supposed to feed each run’s output to the next run and also print the intermediate returns so that we see the sequence build. However, since the run does not converge we will never see an ouput and we need something to limit the runs to a finite number, which allows us to see the sequence:

q){$[0 = x mod 2; x%2; (1+3*x)]}\[18;45]
45
136
68f
34f
17f
52f
26f
13f
40f
20f
10f
5f
16f
8f
4f
2f
1f
4f
2f
q)

I do boolean expression comparisons with 0 = x mod 2 instead of x mod = 0 because q has no precedence and evaluates from right to left. So the x mod 2 = 0 is equal to x mod (2 = 0) which is definitely not the comparison I wanted. That is why this gives 0N as a result. In Q you’re supposed to code golf a bit and I believe that is why 0 = x mod 2 is preferred to (x mod 2)=0.

I really do not know the proper idiomatic way to write this stuff yet, but I hope I stay engaged long enough to find out.

Update: After posting this blog, I googled “q language fizzbuzz” and came accross the Kx solution. I read it between the TV commercials and here is what I came up with:

{fb:sum 1 2*0=x mod/:3 5;$[fb=0;x;fb=1;"fizz";fb=2;"buzz";"fizzbuzz"]}'[1+til 17]

Update 2: Assuming we consider reaching to 1 a stop condition for the Collatz function, then using the convergence feature for scanning we can have the following function in order to compute the sequence: -1 _ {$[1=x;1;0=x mod 2;x%2;1+3*x]}\[18]. We use the drop last item (-1 _) because the scan prints the 1 twice (since convergence stops the iteration until two consecutive iterations return the same result; plus or minus some delta I think).

PS: You may also want to read about the new data platform from the reclusive genius of banking IT

Remembering DK; three stories

The untimely death of old colleague DK found me far away and I was not in a position to attend the service. So in order to honor my friend (he _was_ a friend despite that we did not meet as friends do; he cared) I will remember three stories of him and me:

The weird traceroute

This must have happened around 1998 or 1999. DK was working at the UoA NOC and I was working at the NTUA NOC. He observed that traceroutes to http://www.xoom.com were blocked but tracert (from Windows machines) was working. He mentioned this in the mailing list and he sent me down a rabbit hole to learn stuff. I’d not have bothered had he not asked. He was that kind of guy, he asked and you felt the need to help him out.

Sun’s screen

Still before 2000, this is a time when people write greeklish because different operating systems and window systems treat Greek differently and the path of least resistance is Greeklish. To assist this path of least resistance I had modified the code of script.c in order to translate Greek text to greeklish on the terminal. This worked fine with Linux and any BSD system. Dimitris complained that it did not work with Solaris. Back in the day, you had to have a special license to have access to Solaris code. Well, DK had, and he sent me the required files, I sent him back my version and he promptly sent be back a patch to improve this even further.

Goldratt’s simulation

If you’ve read The Goal, you know there’s an interesting simulation described there. There are a few blog posts and papers dealing with this, one of them being A Simulation Based on Goldratt’s Matchstick/Die Game. I had no access and I wanted to read it. DK had and he shared his copy with me.

This is how we reconnected on twitter after some years

Farewell my friend. It was me who was proud to be talking with you from time to time; I learned stuff.

Understanding the terraform HTTP backend

Terraform is one of the de-facto infrastructure-as-code tools out there. It keeps track of the described infrastructure state and tries to help you not shoot yourself in the foot. It offers a wide variety of backends to store the state, the most popular ones being S3 and keeping state locally (as a beginner). Among the options provided is for example a Postgres backend. However lately my friend Evangelos Balaskas reminded me of the more generic HTTP backend, with an excellent blog post that combines GitLab and the HTTP backend.

I wanted to know more about the HTTP backend for some time now and I thought I should try to write a toy implementation to figure it out. I’ve never written any serious code with Golang and it seemed like a good oportunity to do something more than Go by example.

This is the simplest terraform HTTP backend configuration that assumes a web server running locally:

terraform {
  backend "http" {
    address = "http://127.0.0.1:8090/tf"
    lock_address = "http://127.0.0.1:8090/tf"
    unlock_address = "http://127.0.0.1:8090/tf"
  }
}

The simplest web server that we could write in Golang has to be something like

package main

import (
    "fmt"
    "net/http"
)

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "hello\n")
}

func main() {
    http.HandleFunc("/", hello_terraform)
    http.ListenAndServe(":8090", nil)
}

Running terraform init results in the following error message:

 Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: 2 problems:

- Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
- Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

And ever if we modified the terraform_hello() function to return an empty JSON string we would still get an error about the version attribute. Note that we’d get the same errors with a simple nginx instead of our golang HTTP server too (and this is because we send back a 200 response which terraform cannot parse instead of a 404 or other that would make it think this is a new state). Changing however the function just a bit

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "{"version": 0\n")
}

gives:

Error refreshing state: Unsupported state file format: The state file uses JSON syntax but has a version number of zero. There was never a JSON-based state format zero, so this state file is invalid and cannot be processed.

So we change the version number to 1 and:

PS C:\Users\...\terraform-http-backend> .\terraform.exe init
Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

But we can even have the function serve a pre-existing file on disk:

:
switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
        default:
           return
:

This also has the added value that we are not served with an error about the JSON file; if the file to be served does not exist terraform will get a HTTP 404 and it knows how to handle it.

Terraform is now convinced that somwhere there’s an HTTP server storing its state. In fact you could say that we have just implemented a /dev/null terraform backend.

But we sure want more than that. We want our backend to be able to create, store, serve and update the state file whenever we access it. A function that does this would start looking like:

func hello_terraform(w http.ResponseWriter, req *http.Request) {

	fmt.Printf("%v %v\n", req.Method, req.URL.Path)

	switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
	case "POST":
	   body, _ := ioutil.ReadAll(req.Body)
           b1 :=[]byte(string(body))
           os.WriteFile("tf.state", b1, 0644)
        case "LOCK":
            return
        case "UNLOCK":
            return
	default:
	    /* code */
	    return
	}
}

There: 20 lines of code and stuff I learned from Go by example and you have the beginning of a working terraform HTTP backend. You can remove the fmt.Printf() line as it is there just to show you the HTTP dialogue that takes place. A proper application, using a web framework will deal with this better.

You can find a bigger example, based on the above and far from complete here:

https://git.sr.ht/~adamo/terraform-http-backend

You can expand from here and add whatever functionality you like, like database support, some locking mechanism of your liking, versioning via git or other system, RBAC, anything.

–no-title

I am subscribed to the Fermat’s Library newsletter. Their paper of the week is The Economic Organization of a P.O.W. Camp. The author, R. A. Radford, observed the life of the prisoners and their economic interactions. I guess this is a coping mechanism in a way. Use what you know to understand the situation you are in and survive the best way you can.

Radford’s paper and the use of cigarettes as currency reminded me also of the use of rum as currency in 1790 Australia (the most popular form in fact). Which also came with its own set of problems: Farmers had no incentive to produce crops only to be paid in rum. They had to pay workers in rum and with many workers drinking their pay, productivity was lowered.

It reminds me also of the book Games Prisoners Play, where its author Marek M. Kaminsky, while imprisoned in Communist Poland, started mapping the dynamics between groups of prisoners and analysing them using Game Theory.

TIL: a bit more of YAML and kubectl

For some reason I needed a running pod to be able to access the value of its own container image(s). I initially thought of the Downward API but while it can give you the pod’s name, IP and other stuff, and even access resource limits per container using their name, the image is one of the attributes that is not immediately available. But the Downward API allows for metadata annotations to be available. So one can define a pod like:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    nginx_container: bitnami/nginx:1.23.2
  labels:
    app: nginx
  name: nginx
spec:
  containers:
  - image: bitnami/nginx:1.23.2
    name: nginx
    env:
    - name: IMAGE
      valueFrom:
        fieldRef:
          fieldPath: metadata.annotations['nginx_container']

But this has the drawback of defining the image value in two places in the same file. Unless you can template this out with helm, or you generate the YAML via some other automation, this is an invitation for a bug the time when you will forget to update both places. Is there a better way?

Of course there is, and I might have seen it sooner had I not been fixed on utilizing the Downward API. The answer is YAML anchors and aliases:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  containers:
  - image: &nginx_image bitnami/nginx:1.23.2
    name: nginx
    env:
    - name: IMAGE
      value: *nginx_image

“You cannot always work with the best”

It has been a while since I last blogged, and I find myself repeating the following story in discussions lately, so why not put it down here too:

When I was serving at the Air Force, for some reason we got into a discussion with the Warrant Officer in charge about manning the posts and specific persons on location. And he told me:

George, you cannot always work with the best

This struck me very hard, because up until that time, I was very used to work with "the best" in environments that allowed for rapid personal growth and learning and had not internalised my luck of having world class scientists allowing me to work with them.

The second reason why this struck me was because of Joy’s Law

no matter who you are, most of the smartest people work for someone else

and all this was happening in a discussion about guard duty.

A different kind of life

Williams: A Different Kind of LifeWilliams: A Different Kind of Life by Virginia Williams
My rating: 5 of 5 stars

This is a very important book regardless of whether you are a F1 fan or not. Actually there is very little F1 racing in the book. This is a book about pivotal life changes. It describes the life Virginia Williams had with Frank before and after the accident that left him paralyzed from the neck down.

As such this book is about caregivers; the untrained, totally unprepared family members who are suddenly efforted to be a complete support infrastructure 24×7.

There are many books about disabled persons who overcame and thrived. There are none about caregivers and what it means to them.

I recognized every bit of hope, pain and frustration expressed in the book. Even to the point of (over)using painkillers as a crutch to make it though the day dealing with both the emotional and physical pain.

View all my reviews

TuneIn as a poor-man’s network alert

I used to get irked when colleagues streamed music from radio stations to their desktop at work, when they could have the same result with a cheap radio without eating up bandwidth. Nowadays not so much, all of them are remote, so do what you want in your space.

However, with the rise of digital assistants and the smart speakers you can cast a local station or even stream something from TuneIn (Alexa play Jazz24) and with home bandwidth improving you can get better sound quality than FM stereo.

And of course be alerted when your network is down. Because streaming stops. Just like in the old days when people sonified pings on the Ethernet or even actual traffic.

Yes, it is kind of a slow day today…

It’s a trap; say no

It happened that I switched jobs at the end of August. Upon hearing that this was about to happen, a friend pinged me and asked whether he could hire my services for "one to two hours per week". I rejected the offer. I specifically told my friend that "one hour per week is a trap":

A week is 168 hours. You sleep 56 of them, so let’s say that you have to give 1 hour out of 112. To be more precise, 1 hour out of 72 you can spare. Surely you can allow yourself 1 hour of a side-gig, right? But then it is Monday. And you start your working day, and "oh, I’ll deal with this tomorrow" and this goes on until Friday and you say, "I’ll do it Saturday morning before the other chores". And you squeeze that hour somewhat grudgingly, between Saturday and Sunday night.

If I’m wrong, why are then all those Udemy / whatever vocational courses you’ve purchased unfinished? Did interest wane down on all of them right after the first couple of hours?

So say no to "one hour per week engagements". You’re setting up yourself for frustration and failure.

As for me, I’ve arranged for virtual coffee with my friend once per month to discuss his vision and progress. Because everyone needs someone to talk to.

email is still your best backup social network

We communicate through a host of applications using our own devices. In my phone I count 29 installed. Granted, not all of them are in frequent use, but all of them are installed for a reason: Some contact of mine is using some application, be it much popular, or less, or even something like Amazon Chime which you need to converse with AWS people.

Sometimes, infrastructures supporting those applications suffer outages and depending the userbase, pain gets expressed in Twitter, Facebook or LinkedIn, or even internal jokes like "Slack is down, let’s increase productivity until it gets up".

So it occurred to me, that even though every 5 years or so we see a new application claiming to solve texting, email is still the best backup solution:

  • even though it is mostly controlled by 5 big walled gardens, they are not one garden and you can always run your own with some varying success of compatibility if you wish to undertake the pain.
  • while I used to complain about "email not being FTP", it is still used as a file transfer mechanism.
  • while I used to complain about top posting in replies (I used to prefer inline replies), regardless of how ugly it may seem at times, you have a complete log of the communication history.
  • you can use your email store as document database (yes it was not designed for this, I hated the fact, but let’s move on; people search their emails to find stuff they remember being there).
  • while I used to complain a lot about HTML email, you still have the ability to at least send as plain or flashy emails as you like and they will be received and read (in the majority of cases) as you expect.
  • you are not confined to a single application for reading and sending email. People still write mail clients.
  • almost any booking / purchasing system you use sends you an email of what you transacted about. They do not seek your Skype, Messenger, Discord or what. They fall back to your best backup social network: email.
  • while it is not instant messaging, as I still tell people, most emails arrive soon enough to feel like it is.
  • it can be a 1:1 communication medium, 1 to many (a newsletter), or even many to many (mailing lists, Reply All).
  • spammers and other cyber villains still attack through it, so it is in wide use, even if it is not the cool kid in town.

I’ve come to terms with this kind of underground success: It does everything almost good enough and in the background. It is nobody’s first choice, but it is everybody’s common denominator because of the good enough factor.

Now if only I could persuade my children to fill the Subject: line when emailing me…