Tag: terraform

When terraform requires an IP address but what you have is a DNS name

I needed to expose for a bit an ElastiCache via a Network load balancer. To do so at a point in time you need to create a aws_lb_target_group_attachment. In such cases the target_id needs to be an IP address. 

resource "aws_lb_target_group_attachment" "redis" {
  target_group_arn = aws_lb_target_group.redis.arn
  target_id        = aws_elasticache_replication_group.redis.primary_endpoint_address
}

Now the primary_endpoint_address is a DNS name and not an IP, and what’s more, you cannot get by by thinking, OK it is a hostname, but eventually it will resolve into an IP to be used, no it expects an IP address. So we have to have a level of indirection here to figure it out. dns_a_record_set to the rescue:

data "dns_a_record_set" "redis" {
  host = aws_elasticache_replication_group.redis.primary_endpoint_address
}

However, keep in mind that dns_a_record_set returns a list and not a single record, so it still cannot be used, even if the query returns a single record. And you end up with something like this:

resource "aws_lb_target_group_attachment" "redis" {
  target_group_arn = aws_lb_target_group.redis.arn
  target_id        = data.dns_a_record_set.redis.addrs[0]
}

Understanding the terraform HTTP backend

Terraform is one of the de-facto infrastructure-as-code tools out there. It keeps track of the described infrastructure state and tries to help you not shoot yourself in the foot. It offers a wide variety of backends to store the state, the most popular ones being S3 and keeping state locally (as a beginner). Among the options provided is for example a Postgres backend. However lately my friend Evangelos Balaskas reminded me of the more generic HTTP backend, with an excellent blog post that combines GitLab and the HTTP backend.

I wanted to know more about the HTTP backend for some time now and I thought I should try to write a toy implementation to figure it out. I’ve never written any serious code with Golang and it seemed like a good oportunity to do something more than Go by example.

This is the simplest terraform HTTP backend configuration that assumes a web server running locally:

terraform {
  backend "http" {
    address = "http://127.0.0.1:8090/tf"
    lock_address = "http://127.0.0.1:8090/tf"
    unlock_address = "http://127.0.0.1:8090/tf"
  }
}

The simplest web server that we could write in Golang has to be something like

package main

import (
    "fmt"
    "net/http"
)

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "hello\n")
}

func main() {
    http.HandleFunc("/", hello_terraform)
    http.ListenAndServe(":8090", nil)
}

Running terraform init results in the following error message:

 Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: 2 problems:

- Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
- Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

And ever if we modified the terraform_hello() function to return an empty JSON string we would still get an error about the version attribute. Note that we’d get the same errors with a simple nginx instead of our golang HTTP server too (and this is because we send back a 200 response which terraform cannot parse instead of a 404 or other that would make it think this is a new state). Changing however the function just a bit

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "{"version": 0\n")
}

gives:

Error refreshing state: Unsupported state file format: The state file uses JSON syntax but has a version number of zero. There was never a JSON-based state format zero, so this state file is invalid and cannot be processed.

So we change the version number to 1 and:

PS C:\Users\...\terraform-http-backend> .\terraform.exe init
Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

But we can even have the function serve a pre-existing file on disk:

:
switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
        default:
           return
:

This also has the added value that we are not served with an error about the JSON file; if the file to be served does not exist terraform will get a HTTP 404 and it knows how to handle it.

Terraform is now convinced that somwhere there’s an HTTP server storing its state. In fact you could say that we have just implemented a /dev/null terraform backend.

But we sure want more than that. We want our backend to be able to create, store, serve and update the state file whenever we access it. A function that does this would start looking like:

func hello_terraform(w http.ResponseWriter, req *http.Request) {

	fmt.Printf("%v %v\n", req.Method, req.URL.Path)

	switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
	case "POST":
	   body, _ := ioutil.ReadAll(req.Body)
           b1 :=[]byte(string(body))
           os.WriteFile("tf.state", b1, 0644)
        case "LOCK":
            return
        case "UNLOCK":
            return
	default:
	    /* code */
	    return
	}
}

There: 20 lines of code and stuff I learned from Go by example and you have the beginning of a working terraform HTTP backend. You can remove the fmt.Printf() line as it is there just to show you the HTTP dialogue that takes place. A proper application, using a web framework will deal with this better.

You can find a bigger example, based on the above and far from complete here:

https://git.sr.ht/~adamo/terraform-http-backend

You can expand from here and add whatever functionality you like, like database support, some locking mechanism of your liking, versioning via git or other system, RBAC, anything.

terraform, route53 and lots of records

At work we try to manage as much as we can with terraform. This also includes Route53 for zones and records. In a certain situation we had about 14 zones and 1476 records managed in a single state file.

As it happened I needed a zone recreated (but not erased) and this affected about 409 records. Well deleting them with terraform apply took ages. To the point that the temporary STS token expired and botched the process.  So after a little facepalming, I decided to cleanup the zone from the AWS console and then issue a batch of terraform state rm to reconcile the state. Happily, after that, apply took its time (but reasonably) and all was well.

I am thinking that next time I am faced with such a situation, to lock the state file in Dynamo, copy it over from S3, manipulate it locally, unlock and run a plan to see how it all plays out. Or, wherever I can, use a state per zone instead of a state file encompassing a set of zones.