Terraform is one of the de-facto infrastructure-as-code tools out there. It keeps track of the described infrastructure state and tries to help you not shoot yourself in the foot. It offers a wide variety of backends to store the state, the most popular ones being S3 and keeping state locally (as a beginner). Among the options provided is for example a Postgres backend. However lately my friend Evangelos Balaskas reminded me of the more generic HTTP backend, with an excellent blog post that combines GitLab and the HTTP backend.

I wanted to know more about the HTTP backend for some time now and I thought I should try to write a toy implementation to figure it out. I’ve never written any serious code with Golang and it seemed like a good oportunity to do something more than Go by example.

This is the simplest terraform HTTP backend configuration that assumes a web server running locally:

terraform {
  backend "http" {
    address = "http://127.0.0.1:8090/tf"
    lock_address = "http://127.0.0.1:8090/tf"
    unlock_address = "http://127.0.0.1:8090/tf"
  }
}

The simplest web server that we could write in Golang has to be something like

package main

import (
    "fmt"
    "net/http"
)

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "hello\n")
}

func main() {
    http.HandleFunc("/", hello_terraform)
    http.ListenAndServe(":8090", nil)
}

Running terraform init results in the following error message:

 Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: 2 problems:

- Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
- Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

And ever if we modified the terraform_hello() function to return an empty JSON string we would still get an error about the version attribute. Note that we’d get the same errors with a simple nginx instead of our golang HTTP server too (and this is because we send back a 200 response which terraform cannot parse instead of a 404 or other that would make it think this is a new state). Changing however the function just a bit

func hello_terraform(w http.ResponseWriter, req *http.Request) {
    fmt.Fprintf(w, "{"version": 0\n")
}

gives:

Error refreshing state: Unsupported state file format: The state file uses JSON syntax but has a version number of zero. There was never a JSON-based state format zero, so this state file is invalid and cannot be processed.

So we change the version number to 1 and:

PS C:\Users\...\terraform-http-backend> .\terraform.exe init
Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

But we can even have the function serve a pre-existing file on disk:

:
switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
        default:
           return
:

This also has the added value that we are not served with an error about the JSON file; if the file to be served does not exist terraform will get a HTTP 404 and it knows how to handle it.

Terraform is now convinced that somwhere there’s an HTTP server storing its state. In fact you could say that we have just implemented a /dev/null terraform backend.

But we sure want more than that. We want our backend to be able to create, store, serve and update the state file whenever we access it. A function that does this would start looking like:

func hello_terraform(w http.ResponseWriter, req *http.Request) {

	fmt.Printf("%v %v\n", req.Method, req.URL.Path)

	switch req.Method {
	case "GET":
	   http.ServeFile(w, req, "tf.state")
	case "POST":
	   body, _ := ioutil.ReadAll(req.Body)
           b1 :=[]byte(string(body))
           os.WriteFile("tf.state", b1, 0644)
        case "LOCK":
            return
        case "UNLOCK":
            return
	default:
	    /* code */
	    return
	}
}

There: 20 lines of code and stuff I learned from Go by example and you have the beginning of a working terraform HTTP backend. You can remove the fmt.Printf() line as it is there just to show you the HTTP dialogue that takes place. A proper application, using a web framework will deal with this better.

You can find a bigger example, based on the above and far from complete here:

https://git.sr.ht/~adamo/terraform-http-backend

You can expand from here and add whatever functionality you like, like database support, some locking mechanism of your liking, versioning via git or other system, RBAC, anything.