Terraform and YAML related notes
—

If you manage applications, you likely come across a ton of structured data that stays mostly static but must be tracked, version-controlled and be visible, especially when someone modifies it. For some folks, that is a CMS, for others a stack of papyri and for some more it is just a long oral tradition of chanting or yelling. I, however, prefer readable, structured text files. That's why I like the idea of Terraform or infrastructure-as-code.
Last year, we faced a problem for management of some of our infrastructure-data. We wanted to manage the Consul KV store data via Terraform and not depend on Consul's UI. The problem was that there was no default Terraform provider for the particular task of manipulation of Consul KV store. There are provider resources like "consul_keys" and "consul_key_prefix" but they alone were not adequate for a cleaner solution. We have multiple datacenters and it is tiresome to edit keys and values when needed. The data had a ton of repetition across various datacenters with some common parts. Moreover, Web UI does not keep a record of edits. As a result, the visibility within the team was close to zero.
Given the lack of any such plugin, at that time, we went on a path to use Terraform's external data source and decided to store the KV data in a YAML file which will be read by the external data source and will populate the "consul_key_prefix". First, external data source is your custom program that reads from stdin and spits out on stdout, both as JSON objects. The official documentation for this "external data source" provider shows that you define it as

data "external" "example" {
  # Call "python /path/to/mod/example.py"
  program = [
      "python", 
      "${path.module}/example.py"
    ]

  # with JSON query
  query = {
    id = "abc123"
  }
}

In the problem that I stated above, it was decided to store the Consul Key/Value pairs as nested YAML file for each datacenter and a common section for common keys. Something like:

common:
    key_common_1: value_common_1

dc1:
    <<: *common

    key11: value11
    key12: value12
    key13:
        key131: value131

dc2:
    <<: *common

    key21: value21
    key22: value22
    key23:
        key231: value231

The wonderful part of YAML is that it is machine-and-human-readable, and can succinctly represent hierarchy in a document. Hence, it is well-suited for the representation of various datacenters and their respective KV data, which is what we wanted to do. Take the key "key131" in "Datacenter 1". You can depict the "path" to "key131" as Consul KV via dc1/key13/key131

Now, comes the question of what to do with external datasource. You can write that in any language/platform that can read from stdin and write to stdout. In the official documentation they use bash. However, I chose Python for its simplicity, good YAML support and JSON interoperability. Also, I could get help from some team members if I ever got stuck. With these tools, the last thing I want to add is that you can use Python's flatdict with delimiter="/" as

import flatdict


with open(file_path, 'r') as yf:
    yaml_dict = flatdict.FlatDict(yaml.load(yf), 
                                  delimiter='/')

This creates a flattened dictionary, each key of which is the "path" with segments delimited by "/". Example dc1/key13/key131

All that is left now is manipulation of the above dict if needed and writing JSON to stdout so it can be read by the Terraform resource "consul_key_prefix". 😇

resource "consul_key_prefix" "keys" {
  datacenter = "${var.datacenter}"
  path_prefix = "${var.prefix}${var.path}"
  subkeys = "${data.external.mydata.result}"
}

But, I am not done telling you about this yet. My friend and colleague Borys Pierov wrote new set of Terraform provider plugins because there was a need for a good Consul ACL management provider. He abstracted a bunch of stuff into independent plugins so you can go from flexible to powerful, if you want.

So, if you use Consul and you need to manage KV or ACL, you do not have any excuses to not use the above tools. Please let me know if I missed something or there are alternate approaches that you've taken. Thanks for reading. 😊