The Dhall configuration language as a different method for creating Kubernetes manifests
21.04.2023
3 minues
There is a certain language of programming that is called Dhall which was created in order to generate configuration files in many ways. This open-source project was first released in 2018 and had been getting popularity since then.
As a new language that generates configuration files, Dhall aims to address the limitations of existing formats such as TOML, YAML, XML, and JSON, when it’s still easy to use. Dhall introduced bindings for Kubernetes, which have further contributed to its popularity in 2020.
Before start discussing how Dhall makes Kubernetes manifest creation easier, it’s important to provide a brief overview of the language itself.
What is the difference between Dhal and other languages?
According to the authors, Dhall should be regarded as a more advanced version of JSON, complete with its own set of functions, types, and imports. However, this raises the question of why a new format is necessary when there are already established formats available.
Probably, that’s because?..
The primary motivation for developing Dhall, according to the authors, was the limited programmability of existing formats like JSON and YAML. This lack of programmability often restricts the capabilities of these formats and can result in suboptimal solutions, including repetitive code.
Direct attention toward DRY
Developers often follow the DRY principle (“Don’t Repeat Yourself”) as a best practice. However, working with JSON and YAML can make it challenging to avoid repetition due to their functional limitations. As a result, repetitive configuration fragments cannot be simplified or eliminated.
Dhall is being promoted as a language that helps developers adhere to the DRY principle. With Dhall, repetitive code blocks (which are common in JSON and YAML) can be replaced with function results or variable values. To illustrate this, Dhall’s documentation provides an example of two configuration files in JSON and Dhall formats, respectively. Both files have the same purpose of describing the location of public and private SSH user keys.
Below is the JSON file:
[
{
“privateKey”: “/home/john/.ssh/id_rsa”,
“publicKey”: “/home/john/.ssh/id_rsa.pub”,
“user”: “john”
},
{
“privateKey”: “/home/jane/.ssh/id_rsa”,
“publicKey”: “/home/jane/.ssh/id_rsa.pub”,
“user”: “jane”
},
{
“privateKey”: “/etc/jenkins/jenkins_rsa”,
“publicKey”: “/etc/jenkins/jenkins_rsa.pub”,
“user”: “jenkins”
},
{
“privateKey”: “/home/chad/.ssh/id_rsa”,
“publicKey”: “/home/chad/.ssh/id_rsa.pub”,
“user”: “chad”
}
]
Here is a similar type of figuration using the Dhall format:
— config0.dhall
let ordinaryUser =
\(user : Text) ->
let privateKey = “/home/${user}/.ssh/id_rsa”
let publicKey = “${privateKey}.pub”
in { privateKey, publicKey, user }
in [ ordinaryUser “john”
, ordinaryUser “jane”
, { privateKey = “/etc/jenkins/jenkins_rsa”
, publicKey = “/etc/jenkins/jenkins_rsa.pub”
, user = “jenkins”
}
, ordinaryUser “chad”
]
Up to this point, the files contain nearly an identical count of lines. Moving forward, if you wish to include a new user, let’s say, Alice.
You will need to insert an additional five lines into the JSON file:
[
…
{
“privateKey”: “/home/alice/.ssh/id_rsa”,
“publicKey”: “/home/alice/.ssh/id_rsa.pub”,
“user”: “alice”
}
]
As a consequence, there is a risk of errors, even with a simple task like copy-pasting. For instance, you may unintentionally copy the configuration meant for Chad but forget to replace the name with Alice.
On the other hand, using Dhall can simplify the process significantly. You can easily call the “ordinaryUser” function, which requires just one line of code.
…
in [ ordinaryUser “john”
, ordinaryUser “jane”
, { privateKey = “/etc/jenkins/jenkins_rsa”
, publicKey = “/etc/jenkins/jenkins_rsa.pub”
, user = “jenkins”
}
, ordinaryUser “chad”
, ordinaryUser “alice” — the line we’re adding
]
As the configuration file grows in complexity, the limitations of JSON compared to Dhall become increasingly apparent.
How to export to various formats
To export the configuration into any preferred format, you only need to execute a single command. For instance, converting the aforementioned Dhall file into JSON can be achieved using the following method:
dhall-to-json –pretty <<< ‘./config0.dhall’
Exporting Dhall to various formats, such as YAML, XML, and Bash, is just as effortless. Indeed, dhall-bash is capable of transforming Dhall instructions into Bash scripts (although it’s worth noting that only a restricted set of Bash functions is supported).
Priority on safety
Dhall is a programming language that prioritizes security over flexibility. Although it is not a Turing complete language, the authors argue that this design decision enhances the safety of both the code and the configuration files it generates.
While some tools use widely-used programming languages for configuring their functionality, such as TypeScript in webpack, Python in Django, and Scala in sbt, this approach comes with potential vulnerabilities such as insecure code, cross-site scripting (XSS), and server-side request forgery (SSRF). In contrast, Dhall is inherently secure and immune to such security risks.
The use of Dhall with Kubernetes
How to use Dhal in order to create K8s manifests?
The limitations of YAML in managing large numbers of Kubernetes objects can be attributed to its design. YAML is primarily a format for storing configurations, and although it does offer basic templating support, this can be inadequate for certain use cases. While solutions such as Helm templates can help alleviate these limitations, they may not always be practical. In such cases, using a more flexible configuration language, like Dhall (or other alternatives), can be a viable option.
Dhall is a language with built-in patterns that can handle multiple abstractions with a simple description of the configuration. Unlike YAML, it is not strongly typed, which can make it more versatile.
Using Dhall expressions, Kubernetes objects can be generated and then exported to YAML using the dhall-to-yaml utility. The dhall-kubernetes repository provides bindings, which are Dhall types and functions designed specifically for Kubernetes objects. As an example, the Deployment configuration can be defined in Dhall as follows:
— examples/deploymentSimple.dhall
let kubernetes =
https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/package.dhall sha256:532e110f424ea8a9f960a13b2ca54779ddcac5d5aa531f86d82f41f8f18d7ef1
let deployment =
kubernetes.Deployment::{
, metadata = kubernetes.ObjectMeta::{ name = Some “nginx” }
, spec = Some kubernetes.DeploymentSpec::{
, selector = kubernetes.LabelSelector::{
, matchLabels = Some (toMap { name = “nginx” })
}
, replicas = Some +2
, template = kubernetes.PodTemplateSpec::{
, metadata = Some kubernetes.ObjectMeta::{ name = Some “nginx” }
, spec = Some kubernetes.PodSpec::{
, containers =
[ kubernetes.Container::{
, name = “nginx”
, image = Some “nginx:1.15.3”
, ports = Some
[ kubernetes.ContainerPort::{ containerPort = +80 } ]
}
]
}
}
}
}
in deployment
If you export the Dhall expression to YAML using the dhall-to-yaml utility, you will get the following regular YAML output:
## examples/out/deploymentSimple.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
name: nginx
template:
metadata:
name: nginx
spec:
containers:
– image: nginx:1.15.3
name: nginx
ports:
– containerPort: 80
When it comes to modularity in Dhall, the authors provide examples of scenarios where it becomes necessary to define a type of service, such as MyService, with settings for different deployments, as well as functions that can generate Kubernetes objects from the defined service type. This approach is useful in that it enables you to define services independently of the Kubernetes context, making it possible to reuse abstractions across different configuration files.
Moreover, the Don’t Repeat Yourself (DRY) principle is upheld by using Dhall, as making a small configuration change to several objects only requires modifying a single function in a Dhall file instead of manually editing all the corresponding YAML files. This saves time and reduces the chances of introducing errors
An instance of “modularity” is demonstrated by the Nginx Ingress controller’s arrangement of TLS certificates and routes for various services:
— examples/ingress.dhall
let Prelude =
../Prelude.dhall sha256:10db3c919c25e9046833df897a8ffe2701dc390fa0893d958c3430524be5a43e
let map = Prelude.List.map
let kubernetes =
https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/package.dhall sha256:532e110f424ea8a9f960a13b2ca54779ddcac5d5aa531f86d82f41f8f18d7ef1
let Service = { name : Text, host : Text, version : Text }
let services = [ { name = “foo”, host = “foo.example.com”, version = “2.3” } ]
let makeTLS
: Service → kubernetes.IngressTLS.Type
= λ(service : Service) →
{ hosts = Some [ service.host ]
, secretName = Some “${service.name}-certificate”
}
let makeRule
: Service → kubernetes.IngressRule.Type
= λ(service : Service) →
{ host = Some service.host
, http = Some
{ paths =
[ { backend =
{ serviceName = service.name
, servicePort = kubernetes.IntOrString.Int +80
}
, path = None Text
}
]
}
}
let mkIngress
: List Service → kubernetes.Ingress.Type
= λ(inputServices : List Service) →
let annotations =
toMap
{ `kubernetes.io/ingress.class` = “nginx”
, `kubernetes.io/ingress.allow-http` = “false”
}
let defaultService =
{ name = “default”
, host = “default.example.com”
, version = ” 1.0″
}
let ingressServices = inputServices # [ defaultService ]
let spec =
kubernetes.IngressSpec::{
, tls = Some
( map
Service
kubernetes.IngressTLS.Type
makeTLS
ingressServices
)
, rules = Some
( map
Service
kubernetes.IngressRule.Type
makeRule
ingressServices
)
}
in kubernetes.Ingress::{
, metadata = kubernetes.ObjectMeta::{
, name = Some “nginx”
, annotations = Some annotations
}
, spec = Some spec
}
in mkIngress services
You can see the outcome of exporting to YAML using dhall-to-yaml:
## examples/out/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.allow-http: ‘false’
kubernetes.io/ingress.class: nginx
name: nginx
spec:
rules:
– host: foo.example.com
http:
paths:
– backend:
serviceName: foo
servicePort: 80
– host: default.example.com
http:
paths:
– backend:
serviceName: default
servicePort: 80
tls:
– hosts:
– foo.example.com
secretName: foo-certificate
– hosts:
– default.example.com
secretName: default-certificate
The services function that was defined in this scenario was invoked twice – once with the defaultService parameters (which had the host default.example.com) and again with manually supplied values (which had the host foo.example.com). As a result, the resulting manifest includes an Ingress resource with both of these hosts.
Instances of usage in the K8s community.
During the OSDNConf 2021 conference, Portside’s Oleg Nikolin shared how his engineering team benefited from Dhall. When the company migrated to Kubernetes and the CI/CD process became more complex, the number of YAML configurations soared to 12,000. Modifying even a single variable in a service could result in changes across 40 different files in various repositories, making code reviews challenging. Moreover, issues arising from incorrect configurations were not always immediately apparent, making troubleshooting difficult.
By switching to Dhall and streamlining their code, the team eliminated 50% of redundant configs. Dhall also enhanced CI/CD security by simplifying the validation of Kubernetes manifests and environment variables before deployment. Additionally, the company developed a global library containing descriptions of all their Kubernetes resources, reducing the workload on DevOps engineers and simplifying configuration validation.
Tailscale’s Christine Dodrill also found Dhall useful in managing YAML files. When neither Helm nor Kustomize could handle the task of verifying Kubernetes configuration files, she turned to Dhall. Her conclusion is? “Dhall is likely the best replacement for Helm and other Kubernetes templating tools I’ve come across recently.”
Extra ways to create manifests
There are other frameworks and programming languages that offer alternatives to YAML and simplify the utilization of Kubernetes manifests. Below are some Open Source projects with comparable features:
- CUE is a programming language equipped with a comprehensive suite of tools that can be used to define, generate, and validate various types of data, such as configuration files, API specifications, and database schemas.
- Jsonnet is a programming language for data templating that combines features of JSON and sonnet, hence the name. The language shares similarities with CUE in several aspects.
- jk is a tool for data templating that simplifies the process of generating structured configurations, including Kubernetes manifests.
- HCL stands for HashiCorp configuration language, and it is used to configure HashiCorp products. The language has two syntax variants: a native syntax that is designed to be easy for humans to read and write, and a JSON-based variant that machines can generate and parse with ease.
- cdk8s is a framework that enables developers to “program” Kubernetes manifests using common programming languages like JavaScript, Java, TypeScript, and Python. We have published a recent review of the framework.
Critiques of Dhall.
Although Dhall’s syntax is relatively straightforward and well-documented, some may find it challenging to learn without some basic programming experience.
While some people acknowledge the limitations of existing configuration languages, they may not find Dhall’s solution appealing. Andy Chu, the creator of Oil Shell, believes that total languages do not offer valuable engineering properties, but a lack of side effects does, which Dhall provides. Similarly, Adam Gordon Bell of Earthly finds Dhall to be more peculiar than HCL, which already has a following in the DevOps community.
Conclusion
Despite being a relatively new framework, Dhall has quickly matured with contributions from its community. With over 3,400 stars on GitHub, a solid contributor base, and frequent releases, Dhall has proven to be a viable alternative for scenarios where basic manifests and templates are no longer sufficient.
Dhall is being utilized in production by several companies, including KSF Media, Earnest Research, and IOHK, to create and manage Kubernetes manifests.
