Open Policy Agent (OPA) is een tool om in Kubernetes controles uit te voeren voordat een deployment geïnstalleerd kan worden. Het is een Admission Controller die als tussenlaag de policies uitvoert. Als er niet aan de voorwaardes in de policies wordt voldaan zal de deployment niet uitgevoerd worden.
OPA maakt gebruik van Templates waarin de Constraints (=beperkingen) worden gedefinieerd. Templates bestaan uit Custom Resource Definitions (CRDs) en Targets die worden geschreven in een taal genaamd ‘Rego‘.
De installatie van OPA Gatekeeper kan het best via Helm gedaan worden, in Rancher vind je het terug in ‘Apps & Marketplace’.
Via kubectl op een bestaand cluster kan dat met:
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml
OPA als binary kan ook geïnstalleerd worden als Command Line tool.
MacOS:
curl -L -o opa https://openpolicyagent.org/downloads/v0.36.1/opa_darwin_amd64
Linux:
curl -L -o opa https://openpolicyagent.org/downloads/v0.36.1/opa_linux_amd64_static
Windows:
https://openpolicyagent.org/downloads/v0.36.1/opa_windows_amd64.exe
Voorbeelden voor OPA cli vind je hier.
Voorbeeld om pod containers te beperken met resource limits/requests
De Template hiervoor ziet er als volgt uit:
package k8scontainerlimits
missing(obj, field) = true {
not obj[field]
}
missing(obj, field) = true {
obj[field] == ""
}
canonify_cpu(orig) = new {
is_number(orig)
new := orig * 1000
}
canonify_cpu(orig) = new {
not is_number(orig)
endswith(orig, "m")
new := to_number(replace(orig, "m", ""))
}
canonify_cpu(orig) = new {
not is_number(orig)
not endswith(orig, "m")
re_match("^[0-9]+$", orig)
new := to_number(orig) * 1000
}
# 10 ** 21
mem_multiple("E") = 1000000000000000000000 { true }
# 10 ** 18
mem_multiple("P") = 1000000000000000000 { true }
# 10 ** 15
mem_multiple("T") = 1000000000000000 { true }
# 10 ** 12
mem_multiple("G") = 1000000000000 { true }
# 10 ** 9
mem_multiple("M") = 1000000000 { true }
# 10 ** 6
mem_multiple("k") = 1000000 { true }
# 10 ** 3
mem_multiple("") = 1000 { true }
# Kubernetes accepts millibyte precision when it probably shouldn't.
# https://github.com/kubernetes/kubernetes/issues/28741
# 10 ** 0
mem_multiple("m") = 1 { true }
# 1000 * 2 ** 10
mem_multiple("Ki") = 1024000 { true }
# 1000 * 2 ** 20
mem_multiple("Mi") = 1048576000 { true }
# 1000 * 2 ** 30
mem_multiple("Gi") = 1073741824000 { true }
# 1000 * 2 ** 40
mem_multiple("Ti") = 1099511627776000 { true }
# 1000 * 2 ** 50
mem_multiple("Pi") = 1125899906842624000 { true }
# 1000 * 2 ** 60
mem_multiple("Ei") = 1152921504606846976000 { true }
get_suffix(mem) = suffix {
not is_string(mem)
suffix := ""
}
get_suffix(mem) = suffix {
is_string(mem)
count(mem) > 0
suffix := substring(mem, count(mem) - 1, -1)
mem_multiple(suffix)
}
get_suffix(mem) = suffix {
is_string(mem)
count(mem) > 1
suffix := substring(mem, count(mem) - 2, -1)
mem_multiple(suffix)
}
get_suffix(mem) = suffix {
is_string(mem)
count(mem) > 1
not mem_multiple(substring(mem, count(mem) - 1, -1))
not mem_multiple(substring(mem, count(mem) - 2, -1))
suffix := ""
}
get_suffix(mem) = suffix {
is_string(mem)
count(mem) == 1
not mem_multiple(substring(mem, count(mem) - 1, -1))
suffix := ""
}
get_suffix(mem) = suffix {
is_string(mem)
count(mem) == 0
suffix := ""
}
canonify_mem(orig) = new {
is_number(orig)
new := orig * 1000
}
canonify_mem(orig) = new {
not is_number(orig)
suffix := get_suffix(orig)
raw := replace(orig, suffix, "")
re_match("^[0-9]+$", raw)
new := to_number(raw) * mem_multiple(suffix)
}
violation[{"msg": msg}] {
general_violation[{"msg": msg, "field": "containers"}]
}
violation[{"msg": msg}] {
general_violation[{"msg": msg, "field": "initContainers"}]
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
cpu_orig := container.resources.limits.cpu
not canonify_cpu(cpu_orig)
msg := sprintf("container <%v> cpu limit <%v> could not be parsed", [container.name, cpu_orig])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
mem_orig := container.resources.limits.memory
not canonify_mem(mem_orig)
msg := sprintf("container <%v> memory limit <%v> could not be parsed", [container.name, mem_orig])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
not container.resources
msg := sprintf("container <%v> has no resource limits", [container.name])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
not container.resources.limits
msg := sprintf("container <%v> has no resource limits", [container.name])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
missing(container.resources.limits, "cpu")
msg := sprintf("container <%v> has no cpu limit", [container.name])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
missing(container.resources.limits, "memory")
msg := sprintf("container <%v> has no memory limit", [container.name])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
cpu_orig := container.resources.limits.cpu
cpu := canonify_cpu(cpu_orig)
max_cpu_orig := input.parameters.cpu
max_cpu := canonify_cpu(max_cpu_orig)
cpu > max_cpu
msg := sprintf("container <%v> cpu limit <%v> is higher than the maximum allowed of <%v>", [container.name, cpu_orig, max_cpu_orig])
}
general_violation[{"msg": msg, "field": field}] {
container := input.review.object.spec[field][_]
mem_orig := container.resources.limits.memory
mem := canonify_mem(mem_orig)
max_mem_orig := input.parameters.memory
max_mem := canonify_mem(max_mem_orig)
mem > max_mem
msg := sprintf("container <%v> memory limit <%v> is higher than the maximum allowed of <%v>", [container.name, mem_orig, max_mem_orig])
}
De Constraint die daar bij hoort:
apiVersion: constraints.gatekeeper.sh/v1
kind: K8sContainerLimits
metadata:
name: container-must-have-limits
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces:
- go-app
- default
scope: Namespaced
parameters:
cpu: "200m"
memory: "1Gi"
Zoals in de Constraint is aangegeven, mogen de containers maximaal een resource limiet hebben van 200m voor CPU en 1Gi voor memory. De ‘Scope‘ voor de beperking is opgelegd aan de Namespaces ‘go-app’ en ‘default’.
We kunnen nu twee pods proberen aan te maken, één waarbij de limieten binnen de constraints vallen en één die dat overschrijdt.
apiVersion: v1
kind: Pod
metadata:
labels:
run: good-pod
nico: devops
name: good-pod
namespace: default
spec:
containers:
- image: busybox
name: good-pod
command: ["sleep"]
args: ["3600"]
resources:
requests:
memory: "64Mi"
cpu: "100m"
limits:
memory: "128Mi"
cpu: "200m"
apiVersion: v1
kind: Pod
metadata:
labels:
run: bad-pod
nico: devops
name: bad-pod
namespace: default
spec:
containers:
- image: busybox
name: bad-pod
command: ["sleep"]
args: ["3600"]
resources:
limits:
memory: "2Gi"
cpu: "300m"
Met het kubectl commando de pods aanmaken:
$ kubectl -n default apply -f good-pod.yaml
pod/good-pod created
$ kubectl -n default apply -f bad-pod.yaml
Error from server ([container-must-have-limits] container memory limit <2Gi> is higher than the maximum allowed of <1Gi>
Error from server ([container-must-have-limits] container cpu limit <300m> is higher than the maximum allowed of <200m>)
Meer info en voorbeelden voor OPA: https://github.com/open-policy-agent/gatekeeper-library