Kubernetes Pod Security Admission

August 13, 2022

Kubernetes 1.25 is coming closer to release, one of the biggest deprecation is Pod Security Policy I would say besides of recent deprecation of Docker is the second most famous one. What is it all about ? PSP is an old feature whose beta is dating at Kubernetes release 1.3, a lot has happened since this release in Kubernetes. PSP is not the only admission controller in Kubernetes in security areas, OPA/Gatekeeper or Kyverno offers better flexibility and is more feature rich. In my opinion, the feature less of PSP shouldn’t be a problem, because it’s built-in so it should be slim and fulfill the basic needs like preventing privilege escalation for certain workloads without complicated logic. PSP in reality is not simple and this is the reason to make it replaced by Pod Security Admission. What reason stays behind the PSP is not simple ? I’m not going to present all the cons of PSP, but the one which makes it stay away from this tool is the uncertainty of what would happen if I turn it on. Which PSP policy will be taken by the selected pod depends on the service account, SA needs to have enough permission to ask for a certain PSP policy, when you have a couple of policies the order of them depends on its mutating policy or not and with some order name. In the end, there is no option of dry-run so when you turn on PSP just fingers crossed. For example, EKS managed cluster starts with a default PSP policy eks.privileged de facto blocks the PSP controller, it’s a form of workaround and doesn’t prove that PSP best design, more about. Pod Security Admission promise to make it simple and transparent. It’s working at the namespace level and decide what predefined Pod Security Standard to take. Pods in the selected namespace will be validated against the selected PSP, the result of it can be prevent from being deployed or just emit warning for the end user or warning in the Kubernetes audit log. There are some options to make exemption. From version to version of Kubernetes the predefined PSS can change PSA offers the option to bound to the selected version of PSS with the option of dry-run which answers the question of what would happen.

Let’s see it in action:

Create dedicated k3d cluster with PSA enabled and with added custom PSA settings:

$ cat k3d.yaml
apiVersion: k3d.io/v1alpha3
kind: Simple
name: psa
servers: 1
agents: 1
image: rancher/k3s:latest
options:
  k3d:
    wait: true
    disableLoadbalancer: true
    timeout: "60s"
  k3s:
    extraArgs:
      - arg: "--kube-apiserver-arg=feature-gates=PodSecurity=true"
        nodeFilters:
        - server:*
      - arg: "--kube-apiserver-arg=admission-control-config-file=/mnt/psa.yaml"
        nodeFilters:
        - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
$ cat psa.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1beta1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "privileged"
      enforce-version: "latest"
      audit: "privileged"
      audit-version: "latest"
      warn: "baseline"
      warn-version: "latest"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces:
        - test
$ k3d cluster create -c k3d.yaml -v $(pwd):/mnt@server:0

The default policy says that whenever I broke baseline PSS I should be warned with exception of the test namespace:

$ cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: sample
spec:
  containers:
    - name: test
      securityContext:
        privileged: true
      image: busybox:latest
      command:
        - /bin/sleep
      args:
        - inf
$ kubectl apply -f pod.yaml
Warning: would violate PodSecurity "baseline:latest": privileged (container "test" must not set securityContext.privileged=true)
pod/sample created
$ kubectl delete -f pod.yaml
$ kubectl config set-context k3d-psa --namespace=test
$ kubectl apply -f pod.yaml
pod/sample created

Now I’m going to prevent running pod from 2. by setting baseline PSS in enforce mode:

$ kubectl label namespace default pod-security.kubernetes.io/enforce=baseline
$ kubectl config view --minify -o jsonpath='{..namespace}'
default
$ kubectl apply -f pod.yaml
Error from server (Forbidden): error when creating "pod.yaml": pods "sample" is forbidden: violates PodSecurity "baseline:latest": privileged (container "test" must not set securityContext.privileged=true)

How can we figure out what would have happened if I change the PSS ?

$ kubectl label namespace default pod-security.kubernetes.io/enforce-
namespace/default unlabeled
$ kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
sample   1/1     Running   0          24s
$ kubectl label namespace default pod-security.kubernetes.io/enforce=restricted --dry-run=server
Warning: existing pods in namespace "default" violate the new PodSecurity enforce level "restricted:latest"
Warning: sample: privileged, allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/default labeled

By now it looks more promising than PSP, but until now I haven’t got production experience with PSA. Option with a built-in basic working solution to prevent ie. container escalation is a way to go when you want more use external tools.