alternative access to kubernetes node

March 12, 2022

You as an administrator of kubernetes cluster from time to time you want to get access to the selected node to debug issue. Straightforward solution is to use ssh, but cloud provider often discourage having ssh on kubernetes node by better using secure alternative like AWS Systems Manager Session Manager. But there is also simpler solution when you have permissive pod security policies, you can use pod with privileged option and pid namespace taken from host. These two options allow you to jump out from container namespace into host namespace and get shell access the same as you get via ssh. Use the following pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  hostPID: true
  containers:
    - name: test
      image: ubuntu:latest
      command: ["sleep"]
      args: ["inf"]
      securityContext:
        privileged: true

and verify that you get all the linux capabilites:

$ kubectl exec -i -t test bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@test:/# capsh --print
WARNING: libcap needs an update (cap=40 should have a name).
Current: =eip
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,38,39,40
Ambient set =
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=
Guessed mode: UNCERTAIN (0)

then nsenter comes to the play to jump into host pid 1 (due to hostPID option):

root@test:/# nsenter -t 1 -a bash
nsenter: failed to execute bash: No such file or directory

it means that host doesn’t have bash, in this case we can use bash from container using reference through procfs:

root@test:/# nsenter -t 1 -a /proc/$(pgrep sleep)/root/bin/bash
bash-5.0# journalctl -u kubelet

finally you get access to host and i.e. get kubelet logs. To prevent from doing this implement e.g. Pod Security Policies. More info about “bad pods” in this great article Bad Pods: Kubernetes Pod Privilege Escalation.