GKE Node Debugging

Sun 10 January 2021 by admin

GKE is fully managed k8s cluster in Google Cloud Platform, one of its component is a node pool. Node pool is a managed instance group of compute engine virtual machines to run your workload. In GKE by default compute engine virtual machine runs with COS. COS doesn't provide wide range of debugging tools and moreover most of it's parts are readonly, it's designed to mainly provide security. To make it more customizable, there is tool called toolbox. It's configuration is placed in:

~ # cat /etc/default/toolbox
# Copyright 2016 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

TOOLBOX_BIND="--bind=/:/media/root/ --bind=/mnt/disks/:/media/root/mnt/disks/ --bind=/var/:/media/root/var/ --bind=/home:/media/root/home/ --bind=/etc/resolv.conf:/etc/resolv.conf"
: ${USER:=root}

at first run it create selected docker container just to get filesystem hierarchy from its docker image. This filesystem hierarchy is put into /var/lib/toolbox to make it persist. The next step of toolbox is to start systemd-nspawn lightweight container from recently created filesystem hierarchy. In this container you can install your debug tools.

COS in the subject of debugging also provide perf tool.