As I mentioned before linux psi metrics are exposed in cgroup v2 hierarchy. From node perspective these metrics are gathered by ie.
node_exporter, but how can we collect these metrics from container perspective ?
As far as I found out there is no such tools, in Kubernetes world there is tool called cadvisor to provide metrics from containers, which is also integrated in
Cadvisor itself has support for cgroup v2, but it doesn't
provide rich metrics. I suppose that soon or later these metrics will appear, now the state of adoption cgroup v2 in Kubernetes is at early stage. Latest version of Kubernetes 1.22 has alpha support from cgroup v2, one of the significant change is "true" memory allocation. Now when we define memory request in pod manifest kubelet doesn't set any corresponding value in cgroup v1 memory controller tree, it just count how much memory is requested at all. So in my opinion having rich settings in memory controller kernel oom knows which process needs to be killed. Moreover cgroup v2 memory controller through
memory.oom.group make it more container aware, when oom kills overreaching process it kills also other processes in container. One more metrics worth to be collected from cgroup v2 tree is
memory.events to get better insight about memory pressure.
From performance perspective one of the simplest question is how well your system is handling current load ? You can answer this question by referring to many performance indicators like cpu usage, load average, io queue size, memory usage etc. Most of them can led you to summarize that it depends ...read more
How to match host process pid with pod name ?
When you identify from host perspective the most CPU intensive task you may wonder how to match it with pod name ? It's pretty simple the relation between host pid and pod name is cgroup... more precisely cgroup name. There are multple ways to get cgroup name, one of them ...read more
Kubernetes - RBAC users and groups
RBAC model for Kubernetes assumes existing of:
- service accounts
when you define
ClusterRoleBinding you are pointing them into Subject.
I'm fine with existing service accounts but what about users and groups, I cannot figure out how you define them in k8s. After some research I ...
mkubectx - update
How to get logs from selected container in a pod ?
I faced with the problem where I have to react on specific log entry from an third party application. It wasn't possible to change behaviour of this application, but fortunately it generates specific log when something bad happened. So I decide to use liveness probe with some exec to ...read more
Ways of distributing pods across nodes
How can more evenly distribute pod across nodes ? After quick research I found that this example of deployment should be ok:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest affinity: podAntiAffinity ...
GKE Node Debugging
Kind - local k8s
Few words after using
- works only with docker, there is also a podman provider (not tested)
- docker image node-image simulates a k8s node - all components in one image, started by systemd
- docker container is priviledged
- easy to start just use
kindcommand, under the hood it download right