Linux PSI monitoring

Sun 19 September 2021 by admin

As I mentioned before linux psi metrics are exposed in cgroup v2 hierarchy. From node perspective these metrics are gathered by ie. node_exporter, but how can we collect these metrics from container perspective ? As far as I found out there is no such tools, in Kubernetes world there is tool called cadvisor to provide metrics from containers, which is also integrated in kubelet component. Cadvisor itself has support for cgroup v2, but it doesn't provide rich metrics. I suppose that soon or later these metrics will appear, now the state of adoption cgroup v2 in Kubernetes is at early stage. Latest version of Kubernetes 1.22 has alpha support from cgroup v2, one of the significant change is "true" memory allocation. Now when we define memory request in pod manifest kubelet doesn't set any corresponding value in cgroup v1 memory controller tree, it just count how much memory is requested at all. So in my opinion having rich settings in memory controller kernel oom knows which process needs to be killed. Moreover cgroup v2 memory controller through memory.oom.group make it more container aware, when oom kills overreaching process it kills also other processes in container. One more metrics worth to be collected from cgroup v2 tree is memory.events to get better insight about memory pressure.


Linux PSI

Sun 05 September 2021 by admin

From performance perspective one of the simplest question is how well your system is handling current load ? You can answer this question by referring to many performance indicators like cpu usage, load average, io queue size, memory usage etc. Most of them can led you to summarize that it depends ...

read more

How to match host process pid with pod name ?

Thu 19 August 2021 by admin

When you identify from host perspective the most CPU intensive task you may wonder how to match it with pod name ? It's pretty simple the relation between host pid and pod name is cgroup... more precisely cgroup name. There are multple ways to get cgroup name, one of them ...

read more

Podinfo

Sat 07 August 2021 by admin

Couple days ago I was looking for that tool, but I haven't bookmarked it. So just in case I put it here. Podinfo is a tool which fulfill needs of running something in Kubernetes for test purpose, until I found it the most popular image was a nginx or ...

read more

Kubernetes - RBAC users and groups

Sun 25 July 2021 by admin

RBAC model for Kubernetes assumes existing of:

  • service accounts
  • users
  • groups

when you define RoleBinding and ClusterRoleBinding you are pointing them into Subject. I'm fine with existing service accounts but what about users and groups, I cannot figure out how you define them in k8s. After some research I ...

read more

mkubectx - update

Mon 03 May 2021 by admin

I recently do some changes in mkubectx including:

  • golang testing
  • github workflows for testing when PR is submitted
  • github workflows for releasing version supported by goreleaser

Feel free to use and contribute :)

read more

How to get logs from selected container in a pod ?

Sun 07 March 2021 by admin

I faced with the problem where I have to react on specific log entry from an third party application. It wasn't possible to change behaviour of this application, but fortunately it generates specific log when something bad happened. So I decide to use liveness probe with some exec to ...

read more

Ways of distributing pods across nodes

Tue 02 February 2021 by admin

How can more evenly distribute pod across nodes ? After quick research I found that this example of deployment should be ok:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
      affinity:
        podAntiAffinity ...
read more

GKE Node Debugging

Sun 10 January 2021 by admin

GKE is fully managed k8s cluster in Google Cloud Platform, one of its component is a node pool. Node pool is a managed instance group of compute engine virtual machines to run your workload. In GKE by default compute engine virtual machine runs with COS. COS doesn't provide wide ...

read more

Kind - local k8s

Sun 13 September 2020 by admin

Few words after using kind:

  • works only with docker, there is also a podman provider (not tested)
  • docker image node-image simulates a k8s node - all components in one image, started by systemd
  • docker container is priviledged
  • easy to start just use kind command, under the hood it download right node-image ...
read more