Kubernetes Port Forward in details

One useful switch in kubectl is port-forward. The official definition is pretty short:

Forward one or more local ports to a pod.

but what is happening under the hood is not so short. I decided to figure out what are the steps to establish communication with the selected pod. After looking around and having a good overview I found very informative two blog posts 1 and 2. This could be the end of this blog post entry, but I decided to extend some concepts. I would summarize the port-forward is mostly around proxying connection through api-server, kubelet and selected container runtime. I put some details on what the “last mile” looks like. As we know kubelet is dispatching low-level requests to ie. create containers to selected container runtime. Container runtime can be easily swapped with another one that fulfills the requirements of Container Runtime Interface. The CRI protocol definition has been described here regarding to port-forward we have:

rpc PortForward(PortForwardRequest) returns (PortForwardResponse) {}
...
message PortForwardRequest {
    // ID of the container to which to forward the port.
    string pod_sandbox_id = 1;
    // Port to forward.
    repeated int32 port = 2;
}

message PortForwardResponse {
    // Fully qualified URL of the port-forward streaming server.
    string url = 1;
}

So we expect the CRI would return URL on port-forward request… pretty strange. CRI should respond to simple requests like StartContainer and some stream requests like port-forwarding. This why containerd as the implementation of CRI provides stream server and the previously mentioned URL refers to this stream server. Let’s see it in action:

bash-4.2# netstat -nputa | grep -i listen
tcp        0      0 127.0.0.1:33821         0.0.0.0:*               LISTEN      2685/containerd

it’s an output from one of the k8s node. In this particular example, it’s a random port, because I didn’t specify it. When you pass 0 as a listen port Linux will pickup random one like in containerd. Now we know that we’re dealing with stream server, so we can see the traffic on that port:

.l...l..POST /portforward/6cBPriL6 HTTP/1.1
Host: 10.1.142.77:10250
User-Agent: kubectl/v1.23.3 kubernetes/816c97a
Content-Length: 0
Connection: Upgrade
Kubectl-Command: kubectl port-forward
Kubectl-Session: 5cf1cf1d-691f-499b-918e-05f8e3f7c4f1
Upgrade: SPDY/3.1
X-Forwarded-For: 1.2.3.4, 10.0.32.120
X-Stream-Protocol-Version: portforward.k8s.io


09:06:32.185437 IP 127.0.0.1.33821 > 127.0.0.1.54766: Flags [.], ack 374, win 509, options [nop,nop,TS val 3631013148 ecr 3631013148], length 0
E..4..@....6..............$%.Sj......(.....
.l...l..
09:06:32.185557 IP 127.0.0.1.33821 > 127.0.0.1.54766: Flags [P.], seq 1:161, ack 374, win 512, options [nop,nop,TS val 3631013148 ecr 3631013148], length 160
E.....@...................$%.Sj............
.l...l..HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Upgrade: SPDY/3.1
X-Stream-Protocol-Version: portforward.k8s.io
Date: Sat, 07 May 2022 09:06:32 GMT

From this perspective, we can see that kubelet is working as a proxy and passing requests to stream server. But what about POST /portforward/6cBPriL6 it doesn’t contain the pod/container name, this the URL from PortForwardResponse. Kubelet sent PortForwardRequest with pod name and port through unix socket to containerd and get a response with the URL which is being used by the kubelet to directly communicate with stream server through TCP connection. URL contains token which uniquely identifies which container should be used, it could only be translated by containerd itself. Containerd to establish communication with selected containers in Linux environment simply get into the selected namespace and call golang net.Dial like that. This is how the URL is built on containerd and this is the way how kubelet gets it.