Mirrord for bringing mirrored traffic to your local development environment

I’m observing the growth of the Kubernetes ecosystem around developer experience aka DX. The goal for them is to bring your local environment closer to the Kubernetes like Telepresence which lets your local platform participate as a standard pod or another direction is to use remote CPU and memory for developing purposes like DevSpace. Both of them bring some complexity like you have to prepare the environment for them or convince other guys in a team to start using a new approach. But what about some small handy tools like we’ve many of them in Linux ?

Now comes mirrord it’s a tool that lets you bring mirrored traffic from a selected pod to your locally running application, without involving complex mirroring stuff ie. from Istio. Mirrord is divided into two parts the first one is responsibly capturing traffic from pod and the second part is using “LD_PRELOAD” mechanism by swapping file and socket functions, overwriting versions of those functions receiving mirrored traffic from pods… pretty cool. More info on mirrord and how it’s done in golang in a blog post, because golang is not using standard library libc.

How it looks like in practice ?

Let’s create sample golang application:

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "net/http"
)

const (
    listenAddr       = ":80"
    hostName2Resolve = "nginx"
)

func main() {
    r := net.Resolver{}
    ctx := context.Background()
    addr, err := r.LookupHost(ctx, hostName2Resolve)
    if err != nil {
        log.Fatalf("cannot resolve %s, err: %v\n", hostName2Resolve, err)
    }
    fmt.Printf("resolved %s to %v\n", hostName2Resolve, addr)
    fmt.Printf("starting server on %s\n", listenAddr)
    http.HandleFunc("/", getRoot)
    log.Fatal(http.ListenAndServe(listenAddr, nil))
}

func getRoot(w http.ResponseWriter, r *http.Request) {
    fmt.Printf("request for %s from %s\n", r.RequestURI, r.RemoteAddr)
}

It’s just serving HTTP endpoint and doing DNS resolver from the Kubernetes cluster. On a cluster there is a sample workload:

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
nginx-5b669b4c6-mpzm5            1/1     Running   0          3m53s

Now we start mirroring traffic hitting nginx-5b669b4c6-mpzm5 to our sample app:

$ mirrord exec -d -p nginx-5b669b4c6-mpzm5 ./mirrordgotest
resolved nginx to [192.168.15.204]
starting server on :80
request for / from 10.200.170.230:49582

got it, under the hood we get another pod bound to the network namespace of selected pod:

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
mirrord-agent-wpucojj3m5-gxpbh   1/1     Running   0          3m16s
nginx-5b669b4c6-mpzm5            1/1     Running   0          4m12s

unfortunately pod mirrord-agent-wpucojj3m5-gxpbh run in privileged mode. There is an option of using ephemeral containers, which become stable from version 1.25 of Kubernetes.

Once again nice handy tool 👍