o11yfest

Debugging at Scale in Production - Deep into your Containers with kubectl debug, KoolKits and Continuous Observability

Brian Kernigham said: “Debugging is twice as hard as writing the code in the first place.”

In fact, debugging in a modern production environment is even harder - orchestrators spinning containers up and down and weird networking wizardry that keeps everything glued together, make understanding systems that much more difficult than it used to be.

And, while k8s is well understood by DevOps people by now, it remains a nut that developers are still trying to crack. Where do you start when there’s a production problem? How do you get the tools you’re used to in the remote container? How do you understand what is running where and what is its current state?

In this talk, we will review debugging a production application deployed to a Kubernetes cluster, and review kubectl debug - a new feature from the Kubernetes sig-cli team. In addition, we’ll review the open source KoolKits project that offers a set of (opinionated) tools for kubectl debug.

KoolKits builds on top of kubectl debug by adding everything you need right into the image. When logging into a container, we’re often hit with the scarcity of tools at our disposal. No vim (for better or worse), no DB clients, no htop, no debuggers, etc… KoolKits adds all the tools you need right out of the box and lets you inspect a production container easily without resorting to endless installation and configuration cycles for each needed package.

We’ll finish the talk by delving into how to get better at debugging on a real-world scale. Specifically, we’ll talk about how to be disciplined in our continuous observability efforts by using tools that are built for k8s scale and can run well in those environments, while remaining ergonomic for day to day use.

This session will go back and forth between explanation slides and demonstration of the topic at hand.