How to Hot Reload Kubernetes with Skaffold

Two elements of the development experience that are important to me are fast local development iterations and a production-like local development environment. For a long time, I thought these two things were mutually exclusive. For most projects that I start, I configure two methods of running the app locally. The first is where the application runs in my terminal with hot-reloading to enable super fast turn around times when developing features. Another is where the application runs in a docker container to mimic the deployed environments as closely as possible. Maintaining both of these systems introduces an annoying overhead burden.

On a recent project, our local development environment runs a Minikube Kubernetes cluster in Docker to mimic our deployed Kubernetes clusters. When a team member requested that we be able to hot-reload code changes within the local cluster, I thought there was no way we could make that happen in an ergonomic way. That was until I discovered Skaffold, a command line tool for continuous development of container based applications.

I’ll describe how I leveraged Skaffold to hot-reload a deployment on our Dockerized Minikube cluster, some of the hurdles I experienced along the way, and the benefits we’ve seen since then. Skaffold has many features and capabilities; I am going to focus exclusively on the local hot-reload capabilities in this post. I look forward to trialing its other abilities in the future.

Configuring Skaffold

Configuring Skaffold is not difficult, but the wide variety of supported features can make it tricky to determine which pieces of the configuration you really need. The entire tool is configured with a single file called skaffold.yaml at the root of your repository (or wherever the source of the service to be hot-reloaded lives). For our purposes, you can break the configuration file down into three main sections: build, render, and deploy.

Build

Skaffold’s build process can use several different tools to rebuild your app. They are Docker, Jib, Bazel, Ko, and Buildpacks. It also supports custom build scripts which we needed to use in our case to invoke a special gradle task.

I think the custom build script is a good example because it clearly demonstrates the contract that each builder follows. Skaffold provides a handful of environment variables to your script as an input. In turn, your script is expected to build the image, tag it appropriately, and potentially push it if specified. More details and the exact environment variables are described in the documentation. Additionally, we can define a list of dependent file paths to watch for changes to determine when to rebuild. Below is how we configured our simple custom-build script.

YAML
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
  artifacts:
    - image: localhost:5000/my-app
      context: .
      custom:
        buildCommand: |
          ./gradlew dockerBuild
          docker tag my-app $IMAGE
          if $PUSH_IMAGE
          then
            docker push $IMAGE
          fi
        dependencies:
          paths:
          - src/main
...

This example causes the dockerBuild gradle task to run every time Skaffold detects changes within the src/main directory. The script also tags and pushes the image to our locally running image registry at localhost:5000.

Render

Once Skaffold rebuilds the image, it also must re-render all of the relevant Kubernetes manifests. We use the manifests section of the same configuration file to tell Skaffold where to find our manifests. Again, a handful of renderers are supported including Kustomize, Kpt, Helm, and raw YAML. In my case, we were using Kustomize, which makes for the following simple configuration. You could list many paths if your manifests are scattered around, but we keep ours neatly nested by service and environment.

YAML
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
    ...
manifests:
  kustomize:
  paths:
    - manifests/my-app/local
...

Deploy

The final step in Skaffold’s process is the deploy. Skaffold’s supported deploy tools are Kubectl, Kpt, Helm, and Docker (optionally with Docker Compose). They also support Google Cloud Run, but that’s obviously not applicable in a local hot-reload scenario. We use Kubectl on our project and simply providing the default namespace was enough to complete the basic configuration.

YAML
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
    ...
manifests:
    ...
deploy:
  kubectl:
    defaultNamespace: my-app

Things to Watch Out For

With the configuration defined above, we’re able to run the command skaffold dev and see changes to the source be automatically rebuilt and deployed. Our build and startup takes a bit longer than is ideal and we definitely still feel that pain with Skaffold. It’s not the “blazing fast iterations” that you might hope for in local dev, but it’s a lot smoother than manually building and pushing images for every change.

Issue #1

We use port-forwarding extensively in our cluster to expose some services to the developer’s local machine. The carousel of pods created by Skaffold can cause confusion with Minikube’s port forwarding. But, thankfully Skaffold has a solution for that too. The portForward section of the configuration file allows you to specify how to maintain port forwarding for various resources.

YAML
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
    ...
manifests:
    ...
deploy:
  ...
portForward:
    - resourceType: service
        resourceName: my-app
        namespace: my-app
        port: 5005
        localPort: 5005

By default, Skaffold will try to cleanup its resources when you interrupt the skaffold dev session. We found this behavior to be undesirable in our case because we prefer our local environment to be long living, even beyond local iterations. Having Skaffold tear down the entire namespace, meant losing key resources that might interrupt other ongoing processes in the environment. This behavior can be prevented by disabling the cleanup flag: skaffold dev --cleanup=false.

Issue #2

One other issue we encountered is that Skaffold seems to have trouble with Kubernetes deployments that specify an image pull policy of “Always”. This is currently an open issue in the Skaffold repository that I hope should have a resolution soon. We have worked around this for now by creating a Kustomize patch specifically for Skaffold that overwrites the image pull policy.

YAML
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
    ...
manifests:
  kustomize:
    paths:
      - manifests/my-app/skaffold
...
YAML
# manifests/my-app/skaffold/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patches:
  - path: deployment.patch.yaml
resources:
  - ../local/
YAML
# manifests/my-app/skaffold/deployment.patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: my-app
        imagePullPolicy: IfNotPresent

I’ve only scratched the surface of what Skaffold can do, and I’ve been pleasantly surprised by how rich and robust the tool is. The documentation is pretty good, and I’ve found decent community support, too.

Using Skaffold to hot-reload pods in our local cluster has been a great improvement to our developers’ iterations. It works quick and smooth. All of the “gotchas” that we’ve run into have either been accounted for or are merely relics of the size of our applications. I look forward to exploring more of Skaffold’s features in the future.

Conversation

Join the conversation

Your email address will not be published. Required fields are marked *