Article summary
Two elements of the development experience that are important to me are fast local development iterations and a production-like local development environment. For a long time, I thought these two things were mutually exclusive. For most projects that I start, I configure two methods of running the app locally. The first is where the application runs in my terminal with hot-reloading to enable super fast turn around times when developing features. Another is where the application runs in a docker container to mimic the deployed environments as closely as possible. Maintaining both of these systems introduces an annoying overhead burden.
On a recent project, our local development environment runs a Minikube Kubernetes cluster in Docker to mimic our deployed Kubernetes clusters. When a team member requested that we be able to hot-reload code changes within the local cluster, I thought there was no way we could make that happen in an ergonomic way. That was until I discovered Skaffold, a command line tool for continuous development of container based applications.
I’ll describe how I leveraged Skaffold to hot-reload a deployment on our Dockerized Minikube cluster, some of the hurdles I experienced along the way, and the benefits we’ve seen since then. Skaffold has many features and capabilities; I am going to focus exclusively on the local hot-reload capabilities in this post. I look forward to trialing its other abilities in the future.
Configuring Skaffold
Configuring Skaffold is not difficult, but the wide variety of supported features can make it tricky to determine which pieces of the configuration you really need. The entire tool is configured with a single file called skaffold.yaml
at the root of your repository (or wherever the source of the service to be hot-reloaded lives). For our purposes, you can break the configuration file down into three main sections: build, render, and deploy.
Build
Skaffold’s build process can use several different tools to rebuild your app. They are Docker, Jib, Bazel, Ko, and Buildpacks. It also supports custom build scripts which we needed to use in our case to invoke a special gradle task.
I think the custom build script is a good example because it clearly demonstrates the contract that each builder follows. Skaffold provides a handful of environment variables to your script as an input. In turn, your script is expected to build the image, tag it appropriately, and potentially push it if specified. More details and the exact environment variables are described in the documentation. Additionally, we can define a list of dependent file paths to watch for changes to determine when to rebuild. Below is how we configured our simple custom-build script.
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
artifacts:
- image: localhost:5000/my-app
context: .
custom:
buildCommand: |
./gradlew dockerBuild
docker tag my-app $IMAGE
if $PUSH_IMAGE
then
docker push $IMAGE
fi
dependencies:
paths:
- src/main
...
This example causes the dockerBuild
gradle task to run every time Skaffold detects changes within the src/main
directory. The script also tags and pushes the image to our locally running image registry at localhost:5000
.
Render
Once Skaffold rebuilds the image, it also must re-render all of the relevant Kubernetes manifests. We use the manifests
section of the same configuration file to tell Skaffold where to find our manifests. Again, a handful of renderers are supported including Kustomize, Kpt, Helm, and raw YAML. In my case, we were using Kustomize, which makes for the following simple configuration. You could list many paths if your manifests are scattered around, but we keep ours neatly nested by service and environment.
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
...
manifests:
kustomize:
paths:
- manifests/my-app/local
...
Deploy
The final step in Skaffold’s process is the deploy. Skaffold’s supported deploy tools are Kubectl, Kpt, Helm, and Docker (optionally with Docker Compose). They also support Google Cloud Run, but that’s obviously not applicable in a local hot-reload scenario. We use Kubectl on our project and simply providing the default namespace was enough to complete the basic configuration.
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
...
manifests:
...
deploy:
kubectl:
defaultNamespace: my-app
Things to Watch Out For
With the configuration defined above, we’re able to run the command skaffold dev
and see changes to the source be automatically rebuilt and deployed. Our build and startup takes a bit longer than is ideal and we definitely still feel that pain with Skaffold. It’s not the “blazing fast iterations” that you might hope for in local dev, but it’s a lot smoother than manually building and pushing images for every change.
Issue #1
We use port-forwarding extensively in our cluster to expose some services to the developer’s local machine. The carousel of pods created by Skaffold can cause confusion with Minikube’s port forwarding. But, thankfully Skaffold has a solution for that too. The portForward
section of the configuration file allows you to specify how to maintain port forwarding for various resources.
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
...
manifests:
...
deploy:
...
portForward:
- resourceType: service
resourceName: my-app
namespace: my-app
port: 5005
localPort: 5005
By default, Skaffold will try to cleanup its resources when you interrupt the skaffold dev
session. We found this behavior to be undesirable in our case because we prefer our local environment to be long living, even beyond local iterations. Having Skaffold tear down the entire namespace, meant losing key resources that might interrupt other ongoing processes in the environment. This behavior can be prevented by disabling the cleanup flag: skaffold dev --cleanup=false
.
Issue #2
One other issue we encountered is that Skaffold seems to have trouble with Kubernetes deployments that specify an image pull policy of “Always”. This is currently an open issue in the Skaffold repository that I hope should have a resolution soon. We have worked around this for now by creating a Kustomize patch specifically for Skaffold that overwrites the image pull policy.
# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
build:
...
manifests:
kustomize:
paths:
- manifests/my-app/skaffold
...
# manifests/my-app/skaffold/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patches:
- path: deployment.patch.yaml
resources:
- ../local/
# manifests/my-app/skaffold/deployment.patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
- name: my-app
imagePullPolicy: IfNotPresent
I’ve only scratched the surface of what Skaffold can do, and I’ve been pleasantly surprised by how rich and robust the tool is. The documentation is pretty good, and I’ve found decent community support, too.
Using Skaffold to hot-reload pods in our local cluster has been a great improvement to our developers’ iterations. It works quick and smooth. All of the “gotchas” that we’ve run into have either been accounted for or are merely relics of the size of our applications. I look forward to exploring more of Skaffold’s features in the future.