Long running software development jobs have long been a bug-prone area of many applications. Fortunately for me, the most recent experience I’ve had with this type of development was in the .NET space using C#. This means I was able to leverage the CancellationToken class, which has a lot of useful, easy-to-understand features. CancellationToken has taken what might have been a messy, hard-to-manage part of our application and made it pretty simple and lightweight. I’m here to share a few winning patterns and pitfalls to avoid that we’ve learned first hand in the last year.
Don’t Try to Persist the Token Out of Memory
When we first implemented a feature in our application that required a long-running job, we knew we needed a way to ask our app to stop that job later if needed. We were also aware that if the server our application was running on shut down, we would want to clean things up as gracefully as possible before that happened.
One small issue we ran into was the question of retrieval. Our application has an API, and the feature we wanted to add was a new endpoint that would allow a user to request the stoppage of a long running job. Because there are lots of potential issues with in-memory storage in an API, we wanted to avoid storing the cancellation tokens in memory for later use. It turns out that this is the only simple option since cancellation tokens are tied to the lifecycle of the application, which means handing one off to a database or a user is not straightforward. We ended up going with an in memory approach with some guard rails, but this is definitely something to watch for if you plan to use a CancellationToken pattern in your application.
Different Cancellation Implementation Across Libraries
One bug we ran into recently was that jobs were not cleaning up gracefully when a user requested cancellation. This turned out to be due to a highly opinionated implementation of cancellation in the asynchronous version of Parallel.ForEachAsync. This implementation will immediately kill anything it’s running when it receives a cancellation request from a token. That behavior did not fit what we needed because we wanted to honor any actively running processes spawned by the loop and give them time to clean up.
This was a tricky issue to track down because of the parallel nature of the code. The final clue was the documentation for the function, which hints that this behavior is expected. We ended up not being able to use a CancellationToken for that part of our application, and, instead, we passed the token directly to the function being called by the loop. That way, we could manage cancellation ourselves at a lower level.
The Limit of CancellationToken’s Authority
Probably the most frustrating pitfall we’ve hit with this way of managing long-running jobs is that the authority of a CancellationToken only goes so far. This shows up often when your application is running on a monolithic API that lives on a single instance on one server. Because of the unique constraints of our project, we deploy new releases many times a day to this server, which causes it to restart.
There is some support for a server restart to wait for the running application to shut down, but it does not care about the underlying implementation and does not honor CancellationToken behavior. We’ve had to build our application to be resilient to these shutdowns, which we were hoping to get for free through our use of CancellationToken.
Working with long-running jobs always comes with a few surprises hiding around the corner. CancellationToken gives you a solid starting point and removes a lot of the chaos that usually comes with this territory. At the same time, it has limits and quirks that your code has to respect. Once you know where the edges are, you can build patterns that feel reliable instead of duct taped. Our team has leaned on it heavily over the last year, and the experience has been a mix of wins, odd discoveries, and a few facepalm moments. If you are heading into similar work, hopefully these notes save you from at least one of those moments.