I’ve recently had the unique opportunity to work with both CircleCI and Azure Pipelines at the same time on one project. In this post, I’ll compare and contrast my experiences using these two tools. If you’re trying to decide between the two, I hope you can benefit from my experience.
Context & Caveats
You should probably know where I’m coming from. I was brand new to Pipelines and didn’t have much of a template to work from; I’d used CircleCI for longer and had more exposure to CircleCI examples. I was using these platforms to set up redundant continuous integration testing and build workflows for a mobile application (iOS and Android) and a Rails backend service.
I am by no means a superuser of either tool, and I certainly haven’t worked with the full toolset of either. If you are a superuser, you probably don’t need this post; you should continue using the platform you are familiar with.
I will not be comparing price or performance. Atomic Object is lucky enough to have awesome IT support that handles our CI platform subscriptions, so I have very little exposure to the pricing models of these services. I’ve also never been much of a stickler for CI performance or parallelizability. As long as my jobs complete in a reasonable length of time, I pay little attention to measurable performance.
Setup & Configuration
Out of the gate, I was immediately impressed by Pipelines. Azure handles GitHub integration seamlessly. All I had to do was sign in with GitHub, and it handled the rest. When I was getting started with CircleCI, I recall a fair amount of frustration with initial configuration items like private key setup.
That being said, once I got repository integration established, the next steps came much easier with CircleCI. This may be influenced by my previous experience with CircleCI examples set up by other teammates, but starting from scratch with Pipelines was quite difficult. I got the feeling that, in an effort to create higher and higher levels of abstraction of features, they forgot to polish the basics. Things like getting the correct VM image and setting up environment variables required a frustrating amount of research. Azure Pipelines does offer templates for various project configurations at setup time, but I never found them particularly useful.
Built-in Helpers
Both CircleCI and Azure Pipelines offer some form of built-in helper functions. I have not used many of these in CircleCI, but the ones I have used seem to just work whenever I call on them. Things like caching dependencies and exporting workflow outputs have been really nice features. These functions are documented pretty well, and the CircleCI developer community provides a nice supply of examples to follow.
In contrast, I’ve yet to figure out how to use any helpers from Azure Pipelines other than PublishTestResults@2. My understanding is that Pipelines allows third-party helper functions (or “Tasks,” as they call them). This sounds like a great benefit over CircleCI, but I suspect it also contributes to the confusion around using them and finding appropriate documentation.
I also can’t wrap up my section on Tasks without a quick complaint about Task versioning. The “@2” tacked onto the name feels gross to me. Why couldn’t Tasks just have another parameter for the version? If you’re lucky enough to find documentation about a third-party Task, I would be surprised if it also includes any versioning specifications.
Broader Integration
I mentioned before that I feel Pipelines takes the repository integration award, but what about integration with the broader developer ecosystem? Again, it seems to me that Pipelines wins here — but only in theory. We have not yet fully adopted the Azure suite of tools, but the promise of Azure pipelines linking with deploy pipelines, application servers, asset libraries, and more is tantalizing. If it’s implemented and documented well, I think that could be a big benefit to going all-in on Azure.
To be fair, I don’t feel like I can knock CircleCI for their integration support. CircleCI is a third-party provider without much other infrastructure of its own, so I’ve always been impressed by how many places it will plug in smoothly. They don’t have the benefit of a full dev-ops toolkit, but I’ve never found a tool where I wanted CircleCI integrations and haven’t been able to make it happen.
User Interface
I found Azure Pipelines’ user interface to be confusing and bulky at first, but after using it for a little while and finding my main touchpoints, I have fewer issues with it. In some areas, it provides a more verbose and data-packed interface, even if that means duplicating information across multiple views. On the other hand, I feel like CircleCI tries to push less data at me and forces me to go looking for it when I need that information. I don’t often find myself interested in CI statistics; I mostly just care whether my jobs pass or fail, so this isn’t make-or-break for me.
I should note that most of my experience with CircleCI has been with the “old” user interface. As of writing this post, my projects were forced into the new UI within the last few weeks. I have not yet been blown away by anything in the new interface, but it’s by no means a downgrade.
One interesting aspect of Pipelines UI that CircleCI does not offer is the GUI pipeline builder. I have not used this feature extensively. I opted for the yaml configuration in hopes that it would mirror my pre-existing CircleCI yaml config, but I did explore a little. I couldn’t quite do everything that I wanted with the GUI builder, but I think smaller projects or DevOps operations led by a less technical project manager could find great uses for this tool.
Local Tools
I saved this topic for last because it’s currently my biggest deterrent against using Azure Pipelines. CircleCI offers a nice commandline tool that allows you to validate your yaml configuration and, in some cases, run the workflow without using resources on the remote server. This is incredibly helpful while setting up your initial workflow, which tends to be a slow, iterative process of building up the workflow step-by-step.
I haven’t been able to find anything similar for Pipelines. In fact, from forum threads I’ve read, it sounds like it’s not even planned. This means any time you make a change to your configuration, the iteration process is slow and painful: make a change, push to the remote, wait for the build to run, validate results. Hopefully, you didn’t add an extra space or misplace a colon in your configuration, because that means another commit, another push, and another pipeline run. Maybe there’s a smarter iteration process than this, but I haven’t discovered it yet.
Recommendation
Overall, each of these platforms has its own strengths and weaknesses, and you really can’t go too far wrong with either of them. At the end of the day, I think I will continue to reach for CircleCI when I’m setting up new continuous integration workflows. If you have your whole project configured in Azure services, then Pipelines should be your choice. But otherwise, if you value strong documentation and community support, I think CircleCI is the way to go.