.NET Core, OSX, libcurl, and OpenSSL

.NET Core makes it convenient to develop and test C# code across platforms. On my current project, this means we can do much of our work on Macs without ever firing up a Windows VM.

Even the best abstraction layers occasionally leak, though. Here’s a story of an OSX-specific issue we encountered, what we learned, and how to resolve it.

The Problem

So there I was, running an existing .NET Framework project on .NET Core for the first time (using .NET Core 1.1). The project uses a third-party library to communicate with a REST API. Like many problems in software development, it started with an error message:

System.PlatformNotSupportedException: The libcurl library in use (7.54.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.

libcurl

provides an HTTP client, and it can be built with one of several TLS backends. It looked like we needed to be using OpenSSL instead of SecureTransport (which is Apple’s TLS Implementation).

For us, the exception came from within a third-party library, but you might encounter it if you were doing something advanced with HttpClient. For example, a contributor on the GitHub issue provided a short repro that attempts to check a certificate revocation list.

Get libcurl

So how can we get libcurl built with OpenSSL? The easiest way is to install Homebrew’s curl package, which provides both the curl command line tool and the libcurl library. It, too, uses SecureTransport by default, but we can choose OpenSSL instead:

brew install curl --with-openssl

After it’s installed, you’ll get a warning:

This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble.

Let’s see what kind of trouble, shall we?

A Bad Solution

One suggestion on the issue thread is to disregard the warning and force-link libcurl (brew link --force curl). This would promote curl and libcurl to paths within /usr/local. Then feed the library path to the loader by setting an environment variable: DYLD_LIBRARY_PATH=/usr/local/lib.

_Side note: DYLD_LIBRARY_PATH is OSX’s version of LD_PRELOAD, which is awesome, and you can read more about it here.

With this environment variable in place, when the loader goes looking for libraries, it will look in /usr/local/lib first. This fixes our libcurl issue by prioritizing Brew’s version over the system version. Sweet!

But what if there were some other libraries in there that we didn’t want? Like, what if we needed Apple’s special version of libJPEG because it provides extra symbols that Homebrew’s libJPEG doesn’t offer?

 Failed to load /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/libcoreclr.dylib, error: dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/libcoreclr.dylib, 1): Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /usr/local/lib/libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
Failed to bind to CoreCLR at '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/libcoreclr.dylib'

Oops. This also happens with libtiff (__cg_TIFFClientOpen) and libpng (__cg_png_create_info_struct). For a practical repro of the conflict, brew install imagemagick, which depends on all three. If you ever need to analyze problems like this more thoroughly, you can view library and symbol dependencies with the otool and nm commands.

To fix this, we could uninstall or unlink those libs, but that feels like setting another hack on a leaning pile of hacks.

A Better Solution

Instead of prioritizing all of Homebrew’s libraries, it’s safer to take just what we need: DYLD_LIBRARY_PATH=/usr/local/opt/curl/lib. No force link necessary. Tada!

You could also use this technique to provide Homebrew’s OpenSSL to dotnet, instead of manually symlinking the libs as the current installation guide instructs.

The Best Solution

In the future, this won’t be necessary at all, as .NET Core 2.0 won’t rely on OpenSSL on OSX. If you install the 2.0 preview and target e.g. netcoreapp2.0, the problem goes away!

If you’re stuck on 1.x, the Better Solution above offers a feasible workaround that doesn’t require you to ignore your package manager’s warnings and introduce risk to other software on your system. If you’re able, though, now’s a good time to try out .NET Core 2.0. It should be releasing very soon!

Update: since this post was written, .NET Core 2.0 has been released!