Create and Implement .NET-Compatible JSON appSettings Using Consul Template and HashiCorp Vault Secrets

As secret handling gets sophisticated, applications must adopt measures that ensure they can easily secure and manage their secrets. Below, I’ll detail one way to securely manage secrets using HashiCorp Vault. We’ll transform them via Consul Template into .NET-compatible JSON appSettings and then integrate these secrets into a .NET application.

Creating a Secret in Your HashiCorp Vault

The first step involves creating a secret in your HashiCorp Vault. Secrets in a vault are the keys and corresponding values that should be secured from unauthorized access.

vault kv put secret/path/to/secret database/username=value
vault kv put secret/path/to/secret database/password=value
vault kv put secret/path/to/secret opensearch/username=value
vault kv put secret/path/to/secret opensearch/password=value

This command adds keys and corresponding values to the provided path. With this, you can define your application secrets such as database connection strings or API keys.

{
    "database/username": "value",
    "database/password": "value",
    "opensearch/username": "value",
    "opensearch/password": "value"
}

Alternatively, you could create and update the secret using the provided interface. Using the JSON toggle allows you are quickly confirm the correct formatting.

Fetching Configuration using HashiCorp Consul Template

Once the secrets are safely stored in HashiCorp Vault, the next step is pulling these secrets using a Consul Template. There are several ways to use Consul to pull in secrets, but, in this example, we are going to look at how to configure in Kubernetes.

In your Kubernetes deployment, we need to add annotations to the secret defining the file we want to be added. This article will not go into depth on integrating Vault into your deployment, but the important annotations are:

vault.hashicorp.com/agent-inject-secret-appsecrets.json: "secret/path/to/secret"
vault.hashicorp.com/agent-inject-template-appsecrets.json: |
          {{ (secret "secret/path/to/secret").Data.data | explodeMap | toJSONPretty }}

These annotations define a few key things:

•  The file *appsecrets.json*, which will be written to */vault/secret/appsecrets.json*.
•  The default secret that will be injected “secret/path/to/secret”.
•  The template used to format the data from Vault.

The interesting bit here is the template spec.

{{ (secret "secret/path/to/secret").Data.data | explodeMap | toJSONPretty }}

This template directive pulls in secrets from Vault, explodes the / characters to create a nested structure, and then it outputs the compiled secrets as a JSON structure.

    "database": {
        "username": "value",
        "password": "value"
    },
    "opensearch": {
        "username": "value",
        "password": "value"
    }

This will result in a JSON file that looks like this (based on the above Vault entries):

Incorporating Secrets into the .NET Configuration

At this stage, it’s time to use the generated JSON file consisting of your secrets.

public Startup(IHostingEnvironment env)
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("/vault/secret/appsecrets.json", optional: false, reloadOnChange: true)
        .AddEnvironmentVariables();
    Configuration = builder.Build();
}

The .AddJsonFile(“/vault/secret/appsecrets.json”) helps pull in the secrets and adds them to the configuration pipeline. Any change detected in the “appsecrets.json” triggers a reload to ensure your application is always using the most recent secrets. However, triggering this reload is not enough to have your Dotnet application dynamically update. Structuring your Dotnet application to have dynamically update Services and Configurations is enough for another article.

Setting up Appsettings in .NET Application

Here we define ApplicationSettings – a POCO representing the appsettings. It should have properties that match the keys of our secrets.

public class ApplicationSettings
{
    public DatabaseConfig Database { get; set; }
    public OpensearchConfig Database { get; set; }
}

Using this higher level ApplicationSettings we will then define the expected Database and Opensearch configurations.

public class DatabaseConfig
{
    public string Server { get; set; }
    public string Username { get; set; }
    public string Password { get; set; }
}

public class OpensearchConfig
{
    public string Address { get; set; }
    public string Username { get; set; }
    public string Password { get; set; }
}

Note: This defined additional properties that are not defined in our appsecrets.json file. Generally, most configurations are not secrets and would be provided from other JSON files, environment variables, or ConfigMaps in Kubernetes.

To use these properties, we leverage IOptions in service constructors, like this:

private readonly IOptions _appSettings;
public MyController(IOptions appSettings)
{
    _appSettings = appSettings;
}

This is where you would potentially pull in IOptionsSnapshot or IOptionsMonitor to have services update based on changing configurations.

Assigning Configuration Settings Using .NET GetSection

With the secrets now within the configuration, we can proceed to assign these to our application settings using the Get utility function and the Configuration.GetSection function.

public void ConfigureServices(IServiceCollection services)
{
    services.Configure(Configuration.GetSection("ApplicationSettings"));
}

Wrapping Up

To recap, we’ve successfully gone from creating secrets in HashiCorp Vault to incorporating them effectively within a .NET application. This secure process allows for quick adaptation to new secrets and mitigates the risk of unauthorized secret access. This implementation results in a dynamic application with secret data separate from code, ensuring safe, secure, and robust applications.