Goal

The objective of this integration is to be able to work with HashiCorp Vault from the platform, so that its capabilities can be used transparently to a user.

There is a functionality in the Platform that allows encrypting and decrypting attributes of an Entity (encryption/decryption functionality) automatically from a master key stored in the platform. With this integration, the Vault manages this master key.

Integration in the Onesait Platform

We have integrated this tool into the platform so that the encryption and decryption of entity data is delegated to it, thus it is the Vault that manages the data encryption key:

Open

As an example, let's take this entity that has the attributes “file” and “stages” marked as encryptable:

Open

When we query the data, we will see it in plain view, since the vault is being used below to encrypt/decrypt:

However, at rest, they will be encrypted:

What is HashiCorp Vault

HashiCorp’s Vault is a popular tool used to securely manage sensitive information in today's applications.

As we see in the image, HashiCorp Vault comes with several pluggable components called secrets engines and authentication methods that allow it to integrate with external systems. The purpose of these components is to manage and protect your secrets in dynamic infrastructures (for example, database credentials, passwords, or API keys).

What does a Vault solve?

Before we look at the Vault in detail, let's look at the problem it tries to solve: the management of sensitive information.

Most apps need access to sensitive data to function, for example, an app might have a username/password configured somewhere to connect to its database, or need API keys to integrate with other service providers, like payment gateways, logistics,…

Database credentials and API keys are two examples of sensitive information that we need to store and make available to our applications securely.

A simple solution is to store those credentials in a config file and read them at boot time, although the problem with this approach is obvious: whoever has access to this file shares the same database privileges as our application.

We can also encrypt these files, although we make things a little more difficult by encrypting those files, although since our application will know how to decrypt them, it will only give a "false" sense of security.

Today's applications and cloud environments add additional complexity: distributed services, multiple databases, messaging systems, etc., have sensitive information scattered everywhere, increasing the risk of a security breach.

This is where a Vault fits in.

What is HashiCorp Vault?

HashiCorp Vault addresses the problem of managing sensitive information (secret in Vault slang). "Manage" in this context means that Vault controls all aspects of sensitive information: its generation, storage, use, and last but not least, its revocation.

HashiCorp offers two versions of Vault. The open-source version and a paid version, which includes technical support in different SLAs and additional features, such as HSM (Hardware Security Module) support.

HashiCorp Vault Architecture

Vault's architecture is simple. Its main components are:

• A persistence backend: for the storage of all the secrets.

• An API server that handles client requests and performs operations on secrets.

• A number of secret engines, one for each type of secret supported.

By delegating all secret management to Vault, we can mitigate some security issues:

• Our apps no longer have to store them, just request them from Vault when needed, and later throw them away.

• We can use short-lived secrets, thereby limiting the "window of opportunity" in which an attacker can use a stolen secret.

• Vault encrypts all data with an encryption key before writing it to the vault. This encryption key is encrypted by another key: the master key, which is only used at startup.

A key point in the Vault implementation is that Vault does not store the master key on the server. This means that even Vault cannot access the save data after boot. At this point, a Vault instance is said to be in a "sealed" state. Once unsealed, Vault will be ready to accept API requests.

Authentication

To access the secrets in the Vault, the client must authenticate using one of the supported methods. The simplest method uses tokens (strings that are sent on each API request using a special HTTP header).

When first installed, Vault automatically generates a "root token". This token is the equivalent of the root superuser on Linux systems, so its use should be kept to a minimum. As a best practice, we should use this root token only to create other less privileged tokens, and then revoke it (this is not a problem, however, since we can later generate another root token using unseal keys).

Vault also supports other authentication mechanisms such as LDAP, JWT or TLS Certificates. All of these mechanisms are based on the basic token mechanism: once Vault validates our client, it will provide a token that we can use to access other APIs.

Tokens have some associated properties. The main ones are:

• A set of associated Policies (next section).

• Time-to-live.

• Whether it can be renewed.

• Max Usage Count.

• Unless otherwise stated, tokens created by Vault will form a parent-child relationship. A child token can have at most the same privilege level as its parent, although it is normal to create a child token with restrictive policies.

• Another key point about this relationship: When we invalidate a token, all child tokens and their descendants are also invalidated.

Policies

Policies define exactly which secrets a client can access, and what operations the client can perform on them. Let's see what a simple policy looks like:

Here we have used the HCL (HashiCorp Configuration Language) syntax to define our policy. Vault also supports JSON for this purpose.

Policies in Vault are "denied by default". A token associated with this example policy will have access to the secrets stored in secret/accounting and nothing else. At creation time, a token can be associated with multiple policies. This is very useful because it allows us to create and test smaller policies, and then apply them as needed.

If we update a given policy, all the associated tokens will be affected immediately.

The policies described so far are also called ACL Policies.

Vault also supports two additional policy types: EGP and RGP policies only available in paid versions.

When available, they allow us to factor additional attributes such as time of day, multi-factor authentication, customer network origin, etc. into our policies. For example, we can define a policy that allows access to a certain secret only during business hours.

Types of secrets

Vault supports a number of different secret types that serve different use cases:

• Key-Value: Simple static key-value pairs.

• Dynamically generated credentials: Generated by Vault at the request of a client.

• Cryptographic keys: Used to perform cryptographic functions with customer data

Each type of secret is defined by the following attributes:

• A mount point, which defines its REST API prefix.

• A set of operations exposed through the corresponding API.

• A set of configuration parameters.

We can access a given secret instance via a path, similar to a directory tree on a file system. The first component of this path corresponds to the mount point where all secrets of this type are located.

For example, the string secret/my-application corresponds to the path where we can find key-value pairs for my-application.

Key-Value Secrets

Key-value secrets are pairs available under a given path.

For example, we can store the pair foo=bar in the path /secret/my-application. and later, we use the same path to retrieve the same pair or pairs: multiple pairs can be stored in the same path.

Vault supports three types of key-value secrets:

• Unversioned key pairs, where updates replace existing values.

• Versioned key pairs, which retain up to a configurable number of old versions

• Cubbyhole, a special type of unversioned key pairs whose values are circumscribed to a given access token

Key-value secrets are static by nature, so there is no concept of expiration associated with them. In these cases, credential updates are a semi-manual process, typically requiring someone to acquire new credentials and use the Vault command line or user interface to enter the new values.

The main use case for this type of secret is to store credentials to access external systems, such as API keys.

Dynamically generated secrets

Dynamic secrets are generated by Vault on the fly when requested by an application.

Vault supports several types of dynamic secrets, including the following:

• Database credentials.

• SSH key pairs.

• X.509 certificates.

• AWS credentials.

• Google Cloud service accounts.

• Active Directory accounts.

They all follow the same usage pattern. First, we configure the secrets engine with the necessary details to connect to the associated service. Then, we define one or more roles, which describe the creation of the secret itself.

Let's take the database secrets engine as an example:

• First, we need to configure Vault with all the user database connection details, including the credentials of a pre-existing user with administrator privileges to create new users.

• We then create one or more roles (Vault roles, not Database roles) that contain the actual SQL statements used to create a new user. These typically include not only the user create statement, but also all the grant statements needed to access schema objects (tables, views, etc.).

• When a client accesses the corresponding API, Vault will create a new temporary user in the database using the statements provided, and return their credentials. The client can then use those credentials to access the database during the period defined by the time-to-live attribute of the requested role.

• Once a credential reaches its expiration time, Vault will automatically revoke any privileges associated with this user. A customer can also request Vault to renew those credentials. The refresh process will only occur if supported by the specific database driver and allowed by the associated policy.

Cryptographic keys

This engine handles cryptographic functions like encryption, decryption, signing, etc. All of these operations use cryptographic keys generated and stored internally by Vault. Unless explicitly told to do so, Vault will never expose a given cryptographic key.

The associated API allows customers to send Vault data in plain text and receive an encrypted version of it. The opposite is also possible: We can send encrypted data and receive back the original text.

Currently, there is only one such engine: the Transit engine. This engine supports the most popular key types, such as RSA and ECDSA, and also supports convergent encryption. When this mode is used, a given plaintext value always results in the same ciphertext.

For example, we can use this mode to encrypt credit card numbers in a transaction log table. With converged encryption, every time we insert a new transaction, the encrypted value of the credit card would be the same, allowing normal SQL queries to be used for reporting, searching, etc.