December 31st, 2020

Integrating Neo4j with Hashicorp Vault for secure secret management

What do you use microservices for?

Microservices is a system architecture where the software is built like small, independent building blocks that communicate with each other through a network, working together to provide a service we want. This pattern gives us many benefits:

Less coupling, making the system easier to understand
Easier to scale and to ensure High Availability
Easier to develop and to deploy, because you can deploy each block separately

Companies are often choosing a microservices architecture because of these benefits. As the system is split into smaller pieces, another benefit that sometimes gets overlooked is that you can use NoSQL database models to provide the best workload for every microservice.

However, choosing this system pattern does come with some drawbacks. Having each part of a system work as its own system may imply to some people that these parts should each work individually and have their own identities, hardware resources, etc. This approach leads to a system with multiple databases, users, APIs, configurations and so on, which makes it much more complex to operate and secure.

Microservices systems tend to have variable databases

In this type of scenario, there are a lot of services communicating with each other as an independent unit, requesting access and data, having their own credentials, authentication methods and API tokens. This setup “sprawls” these important pieces of information all over our infrastructure. So how do we make everything secure? Enter HashiCorp Vault.

How does HashiCorp Vault work?

Vault is an open-source software created by HashiCorp that addresses the scenario above by providing a centralized security entity. This entity applies security best practices when dealing with multiple and many moving parts like authentication providers, user credentials, and encryption. It does so by:

Centralizing security information like credentials in many secrets engines
Communicating with multiple authentication providers like AWS, LDAP, etc.
Managing TLS certificates
Auditing all the accesses, so we can know who’s accessing what
Making everything ephemeral, so that in case of a leak (through the Syslog or human error), this leaked credential will expire

As the scope of this article is about databases, to make a database secure, we need:

Encryption at rest and on transit
Least privilege access grants
Credential rotations
Dynamically generated accesses
Access auditing

Vault provides every one of these features through its Database Secrets Engine.

The main challenge

One of Translucent Computing’s main pillars is security and privacy, so we make sure that our solutions come with bleeding-edge security solutions. We started using Vault to secure databases – one of these is the Neo4j graph database.

When we looked at the Vault secret engine, we found that Neo4j is not one of the databases included on their list of built-in solutions:

At the writing of this post, there are over a hundred different databases out there, so it stands to reason that Vault may not have support for all of them. HashiCorp knows this, so they’ve built the entire database secret engine as a plugin system.

Here is a look at Hashicorp’s database secret engine

After acknowledging this fact, we started developing our own Vault plugin for Neo4j. According to HashiCorp documentation, we only had to implement a simple interface:

type Database interface {

// Initialize the database plugin. This is the equivalent of a constructor for the database object itself.
Initialize(ctx context.Context, req InitializeRequest) (InitializeResponse, error)

// NewUser creates a new user within the database. This user is temporary in that it will exist until the TTL expires.
NewUser(ctx context.Context, req NewUserRequest) (NewUserResponse, error)

// UpdateUser updates an existing user within the database.
UpdateUser(ctx context.Context, req UpdateUserRequest) (UpdateUserResponse, error)

// DeleteUser from the database. This should not error if the user didn't exist prior to this call.
DeleteUser(ctx context.Context, req DeleteUserRequest) (DeleteUserResponse, error)

// Type returns the Name for the particular database backend implementation. This type name is usually set as a constant within the database backend
Type() (string, error)

// Close attempts to close the underlying database connection that was established by the backend.
Close() error

}

Now that we have a direction and a destination, it’s time to build our car to get there. To do that, we implement the plugin in Golang. Fortunately, there’s a Golang Neo4j driver available.

After the struggle of getting used to a new language and exploratory failings, we managed to implement the Neo4j plugin and we were ready for a test drive. To do that, we had to build and deploy it into Vault.

To build it, use the go build command, which gathers all your files and compiles them into a binary. Now that we have a binary, we just need to put it into the plugin directory for Vault, which must be configured.

Results

After understanding Vault internals and deploying our own database engine plugin, here are some screenshots from a demo:

We started with this command:

vault server -dev -dev-root-token-id=root -dev-plugin-dir=./ -log-level

…which starts a development Vault server with an easy root token to manage Vault and configure the plugin directory, so it can find my plugin binary.

Here is my Vault server with the plugin fired up and ready to go.

Afterwards, we spun up a Neo4j on my local Docker and queried the users from the database:

Then we configured the connection between the new Neo4j instance and the Vault plugin using the Vault CLI:

// Enabling the database secret engine
vault secrets enable database

// writing my database configurations on the Vault Server
 vault write database/config/neo4j-plugin \
 plugin_name=neo4j-plugin \
 allowed_roles="readonly" \
 uri=bolt://192.168.99.101:7687 \
 username=neo4j \
 password=test \
 database=neo4j \
 verify_connection=true

If you check the code above, you’ll see that only the “readonly” role is able to access this plugin, so let’s create it:

vault write database/roles/readonly \
db_name=neo4j-plugin \
default_ttl="2m" \
max_ttl="10m"

Now, applications can have dynamically generated credentials for the Neo4j database, making this more secure and auditable:

vault read database/creds/readonly
Result:
Key                Value
lease_id           database/creds/readonly/ibAO9AwunXSfbfUhzvffJQHn
lease_duration     2m
lease_renewable    true
password           LyiYsbxm4Mj-w9Vpoowq
username           v_token_readonly_dfvqcgbux6fea9rnebuw_1608060926

As you can see on the command results, this new user will only live for the next two minutes, so even if it leaks through the application or over the network, it will be a very small leakage window.

Let’s check the database:

Here is an ephemeral user that'll be used by the application for the next two minutes, then it will expire. — Here is an ephemeral user that’ll be used by the application for the next two minutes, then it will expire.

Are we secure now?

Not yet, because what about the Neo4j user that we just used to let Vault access the database? What happens if it leaks? We have a solution to that as well.

Vault also comes with a root credential rotation feature, so we can even secure that problem. Just have this command on CLI:

vault write database/rotate-root/neo4j-plugin

Conclusion

Now, we’ve guaranteed that our next Neo4j database deployments will be secure and integrated with our chosen security technology.

We learned:

How to make least privilege access grants through the roles
How to dynamically generate credentials through vault read database/creds/readonly
How to rotate the root credential

So how secure is Vault?

Vault database plugins allow the addition of TLS certificates on the connection. But we decided to not use TLS certificates because there’s a better solution named Service Mesh, which we’ll blog about in our posts. Stay tuned!

0 0 votes

Article Rating