Lambda config and secrets (Part 1 of 2)

The SSM Parameter Store is currently the best place to store secrets and other dynamic configuration data for your Lambda functions. We’re going to share some details of why and how we use it. Our example code will written in Node.js, but the points made here apply across all Lambda-supported languages.

For an in-depth look at the benefits of Parameter Store compared to other alternatives, you can’t do much better than Yan Cui’s Blog, but a few of our favourite things about it are as follows:

• Integration with KMS provides easy decryption of secrets
• Parameter Store keeps a history of changes
• It's easy to share configuration between Lambdas
• You can fetch groups of parameters that share a common path

We're going to focus on those last two points, because we've found that they've brought the most value to us. As you'll see, they're closely related.

Adding configuration to SSM Parameter Store

First you need to add some configuration data to the SSM Parameter Store. You can get there through the Systems Manager service (SSM historically stood for Simple Systems Manager) from within the AWS Management Console.

From here you can add multiple parameters. Each parameter requires a name and a value. The value can be plaintext (either a single value or an array) or an encrypted value.

The name can contain forward-slashes which allow you to create a hierarchical structure which you can then use to query groups of parameters by path. For instance /dev/authService/connectionString is equivalent to dev ▶ authService ▶ connectionString

AWS have already written about best practise for organising parameters into hierarchies.

Fetch individual parameters by name, or whole groups by hierarchical path

The AWS SDK has two methods for getting back multiple values from the SSM Parameter Store: getParameters, and getParametersByPath.

To use either of them you will need to bring in the AWS SDK and create a new instance of SSM.

const { SSM } = require('aws-sdk')
const ssm = new SSM()

The getParameters method takes an array of names and returns an array of parameters, whilst getParametersByPath only takes a single string as input. This string is a path to the group of parameters you want to fetch (and can work recursively).

getParameters

ssm.getParameters({
  Names: ['/dev/authService/connectionString', '/dev/authService/token'],
  WithDecryption: true,
})

getPrametersByPath

ssm.getParametersByPath({
  Path: '/dev/authService',
  WithDecryption: true,
  Recursive: true,
})

Both of the above will return the same data, however the latter does so without specifying any individual parameter names.

This is very powerful. In fact it's one of the best things about using the SSM Parameter Store to store your config data. It allows you to maintain a very structured hierarchy and use the least code possible to get what you want from it.

However, it comes with two complications that you’ll need to address if you want to implement this in your own project:

getParametersByPath only takes a single path as input

Say we don’t only have config for my authService in the Parameter Store, but also for our locationService. And say they share a piece of configuration data. A connection string for a shared database, perhaps.

We certainly don’t want to duplicate the connection string parameter by creating both /dev/authService/connectionString and /dev/locationService/connectionString with the same value—sharing configuration is one of the reasons we’re using the Parameter Store in the first place! Instead, we want a shared configuration path alongside our /dev/authService and /dev/locationService paths.

Unfortunately, this leaves us with no choice but to make multiple calls to the AWS API. Having to do this is particularly frustrating because getParameters will take as many names as you like and fetch them in one call. Hopefully AWS will update their API to allow for multiple paths in one call, but until then, you can do something like this:

const getParametersByPath = (path) =>
  ssm.getParametersByPath({ Path: path, WithDecryption: true, Recursive: true })
    .Parameters

return [
  ...getParametersByPath('/dev/authService'),
  ...getParametersByPath('/dev/common'),
]

Mapping Parameter Store values to local variable names can be tricky

If you’re using getParametersByPath, then your parameter names have slashes in. And that means that your parameter names aren’t valid JavaScript variable names. So some work definitely needs to be done in order to generate a suitable name for each value from the Parameter Store.

What’s more, if you have multiple stages for your lambda (which you likely will) and you’re using different config values for each one (which you likely will), then you’ve got another issue. In code you only want to refer to connectionString, but in the Parameter Store each version of the connection string will have a different name (e.g. /dev/common/connectionString and /prod/common/connectionString). So as well as getting rid of illegal characters, you need to remove any parts of the path that will vary stage-to-stage.

We will cover some potential solutions to this particular problem and show you the quickest and easiest way to actually use SSM Parameter Store in part 2!