Updating Datomic On-Prem Running on AWS

The Datomic On-Prem documentation provides helpful instructions for getting running on AWS, but doesn’t provide much help if you need to make changes once you’re up and running. This post aims to fill in the gaps so you know how to safely update your running Datomic-on-AWS deployment.

A Mild Disclaimer

I should add that all of this information was gleaned either from experimenting with the datomic tool, or from a careful reading of the public documentation. I can’t guarantee that all of the information is correct or complete, but all of it was born from a real world need to safely update our production database (an operation which was completely successfully).

Motivations

There are a number of reasons why you might want to use the Datomic tools to update your Datomic-on-AWS deployment; here are a few:

• You want to change the memory settings for the transactor.
• You want to enable logging to S3.
• You want to enable memcached for an additional layer of data caching.
• You want to enable CloudWatch metrics.

Note: If you’ve started using Datomic recently then there’s a decent chance you’re running on Datomic Cloud, an AWS Marketplace-managed solution, to which this information does not apply.

Overview

The bin/datomic tool provided in the Datomic distribution provides a number of commands which ultimately generate a CloudFormation template which can be used to launch a Datomic AMI on an EC2 instance—but that’s hardly all they do.

There are two critical things to understand about Datomic’s AWS commands:

1. Despite outputting files which form inputs to later steps in the process, these commands mostly are side-effecting (by modifying resources on AWS).
2. The output files of these commands sometimes contain generated data describing the state of AWS, and are intended to be the repository for that data. More on this later, but when the Datomic docs say (e.g., of the ensure-cf command) that the input and output file names “can be the same”, you nearly always want them to be the same. If you think of the output file as a regenerate-able intermediate file, you’re likely to be surprised when the file changes on a subsequent run of the command. The process is only idempotent once all of the variables whose value can be generated has been generated and stored, and this often only happens after the command has been run at least once. I recommend committing these files to revision control (both before and after running any commands).

The basic process to create or update your Datomic stack on AWS is the following series of commands:

1. bin/datomic ensure-transactor
2. bin/datomic ensure-cf
3. bin/datomic create-cf-template
4. bin/datomic create-cf-stack (in the case of creating a new AWS stack) or uploading the CloudFormation template via the AWS Console (for updating an existing stack).

Note: Usage of the word “ensure” in the command names above implies that the command in question creates-or-updates the resource(s) in question (think: “ensure exists”).

You’ll want to run all commands from your local Datomic directory (if you run them from outside this directory, bin/datomic will likely not be able to access the files referenced). The following examples will assume the current working directory is the Datomic distribution and you have a parallel folder named aws containing your configuration files (i.e. ls ../aws will list those files).

Command Details

ensure-transactor

This command takes a transactor configuration file as input (here called my-transactor.properties). The initial template for this file is typically one of Cognitect’s provided per-storage example files (e.g. config/samples/ddb-transactor-template.properties for DynamoDB). It defines the runtime configuration for the transactor (which will get bundled into the AMI) and is also the source of some data used in configuring AWS.

The ensure-transactor command will “create the necessary AWS constructs, and emit an updated properties file containing information about what it created”—this includes the DynamoDB table (if using that storage backend)¹, IAM roles, S3 bucket, and security group(s)).

You will want to update your input file in place (by providing the same name for input and output files) to capture the generated values for fields like aws-transactor-role, aws-peer-role, and aws-s3-log-bucket-id. If you fail to include these input values on subsequent command runs, the command will make sure to generate new, unused names and new backing AWS resources (e.g. IAM roles)–not what you want when your intention is to update your existing infrastructure.

A typical invocation looks like this:

bin/datomic ensure-transactor ../aws/my-transactor.properties ../aws/my-transactor.properties

If your input file was the output of a previous invocation of this command (as above), then you can run the command again to update the existing AWS resources.

ensure-cf

This command takes a file (here called my-cf.properties) as input that contains some configuration for the transactor machine and Java settings that will become a part of the CloudFront configuration, which will ultimately construct the transactor’s EC2 instance. There is a template for this file at config/samples/cf-template.properties in the Datomic distribution.

The ensure-cf command will “create the necessary AWS constructs, and emit an updated properties file containing information about what it created” (this includes IAM roles).

A typical invocation looks like this:

bin/datomic ensure-cf ../aws/my-cf.properties ../aws/my-cf.properties

If your input file was the output of a previous invocation of this command (as above), then you can run the command again to update the existing AWS resources.

ensure-cf-template

This command generates the CloudFront template. I don’t believe this process creates any AWS resources, but it does appear to query them (and thus requires an internet connection to succeed).

A typical invocation looks like this:

bin/datomic create-cf-template ../aws/my-transactor.properties ../aws/my-cf.properties ../aws/my-cf.json

The output file here (my-cf.json) is derived from its inputs in a deterministic fashion, and thus the command is safe to re-run as needed.

create-cf-stack

To create a new EC2 instance (via CloudFormation), use the create-cf-stack command:

bin/datomic create-cf-stack <aws-region> MyTransactor ../aws/my-cf.json

This command is not used when updating an existing CloudFormation stack.

Updating an Existing CloudFormation Stack

The Datomic tooling does not help with applying an updated CloudFormation template (my-cf.json) to an existing installation. The easiest way to handle this is simply to replace the existing CloudFormation template via the AWS Console. This will update the autoscaling group but not update the existing instance.

Putting it All Together

If you follow the best practices described here, and always use the input files as output destinations for the commands, the script for updating your Datomic-on-AWS stack is safe and formulaic:

1. Update the my-cf.properties and / or my-transactor.properties as desired.

2. Regenerate all derived files, keeping in mind that these commands do alter AWS resources:

   ```sh
   bin/datomic ensure-transactor ../aws/my-transactor.properties ../aws/my-transactor.properties
   bin/datomic ensure-cf ../aws/my-cf.properties ../aws/my-cf.properties
   bin/datomic create-cf-template ../aws/my-transactor.properties ../aws/my-cf.properties ../aws/my-cf.json
   ```
   

3. Upload the replacement CloudFormation template (cf.json) via the AWS Management Console. This will update the autoscaling group but not update the existing instances.

4. Terminate the existing instance via the EC2 console and allow it to be replaced by the autoscaling rules (this will take about 5 minutes, so should obviously be done when some downtime is acceptable).

5. Commit the updated files to version control, assuming everything is working correctly.

Conclusion

This process is pretty simple and smooth if you follow it as intended, but the intended path was not immediately obvious (at least to me). I hope this information can save you at least a few hours, as it would have for me had it been available at the time. Please let me know if anything is unclear or if I missed any critical details. Happy hacking.