AWS CDK Cloudformation Integration Tests Using Jest Snapshot Tests

The challenge with autogenerated infrastructure files such as Cloudformation and ARM

As you might know I’m a big fan of Azure and have a fair bit of experience. It wasn’t until I started working at Greenbyte this year that I started getting familiar with AWS, and I’m not going to lie- it has been challenging. Nonetheless, one of my first tasks at work together with another developer, Per, was to set up a stack that manages the data stream from wind turbines or solar panels for our customers and ensures that the data, calculated and aggregated, finds its way to a separate storage defined by the customer with a reliable resilience policy. The stack consists of several parts, SQS, DLQ, lambdas and intermediate bucket storage and a few more things. We decided on defining the stack using C#, with a tool called CDK. Although the stack is defined in C# the tools produces a JSON template similar to the ARM templates we use with Azure. The stack is nested and complex, and the same stack is used for each customer albeit with minor configuration differences.

The deploy is easily done from the command line, but this also means that one can easily accidentally change something in the stack and deploy those change.

When we started working on the stack we decided early on to this test driven, as much as possible anyway. We have unit tests for our lambdas, integration tests for areas of the code that rely on storage (we support different types of storage), in addition to manual tools for testing the stack pipeline, and clever logging. The biggest challenge was testing the generated templates though. Change in code might not necessarily break a test, and maybe the change is intentional.

AWS CDK has a template and diff feature, but the diff between templates does not work on nested stacks, and using only diff will show you diffs between the deployed stack and the stack, which might have different variables and parameters.

I solved this by using snapshot tests with Jest.js. The CDK tool has support for generating and comparing templates, but not for nested stacks.

About the testing framework Jest and snapshots

Jest) is a JavaScript testing framework that simplifies testing. One of the features they have is snapshot tests. These are typically used for rendered UI components, for example react complements. First time you run the tests a snapshot is created if it doesn’t exist, and future test runs will compare the new input to the existing snapshot. You can easily update a snapshot and remove old snapshots from the command line.

Using Jest snapshots with AWS CDK and Cloudformation files

This is how our snapshot tests are run:

1. We build the solution

2. We package the stack using a AWS tool

3. We synthesize the cloud formation file

4. We run the tests that compare a snapshot of the generated file with a previous snapshot

5. We do something with the result (generate warning or error in the build pipeline)

Although this is done with AWS cloudformation files you can use similar steps to compare Azure ARM templates. JestJS doesn’t care.

Jest has support for ignoring certain properties, however, it doesn’t support nested properties which we have a lot of in a nested stack.

Here is a simple how-to (for readability I’m not using a nested stack):

Assuming you have CDK and dotnet tools installed

Create a new CDK app:

cdk init app --language csharp 

Add to stack– [for example a lambda as in my example in this GitHub] (https://github.com/IrisClasson/AwsCdkSnapshotTests)

Create a folder where you can output the cloudformation file (example: /template/)

Create a folder for the tests and snapshots (example: /test/)

Install JestJs:

npm install --save-dev jest -g 

Add the following section to your package.json in the test folder:

{ 
  "scripts": { 
    "test": "jest" 
  } 
} 

In the test folder create your first test(s)

To run the tests, you will need to: build the solution, create packages, synthesize the cloudformation file (the template), and then run the tests using that file.

I recommend that you script the steps so they can be run as one unit.

1. Build
2. Create package
3. Synthesize cloudformation template and output to custom folder
4. Optional: remove non-template files
5. Run tests

Example with bash (the file is run from the root directory):

#!/bin/bash 

dotnet build src/SnapshotTestExample.sln -c Release 

dotnet lambda package -pl src/ExampleHandler/ -c Release 

cdk synthesize -o template 

find ./template -type f -not -name '*template*' -delete 

npm --prefix test/ test 

When the tests are run the first time, or after snapshots have been removed, a snapshots folder is created, with one snapshot per test file. The snapshot files have the following naming convention:

[test file name].snap

As mentioned earlier, the cloudformation file generates hashes that can show false positive diffs when the snapshots are compared. One way to handle this is to write a custom snapshot serializer for Jest. I’ll cover that in a separate post.

Jest snapshot test results

If I was to change the Runtime version for the function by accident (in the stack declaration) from 3.1 to 2.1, and then run the tests, the test would fail with the following:

FAIL ./stack.test.js 
  ● Stack matches snapshot 
  
    expect(received).toMatchSnapshot() 
  
    Snapshot name: `Stack matches snapshot 1` 
  
    - Snapshot  - 1 
    + Received  + 1 
  
    @@ -67,11 +67,11 @@ 
                "Fn::GetAtt": Array [ 
                  "stringlengthServiceRoleD792339E", 
                  "Arn", 
                ], 
              }, 
    -         "Runtime": "dotnetcore3.1", 
    +         "Runtime": "dotnetcore2.1", 
            }, 
            "Type": "AWS::Lambda::Function", 
          }, 
          "stringlengthServiceRoleD792339E": Object { 
            "Metadata": Object { 
  
      2 | 
      3 | test('Stack matches snapshot', () => { 
    > 4 |   expect(data).toMatchSnapshot(); 
        |                ^ 
      5 | }); 
      6 | 
      7 | 
  
      at Object.<anonymous> (stack.test.js:4:16) 

The result shows the failed test and lines that to not match. If the change was intentional you can update the snapshot with the following command:

npm test -- -u 

If you aren’t in the root folder you can use –prefix to specify the folder.

Npm --prefix test/ test -- -u 

– -u creates a new snapshot and removed the old ones.

Expect more blog posts on this topic as I have a few drafts on the topic of custom serializers for the snapshots, and examples using Azure ARM.

Comments

Last modified on 2020-08-13