AWS Step Functions as CloudFormation Custom Resources - Automatic Certificate Creation Across AWS Accounts

Last year I wrote a CloudFormation example which deployed a CodePipeline for the Hugo CMS. This was an almost fully automated solution for a Hugo deployment, the only manual step was to create the needed certificate with the Amazon Certificate Manager.

Some weeks ago AWS added the possibility of fully automated certificate creation via CloudFormation if you add the HostedZoneId to your CloudFormation certificate resource.

This solution is neat but will not work on our company accounts because we have all Route 53 DNS Zones in a different AWS account. Therefore I needed a solution which works fully automated across different AWS accounts.

Searching for examples gave me a good starting point to create my own solution, a custom CloudFormation resource. Here are some examples which will help you to understand custom CloudFormation resources:

Validate ACM certificates in Cloudformation

Custom Certificate Provider with DNS validation support

Automatic Certificate Validation with Certificate Validator

All these examples work perfectly and can be easily modified to work across different AWS accounts but the fact that they use long running Lambda Functions didn’t satisfy me.
Occasionally executing long running Lambda Functions doesn’t cost much but nevertheless I always prefer short running ones and to use AWS Step Functions for the Workflow logic combined with Lambda Functions with a single purpose.

Challenge accepted, let’s create a CloudFormation Custom Resource which will work with Step Functions.

Unfortunately only Lambda Functions or SNS topics may be used as Custom Resource in CloudFormation, so we first have to create a Lambda Function which can be used as Custom Resource in CloudFormation and which interconnects CloudFormation with our Step Functions.

We have 3 components:

You will find all the examples explained below in this AWS_Cloudformation_Examples Github Repo.

The Custom Resource including CertificateStateMachine Step Functions with all Lambda Functions and Roles/Policies can be found in the certificate_xaccount_customresource.yaml
CloudFormation template.

Most of the work is done by the Step Functions called CertificateStateMachine:

CertificateStateMachine starts with a Choice Action and if you look at the CloudFormation template which creates this Step Functions you see that the variable $.RequestType is used as switch. This variable is sent by AWS CloudFormation and will give us the information if this is a Create, Update or Delete request.

Following the Create or Update path will first call the Create step which triggers a Lamba Function called LambdaCreateCertificateRequest. As the name suggests this simple Function calls ACM and requests to create a certificate. We use the parameters HostedZoneId, WebSiteURL and Region which we will get from CustomResourceCertificate whenever this Custom Resource is used in a CloudFormation template
→ find more details later in this post
As response we will get the CertificateArn which we will need in the next steps.

After Wait_10_seconds another Lambda Function LambdaDescribeCertificateRequest is called in step DescribeCert . This function takes the CertificateArn as input and calls ACM again to get the needed DNS CNAME entries for the validation and the ValidationStatus.

CreateDNS triggers LambdaCreateDNSEntry Lambda Function and takes the CNAME entries as input. Here the magic for the cross account creation happens. The Lambda Function will use the ARN of the Role which is created in our Route 53 Domain AWS Account and will call Route 53 to create the DNS Record Set.

CheckCert will again use LambdaDescribeCertificateRequest to get the ValidationStatus of the Cert creation. Cert Ready? will loop using Wait_100_seconds_for_certificate until the ValidationStatus equals SUCCESS

Last Step SendResultCreation calls the Lambda Function LambdaSendResult. This Function returns a success response to AWS CloudFormation via the cfn-response module. This module knows the AWS CloudFormation Endpoint for the response through the variable responseUrl. This variable is provided when AWS CloudFormation calls the ServiceToken of the Custom Resource.
The CertificateArn is used as physicalResourceId for the Custom Resource, so this will be the Return value of the Custom Resource.

The Delete Path starts with the Lambda Function LambdaDescribeCertificateRequest in step DescribeCertDeletion . In contrast to the use of the same Function in the Create Path the CertificateArn is provided via the variable PhysicalResourceId. The response includes the DNS CNAME entries which were used for the validation.

Delete step will call the Lambda Function LambdaDeleteResource. This Function will first delete the DNS Entries in our Route 53 Domain AWS Account, again done via assuming a Role in this Account. Second the Function deletes the according Certificate in ACM.

Last Step SendResultDeletion calls the Lambda Function LambdaSendResult and returns a success response to AWS CloudFormation equal to the use in the Create Path.

Next let’s look at the LambdaCallStateMachine Python Function. The Function can as well be found in the CloudFormation template certificate_xaccount_customresource.yaml.

246          from botocore.exceptions import ClientError
247          import boto3
248          import cfnresponse
249          import os
250          import json
251
252          statemachineARN = os.getenv('statemachineARN')
253
254          def lambda_handler(event, context):
255              sfn_client = boto3.client('stepfunctions')
256              try:
257                  response = sfn_client.start_execution(stateMachineArn=statemachineARN,input=(json.dumps(event)))
258                  sfn_arn = response.get('executionArn')
259                  print(sfn_arn)
260              except Exception:
261                  print('Could not run the Step Functions')
262                  responseData = {}
263                  responseData['Error'] = "CouldNotCallStateMachine"
264                  response=cfnresponse.send(event, context, FAILED, responseData)
265                  return(response)
266              return(sfn_arn)

As you can see on line 252 we will get the ARN of the CertificateStateMachine Step Functions (aka statemachineARN) as environment variable.
This ARN will be automatically filled with the correct ARN of the CertificateStateMachine Step Functions during CloudFormation deployment (Line 242 → statemachineARN : !Ref CertificateStateMachine).

In line 257 we call the Step Functions and provide the Lambda Function input event unchanged as json string to the Step Functions. This event input will be provided by AWS CloudFormation during Custom Resource Creation/Update/Deletion.

This is an example what you can expect in such an input event:

If everything works as expected the Lambda will be terminated and the Step Functions will take care of returning a response to the cfn-response module. If the Step Functions can’t be triggered we will return an error (line 264) through the cfn-response module.

A custom resource in CloudFormation is defined by a Type starting with 'Custom::' and the custom resource name, here 'CreateCertificate'.
The resource must have a ServiceToken. This token represents the ARN of the Lambda Function or SNS Topic which should be called. In this case we import the ARN of the Lambda Function LambdaCallStateMachine (which was already created by the certificate_xaccount_customresource.yaml CloudFormation template).
This custom resource needs 3 additional properties, the WebSiteURL for which the certificate should be created, the HostedZoneId of the Route 53 domain in which the needed DNS entry for validation will be created and the Region where the certificate should be created.
We already saw these 3 properties inside the Step Functions where LambdaCreateCertificateRequest is called.

37  CustomResourceCertificate:
38    Type: 'Custom::CreateCertificate'
39    Properties:
40      ServiceToken: !ImportValue LambdaCallStateMachineCertArn
41      WebSiteURL: !Ref WebSiteURL
42      HostedZoneId: !Ref HostedZoneId
43      Region: !Ref Region

You can find the full version of the CloudFormation template which creates the certificate here.

I tried to write the Lambda Functions generic so that they can be reused in other Step Functions. This gives us the freedom to use them in a second Step Functions example called DNSStateMachine. These Step Functions will be used for a Custom CloudFormation Resource which creates DNS entries in our Route 53 Domain AWS Account.

As you can see we reuse the same Lambda Functions LambdaCreateDNSEntry, LambdaDeleteResource and LambdaSendResult. You find the DNSStateMachine example in the CloudFormation template certificate_xaccount_customresource.yaml as well.

This example showed how you can combine a Custom CloudFormation Resource with Step Functions and automatically create an ACM certificate even if the Route 53 Domain for validation is in another AWS account. This gives you an idea how you can start using Step Functions for your own CloudFormation resources.