Read Model using AWS Api Gateway and S3 - Explained

In this mini-series of blogs I discuss a technique to provide fast read time to a consumer facing website using a read model architecture.

In the first article I explained the problem and proposed solution.

In the second article I explained how to deploy the solution.

In this article I explain how the solution was created.

The datastore

First thing I do is create an S3 bucket as a data store for our json payload.

This template section:


"DataStore": {
    "Type": "AWS::S3::Bucket",
    "Properties": {
        "BucketName": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-data-store"
                ]
            ]
        }
    }
},

Creates this S3 bucket: resulting s3 bucket

The datastore writer

To provide for least privileges, I create a specific user for writing to the S3 bucket.

This user will be restricted (via policy) to our S3 bucket.

The template then generates API keys - which are made available in the Cloud Formation output. The API keys will be used to update the json payload.

This template section:


"DataStoreWriter": {
    "Type": "AWS::IAM::User",
    "Properties": {
        "UserName": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-data-store-writer"
                ]
            ]
        },
        "Policies": [
            {
                "PolicyName": {
                    "Fn::Join": [
                        "",
                        [
                            {
                                "Ref": "StackName"
                            },
                            "-data-store-writer"
                        ]
                    ]
                },
                "PolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Action": [
                                "s3:*"
                            ],
                            "Resource": {
                                "Fn::Join": [
                                    "",
                                    [
                                        {
                                            "Fn::GetAtt": [
                                                "DataStore",
                                                "Arn"
                                            ]
                                        },
                                        "/*"
                                    ]
                                ]
                            }
                        }
                    ]
                }
            }
        ]
    }
},
"DataStoreWriterAccessKey": {
    "Type": "AWS::IAM::AccessKey",
    "DependsOn": "DataStoreWriter",
    "Properties": {
        "Serial": 1,
        "Status": "Active",
        "UserName": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-data-store-writer"
                ]
            ]
        }
    }
},

Creates this user:

resulting writer user

The API Gateway role

Ahead of creating the API Gateway itself, I want to create a role - again for security using least privilege.

The important thing here is that the role allows (via the AssumeRole) the API Gateway to assume the read privileges to the S3 bucket.

This template section:


"ApiGatewayRole": {
    "Type": "AWS::IAM::Role",
    "Properties": {
        "RoleName": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-api-gateway-role"
                ]
            ]
        },
        "AssumeRolePolicyDocument": {
            "Statement": [
                {
                    "Sid": "",
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "apigateway.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
},
"ApiGatewayDataStorePolicy": {
    "Type": "AWS::IAM::Policy",
    "Properties": {
        "PolicyName": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-data-store-read-only"
                ]
            ]
        },
        "PolicyDocument": {
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "s3:Get*"
                    ],
                    "Resource": {
                        "Fn::Join": [
                            "",
                            [
                                {
                                    "Fn::GetAtt": [
                                        "DataStore",
                                        "Arn"
                                    ]
                                },
                                "/*"
                            ]
                        ]
                    }
                }
            ]
        },
        "Roles": [
            {
                "Ref": "ApiGatewayRole"
            }
        ]
    }
},

Creates this role: resulting gateway role

API Gateway

We then build out the API Gateway for the public interface.

This creates a new API Gateway with "sample" resource with a "get" method which proxies to the S3 bucket.

As per the above, the API Gateway, assumes the read privileges for the S3 bucket using the above role.

This template section:


"ApiGateway": {
    "Type": "AWS::ApiGateway::RestApi",
    "Properties": {
        "Name": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    " Api"
                ]
            ]
        },
        "Description": "A simple Read Model example using S3",
        "FailOnWarnings": true
    }
},
"SampleResource": {
    "Type": "AWS::ApiGateway::Resource",
    "Properties": {
        "RestApiId": {
            "Ref": "ApiGateway"
        },
        "ParentId": {
            "Fn::GetAtt": [
                "ApiGateway",
                "RootResourceId"
            ]
        },
        "PathPart": "samples"
    }
},
"SamplesGetMethod": {
    "Type": "AWS::ApiGateway::Method",
    "Properties": {
        "AuthorizationType": "NONE",
        "HttpMethod": "GET",
        "Integration": {
            "Type": "AWS",
            "IntegrationHttpMethod": "GET",
            "IntegrationResponses": [
                {
                    "StatusCode": "200",
                    "SelectionPattern": "200"
                }
            ],
            "Uri": {
                "Fn::Join": [
                    "/",
                    [
                        "arn:aws:apigateway:eu-west-1:s3:path",
                        {
                            "Fn::Join": [
                                "",
                                [
                                    {
                                        "Ref": "StackName"
                                    },
                                    "-data-store"
                                ]
                            ]
                        },
                        "payload.json"
                    ]
                ]
            },
            "Credentials": {
                "Fn::GetAtt": [
                    "ApiGatewayRole",
                    "Arn"
                ]
            },
            "PassthroughBehavior": "WHEN_NO_MATCH"
        },
        "ResourceId": {
            "Ref": "SampleResource"
        },
        "RestApiId": {
            "Ref": "ApiGateway"
        },
        "MethodResponses": [
            {
                "StatusCode": 200
            }
        ]
    }
},

Creates this API Gateway: resulting api gateway

We make a deployment

The API Gateway on its own is a definition - you need a deployment before it is available for use.

This makes a deployment using the above API Gateway as "v1".

This template sections:


"ApiGatewayDeployment": {
    "Type": "AWS::ApiGateway::Deployment",
    "DependsOn": "SamplesGetMethod",
    "Properties": {
        "RestApiId": {
            "Ref": "ApiGateway"
        },
        "Description": "Initial Deployment",
        "StageName": "v1"
    }
}

Creates this deployment: resulting deployment

And finally

We have our outputs so we can use the solution.

The bucket and access key/ secret allow us to upload the json payload.

The sample url then allows us to valid that the solution all works.

This template sction:


"Outputs": {
    "SamplesUrl": {
        "Description": "Url to access the Samples data",
        "Value": {
            "Fn::Join": [
                "",
                [
                    "https://",
                    {
                        "Ref": "ApiGateway"
                    },
                    ".execute-api.eu-west-1.amazonaws.com/v1/samples"
                ]
            ]
        }
    },
    "S3BucketName": {
        "Description": "Name of S3 bucket created",
        "Value": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-data-store"
                ]
            ]
        }
    },
    "DataStoreAccessKeyId": {
        "Description": "Access Key Id for the Data Store (S3)",
        "Value": {
            "Ref": "DataStoreWriterAccessKey"
        }
    },
    "DataStoreSecretAccessKey": {
        "Description": "Secret Access Key for the Data Store (S3)",
        "Value": {
            "Fn::GetAtt": [
                "DataStoreWriterAccessKey",
                "SecretAccessKey"
            ]
        }
    },
    "DataStoreUser": {
        "Description": "User to upload to the Data Store (S3)",
        "Value": {
            "Fn::Join": [
                "",
                [
                    {
                        "Ref": "StackName"
                    },
                    "-data-store-writer"
                ]
            ]
        }
    }
    }

Creates these outputs: outputs

Further reading

This solution hasn't been used in anger, so there are a few things that should probably be validated before production use:

Concurrency - I'm fairly certain that there shouldn't be any blocking on the read while new payloads are uploaded. You should continue to read the old version until the new version has been completed. Note also that S3 will replicate across multipule relicas - I would expect that it is possible to get different versions of the payload depending on the replica you hit.

ETag - I believe that S3 will take advantage of ETag - which should allow you to reduce network traffic if the payload hasn't changed. I've not tested if the API Gateway will honour the ETag mechanism - well worth trying.

About the author:

Mark Taylor is an experience IT Consultant passionate about helping his clients get better ROI from their Software Development.

He has over 20 years Software Development experience - over 15 of those leading teams. He has experience in a wide variety of technologies and holds certification in Microsoft Development and Scrum.

He operates through Red Folder Consultancy Ltd.