In this mini-series of blogs I discuss a technique to provide fast read time to a consumer facing website using a read model architecture.
In the first article I explained the problem and proposed solution.
In the second article I explained how to deploy the solution.
In this article I explain how the solution was created.
The datastore
First thing I do is create an S3 bucket as a data store for our json payload.
This template section:
"DataStore": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store"
]
]
}
}
},
Creates this S3 bucket:
The datastore writer
To provide for least privileges, I create a specific user for writing to the S3 bucket.
This user will be restricted (via policy) to our S3 bucket.
The template then generates API keys - which are made available in the Cloud Formation output. The API keys will be used to update the json payload.
This template section:
"DataStoreWriter": {
"Type": "AWS::IAM::User",
"Properties": {
"UserName": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store-writer"
]
]
},
"Policies": [
{
"PolicyName": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store-writer"
]
]
},
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"DataStore",
"Arn"
]
},
"/*"
]
]
}
}
]
}
}
]
}
},
"DataStoreWriterAccessKey": {
"Type": "AWS::IAM::AccessKey",
"DependsOn": "DataStoreWriter",
"Properties": {
"Serial": 1,
"Status": "Active",
"UserName": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store-writer"
]
]
}
}
},
Creates this user:
The API Gateway role
Ahead of creating the API Gateway itself, I want to create a role - again for security using least privilege.
The important thing here is that the role allows (via the AssumeRole) the API Gateway to assume the read privileges to the S3 bucket.
This template section:
"ApiGatewayRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-api-gateway-role"
]
]
},
"AssumeRolePolicyDocument": {
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "apigateway.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
}
},
"ApiGatewayDataStorePolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store-read-only"
]
]
},
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*"
],
"Resource": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"DataStore",
"Arn"
]
},
"/*"
]
]
}
}
]
},
"Roles": [
{
"Ref": "ApiGatewayRole"
}
]
}
},
Creates this role:
API Gateway
We then build out the API Gateway for the public interface.
This creates a new API Gateway with "sample" resource with a "get" method which proxies to the S3 bucket.
As per the above, the API Gateway, assumes the read privileges for the S3 bucket using the above role.
This template section:
"ApiGateway": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Name": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
" Api"
]
]
},
"Description": "A simple Read Model example using S3",
"FailOnWarnings": true
}
},
"SampleResource": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "ApiGateway"
},
"ParentId": {
"Fn::GetAtt": [
"ApiGateway",
"RootResourceId"
]
},
"PathPart": "samples"
}
},
"SamplesGetMethod": {
"Type": "AWS::ApiGateway::Method",
"Properties": {
"AuthorizationType": "NONE",
"HttpMethod": "GET",
"Integration": {
"Type": "AWS",
"IntegrationHttpMethod": "GET",
"IntegrationResponses": [
{
"StatusCode": "200",
"SelectionPattern": "200"
}
],
"Uri": {
"Fn::Join": [
"/",
[
"arn:aws:apigateway:eu-west-1:s3:path",
{
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store"
]
]
},
"payload.json"
]
]
},
"Credentials": {
"Fn::GetAtt": [
"ApiGatewayRole",
"Arn"
]
},
"PassthroughBehavior": "WHEN_NO_MATCH"
},
"ResourceId": {
"Ref": "SampleResource"
},
"RestApiId": {
"Ref": "ApiGateway"
},
"MethodResponses": [
{
"StatusCode": 200
}
]
}
},
Creates this API Gateway:
We make a deployment
The API Gateway on its own is a definition - you need a deployment before it is available for use.
This makes a deployment using the above API Gateway as "v1".
This template sections:
"ApiGatewayDeployment": {
"Type": "AWS::ApiGateway::Deployment",
"DependsOn": "SamplesGetMethod",
"Properties": {
"RestApiId": {
"Ref": "ApiGateway"
},
"Description": "Initial Deployment",
"StageName": "v1"
}
}
Creates this deployment:
And finally
We have our outputs so we can use the solution.
The bucket and access key/ secret allow us to upload the json payload.
The sample url then allows us to valid that the solution all works.
This template sction:
"Outputs": {
"SamplesUrl": {
"Description": "Url to access the Samples data",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "ApiGateway"
},
".execute-api.eu-west-1.amazonaws.com/v1/samples"
]
]
}
},
"S3BucketName": {
"Description": "Name of S3 bucket created",
"Value": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store"
]
]
}
},
"DataStoreAccessKeyId": {
"Description": "Access Key Id for the Data Store (S3)",
"Value": {
"Ref": "DataStoreWriterAccessKey"
}
},
"DataStoreSecretAccessKey": {
"Description": "Secret Access Key for the Data Store (S3)",
"Value": {
"Fn::GetAtt": [
"DataStoreWriterAccessKey",
"SecretAccessKey"
]
}
},
"DataStoreUser": {
"Description": "User to upload to the Data Store (S3)",
"Value": {
"Fn::Join": [
"",
[
{
"Ref": "StackName"
},
"-data-store-writer"
]
]
}
}
}
Creates these outputs:
Further reading
This solution hasn't been used in anger, so there are a few things that should probably be validated before production use:
Concurrency - I'm fairly certain that there shouldn't be any blocking on the read while new payloads are uploaded. You should continue to read the old version until the new version has been completed. Note also that S3 will replicate across multipule relicas - I would expect that it is possible to get different versions of the payload depending on the replica you hit.
ETag - I believe that S3 will take advantage of ETag - which should allow you to reduce network traffic if the payload hasn't changed. I've not tested if the API Gateway will honour the ETag mechanism - well worth trying.