About
Adapted from the Aws tutorial - Building a Modern app (module 5).
Articles Related
Architecture
Click Event records will be processed in real-time by a serverless code function, aggregated, and stored for any future analysis that you may want to perform.
Steps
Create the code and save it to a repository
aws codecommit create-repository --repository-name MythicalMysfitsStreamingService-Repository
{
"repositoryMetadata": {
"accountId": "REPLACE_ME_ACCOUNT_ID",
"repositoryId": "32941d10-66e1-4230-b3d6-eeb55e64a60b",
"repositoryName": "MythicalMysfitsStreamingService-Repository",
"lastModifiedDate": 1554965738.429,
"creationDate": 1554965738.429,
"cloneUrlHttp": "https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository",
"cloneUrlSsh": "ssh://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository",
"Arn": "arn:aws:codecommit:eu-central-1:REPLACE_ME_ACCOUNT_ID:MythicalMysfitsStreamingService-Repository"
}
}
- Clone
git clone https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository
Cloning into 'MythicalMysfitsStreamingService-Repository'...
warning: You appear to have cloned an empty repository.
- Copy the source
cd MythicalMysfitsStreamingService-Repository
cp ..\aws-modern-application-workshop\module-5\cfn .
cp ..\aws-modern-application-workshop\module-5\app\streaming .
:: A CFN template for creating the full stack.
..\aws-modern-application-workshop\module-5\cfn\real-time-streaming.yml
1 file(s) copied.
:: A Python script that contains the code for our Lambda function: streamProcessor.py
..\aws-modern-application-workshop\module-5\app\streaming\streamProcessor.py
1 file(s) copied.
- package all of library dependencies together with the Lambda code function
:: Install the request package into the local directory
pip install requests -t .
Collecting requests
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
100% | | 163kB 163kB/s
Collecting urllib3<1.25,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, certifi, urllib3, idna, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
- List
ls -A1
.git
bin
certifi
certifi-2019.3.9.dist-info
chardet
chardet-3.0.4.dist-info
idna
idna-2.8.dist-info
real-time-streaming.yml
requests
requests-2.21.0.dist-info
streamProcessor.py
urllib3
urllib3-1.24.1.dist-info
- Replace REPLACE_ME_API_ENDPOINT in the Lambda Function Code streamProcessor.py by the ApiEndpoint of the Mysfits service API (this is the same service ApiEndpoint that you created for the website frontend).
def retrieveMysfit(mysfitId):
apiEndpoint = 'REPLACE_ME_API_ENDPOINT' + '/mysfits/' + str(
mysfitId) # eg: 'https://ljqomqjzbf.execute-api.us-east-1.amazonaws.com/prod/'
mysfit = requests.get(apiEndpoint).json()
return mysfit
- Package
git add .
git commit -m "New stream processing service."
git push
Create the streaming service stack with SAM
The creation of the streaming stack is done using the AWS SAM CLI to:
- package all of our function code,
- upload it to S3,
- and create the deployable CloudFormation template
Create a S3 bucket
Create a bucket named real-time-streaming
aws s3 mb s3://real-time-streaming
Package the code with SAM
With the Sam cli
cd aws-modern-application-workshop\module-5
sam package ^
--template-file .\cfn\real-time-streaming.yml ^
--output-template-file .\cfn\transformed-streaming.yml ^
--s3-bucket real-time-streaming
Uploading to 31ae6f83bb35657cc2aad3d168246ada 2629 / 2629.0 (100.00%)
Successfully packaged artifacts and wrote output template to file .\cfn\transformed-streaming.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file d:\code-external\aws-modern-application-workshop\module-5\cfn\transformed-streaming.yml --stack-name <YOUR STACK NAME>
- The CodeUri parameter of the serverless Lambda function has been updated in the transformed-streaming.yml file with the object location where the SAM CLI has uploaded your packaged code
.........
MysfitsClicksProcessor:
Type: AWS::Serverless::Function
Properties:
Handler: streamProcessor.processRecord
Runtime: python3.6
CodeUri: s3://real-time-streaming/31ae6f83bb35657cc2aad3d168246ada
..........
Deploy The Stack Using AWS CloudFormation
Returned by the SAM CLI command is the CloudFormation command needed to be executed to create our new full stack. But because our stack creates IAM resources, you'll need to add one additional parameter to the command.
- deploy the streaming stack (deploy the function)
aws cloudformation deploy ^
--template-file .\cfn\transformed-streaming.yml ^
--stack-name MythicalMysfitsStreamingStack ^
--capabilities CAPABILITY_IAM
Once this stack creation is complete, the full real-time processing microservice will be created.
Documentation / Reference
Support
AccessDenied when calling the CreateChangeSet operation
An error occurred (AccessDenied) when calling the CreateChangeSet operation: User: arn:aws:iam::09000999845148:user/cli is not authorized to perform: cloudformation:CreateChangeSet on resource: arn:aws:cloudformation:eu-central-1:094473333148:stack/MythicalMysfitsStreamingStack/*