Adapted from the Aws tutorial - Building a Modern app (module 5).
Click Event records will be processed in real-time by a serverless code function, aggregated, and stored for any future analysis that you may want to perform.
aws codecommit create-repository --repository-name MythicalMysfitsStreamingService-Repository
{
"repositoryMetadata": {
"accountId": "REPLACE_ME_ACCOUNT_ID",
"repositoryId": "32941d10-66e1-4230-b3d6-eeb55e64a60b",
"repositoryName": "MythicalMysfitsStreamingService-Repository",
"lastModifiedDate": 1554965738.429,
"creationDate": 1554965738.429,
"cloneUrlHttp": "https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository",
"cloneUrlSsh": "ssh://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository",
"Arn": "arn:aws:codecommit:eu-central-1:REPLACE_ME_ACCOUNT_ID:MythicalMysfitsStreamingService-Repository"
}
}
git clone https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository
Cloning into 'MythicalMysfitsStreamingService-Repository'...
warning: You appear to have cloned an empty repository.
cd MythicalMysfitsStreamingService-Repository
cp ..\aws-modern-application-workshop\module-5\cfn .
cp ..\aws-modern-application-workshop\module-5\app\streaming .
:: A CFN template for creating the full stack.
..\aws-modern-application-workshop\module-5\cfn\real-time-streaming.yml
1 file(s) copied.
:: A Python script that contains the code for our Lambda function: streamProcessor.py
..\aws-modern-application-workshop\module-5\app\streaming\streamProcessor.py
1 file(s) copied.
:: Install the request package into the local directory
pip install requests -t .
Collecting requests
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
100% | | 163kB 163kB/s
Collecting urllib3<1.25,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, certifi, urllib3, idna, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
ls -A1
.git
bin
certifi
certifi-2019.3.9.dist-info
chardet
chardet-3.0.4.dist-info
idna
idna-2.8.dist-info
real-time-streaming.yml
requests
requests-2.21.0.dist-info
streamProcessor.py
urllib3
urllib3-1.24.1.dist-info
def retrieveMysfit(mysfitId):
apiEndpoint = 'REPLACE_ME_API_ENDPOINT' + '/mysfits/' + str(
mysfitId) # eg: 'https://ljqomqjzbf.execute-api.us-east-1.amazonaws.com/prod/'
mysfit = requests.get(apiEndpoint).json()
return mysfit
git add .
git commit -m "New stream processing service."
git push
The creation of the streaming stack is done using the AWS SAM CLI to:
Create a bucket named real-time-streaming
aws s3 mb s3://real-time-streaming
With the Sam cli
cd aws-modern-application-workshop\module-5
sam package ^
--template-file .\cfn\real-time-streaming.yml ^
--output-template-file .\cfn\transformed-streaming.yml ^
--s3-bucket real-time-streaming
Uploading to 31ae6f83bb35657cc2aad3d168246ada 2629 / 2629.0 (100.00%)
Successfully packaged artifacts and wrote output template to file .\cfn\transformed-streaming.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file d:\code-external\aws-modern-application-workshop\module-5\cfn\transformed-streaming.yml --stack-name <YOUR STACK NAME>
.........
MysfitsClicksProcessor:
Type: AWS::Serverless::Function
Properties:
Handler: streamProcessor.processRecord
Runtime: python3.6
CodeUri: s3://real-time-streaming/31ae6f83bb35657cc2aad3d168246ada
..........
Returned by the SAM CLI command is the CloudFormation command needed to be executed to create our new full stack. But because our stack creates IAM resources, you'll need to add one additional parameter to the command.
aws cloudformation deploy ^
--template-file .\cfn\transformed-streaming.yml ^
--stack-name MythicalMysfitsStreamingStack ^
--capabilities CAPABILITY_IAM
Once this stack creation is complete, the full real-time processing microservice will be created.
An error occurred (AccessDenied) when calling the CreateChangeSet operation: User: arn:aws:iam::09000999845148:user/cli is not authorized to perform: cloudformation:CreateChangeSet on resource: arn:aws:cloudformation:eu-central-1:094473333148:stack/MythicalMysfitsStreamingStack/*