Aws - Kinesis Data Firehose Delivery Stream

Stream management

• auto-scaling (automatically provisions, manages and scales compute, memory, and network resources required)
• sharding,
• monitoring (metrics through the console)
• data can be loaded
  • by batch (interval)
  • compressed,
  • encrypted
• Pay-as-you-go pricing

Data Structure

A Request mapping template can be created in API Gateway to define the request payload structure that will restrict requests to an expected structure and then transform those well-formed requests into the structure that the Kinesis Firehose PutRecord API requires.

IAM Role

• Kinesis Firehose requires a service role that allows it to deliver received records as events to:
  • the created Lambda function
  • as well as the processed records to the destination S3 bucket.
• The Amazon API Gateway API also requires a new role that permits the API to invoke the PutRecord API within Kinesis Firehose for each received API request.

Firehose uses an IAM role to access your specified resources, such as the S3 bucket and KMS key.

Doc:

• https://docs.aws.amazon.com/console/firehose/iam-role

document policy where

• REPLACE_ME_ACCOUNT_ID is the account id
• REPLACE_ME_STREAM_NAME is the firehose stream name
• REPLACE_ME_BUCKET_NAME is the s3 destination bucket name

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "glue:GetTableVersions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::REPLACE_ME_FIREHOSE_BUCKET_NAME",
        "arn:aws:s3:::REPLACE_ME_FIREHOSE_BUCKET_NAME/*",
        "arn:aws:s3:::%FIREHOSE_BUCKET_NAME%",
        "arn:aws:s3:::%FIREHOSE_BUCKET_NAME%/*"
      ]
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "lambda:InvokeFunction",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:eu-central-1:REPLACE_ME_ACCOUNT_ID:function:%FIREHOSE_DEFAULT_FUNCTION%:%FIREHOSE_DEFAULT_VERSION%"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:eu-central-1:REPLACE_ME_ACCOUNT_ID:log-group:/aws/kinesisfirehose/REPLACE_ME_FIREHOSE_STREAM_NAME:log-stream:*"
      ]
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": "arn:aws:kinesis:eu-central-1:REPLACE_ME_ACCOUNT_ID:stream/%FIREHOSE_STREAM_NAME%"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:kms:eu-central-1:REPLACE_ME_ACCOUNT_ID:key/%SSE_KEY_ID%"
      ],
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "kinesis.%REGION_NAME%.amazonaws.com"
        },
        "StringLike": {
          "kms:EncryptionContext:aws:kinesis:arn": "arn:aws:kinesis:%REGION_NAME%:REPLACE_ME_ACCOUNT_ID:stream/%FIREHOSE_STREAM_NAME%"
        }
      }
    }
  ]
}

Record

A record can be as large as 1,000 KB.

Buffer size and buffer interval

Kinesis Data Firehose buffers incoming streaming data to a certain size or for a certain period of time before delivering it to destinations. Buffer Size is in MBs and Buffer Interval is in seconds.

Name

Acceptable characters are uppercase and lowercase letters, numbers, underscores, hyphens, and periods.

Update

Interface:

• Kinesis Data Firehose console
• API UpdateDestination.

Your Kinesis Data Firehose delivery stream remains in the ACTIVE state while your configuration is updated, and you can continue to send data. The updated configuration normally takes effect within a few minutes. The version number of a Kinesis Data Firehose delivery stream is increased by a value of 1 after you update the configuration. It is reflected in the delivered Amazon S3 object name

Version

The version number of a Kinesis Data Firehose delivery stream is increased by a value of 1 after you update the configuration. It is reflected in the delivered Amazon S3 object_name.

Object name

https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-name

Error logging

Firehose can log record delivery errors to CloudWatch Logs. If enabled, a CloudWatch log group and corresponding log streams are created on your behalf. https://docs.aws.amazon.com/console/firehose/monitor-with-cloudwatch