AWS Kinesis

Amazon Kinesis is an Amazon Web Service (AWS) for processing big data in real time. Kinesis is capable of processing hundreds of terabytes per hour from high volumes of streaming data from sources such as operating logs, financial transactions and social media feeds.

• What is Discovered and Monitored
• Event Types
• Rules
• Reports
• Configuring AWS Kinesis
• Configuring FortiSIEM
• Configuring AWS CloudTrail Logs through Kinesis Streams
• Configuring VPC Flow Logs through Kinesis Streams
• Sample Events

What is Discovered and Monitored

Protocol	Information collected	Used for
Amazon AWS Client Library	Streaming data	Collect, process, and analyze real-time streaming data.

Event Types

In RESOURCES > Event Types, enter "Kinesis" in the main content panel Search... field to see the event types associated with this device.

Rules

No defined rules.

Reports

No defined reports.

Configuring AWS Kinesis

1. Use your AWS account ID or account alias, your IAM user name, and your password to sign in to the IAM console.
2. In the navigation bar on the upper right, choose your user name, and then choose My Security Credentials.
3. On the AWS IAM Credentials tab, in the Access keys for CLI, SDK, and API access section, do any of the following:
   • To create an access key, choose Create access key. Then choose Download .csv file to save the access key ID and secret access key to a .csv file on your computer. Store the file in a secure location. You will not have access to the secret access key again after this dialog box closes. After you have downloaded the .csv file, choose Close. When you create an access key, the key pair is active by default, and you can use the pair right away.
   • To disable an active access key, choose Make inactive.
   • To reenable an inactive access key, choose Make active.
   • To delete an access key, choose its X button at the far right of the row. Then choose Delete to confirm. When you delete an access key, it's gone forever and cannot be retrieved. However, you can always create new keys.

Configuring FortiSIEM

1. Go to the ADMIN > Setup > Credentials tab.
2. In Step 1: Enter Credentials:
   1. Follow the instructions in “Setting Credentials“ in the User's Guide to create a new credential.
   2. Enter these settings in the Access Method Definition dialog box and click Save:

      Settings	Description
      Name	Enter a name for the credential
      Device Type	Amazon AWS Kinesis
      Access Protocol	AWS Kinesis Client Library
      Region	You can enter one or more regions separated by a space, for example, “us-east-1 us-west-2”. See Supported Regions in AWS for a list of valid regions.
      Password Config	Choose Manual, CyberArk SDK, CyberArk REST API, or RAX_Janus from the drop down list. For CyberArk SDK, see CyberArk SDK Password Configuration. For CyberArk REST API, see CyberArk REST API Password Configuration. For RAX_Janus, see RAX_Janus Password Configuration.
      Access Key	Access key for your AWS Kinesis instance. See Configuring AWS Kinesis.
      Secret Key	Secret key for your AWS Security Hub instance
      Organization	The organization the device belongs to.
      Description	Description of the device.

3. In Step 2: Enter IP Range to Credential Associations, click New.
   1. Enter a host name, an IP, or an IP range in the IP/Host Name field.
   2. Select the name of your AWS Kinesis credential from the Credentials drop-down list.
   3. Click Save.
4. Click the Test drop-down list and select Test Connectivity to test the connection to AWS Kinesis.
5. To see the jobs associated with AWS Kinesis, select ADMIN > Setup > Pull Events.
6. To see the received events select ANALYTICS, then enter "AWS Kinesis" in the search box.

Configuring AWS CloudTrail Logs through Kinesis Streams

The data flow for this setup is AWS CloudTrail -> AWS CloudWatch -> AWS Kinesis Data Streams.

Taking the following steps to configure.

Prerequisite

• Obtain your access key and secret for a user account with administrator privileges.

• Ensure the AWS CLI is installed. Fore more information on AWS CLI, see here.

1. Run from cmd:
   aws configure

   Enter your access key, secret and default region.

   Example:

   Account ID: 11111111111

   Access Key ID: aaaaaaaaaaaaaaaaaaaaa

   Access Secret: 13452322222222222222

   Region: us-west-1

   Kinesis Role: SIEMKinesisRole

   Note: This will be created in a later portion of this guide.

2. Create log group by entering the following:

   aws logs create-log-group --log-group-name "cloudwatch-group"

   aws logs describe-log-groups

3. Create a cloudtrail role that permits sending logs to the CloudWatch log group by putting the following JSON into a file, and save it as a JSON file on your desktop.
   

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "Service": "cloudtrail.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

4. Use the AWS CLI to run the following command, referencing the file path where you saved the JSON file. In this command example, the file above was saved as cloudtrail-role.json.
   

   aws iam create-role --role-name CTRole --assume-role-policy-document file://cloudtrail-role.json
   

5. Create the CloudTrail policy-document by saving the following JSON into a file, saving it to your desktop.
   

   {
     "Version": "2012-10-17",
     "Statement": [
       {
   
         "Sid": "AWSCloudTrailCreateLogStream2014110",
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogStream"
         ],
         "Resource": [
           "arn:aws:logs:us-west-1:11111111111:log-group:CloudTrail:log-stream:11111111111_CloudTrail_us-west-1*"
         ]
   
       },
       {
         "Sid": "AWSCloudTrailPutLogEvents20141101",
         "Effect": "Allow",
         "Action": [
           "logs:PutLogEvents"
         ],
         "Resource": [
           "arn:aws:logs:us-west-1:11111111111:log-group:CloudTrail:log-stream:11111111111_CloudTrail_us-west-1*"
         ]
       }
     ]
   }
   

6. Use the AWS CLI to run the following command, referencing the file path where you saved the JSON file. In this command example, the file above was saved as cloudtrail-policy.json. Note: What we're doing here is creating a role called “CTRole” and applying a policy to it.

   aws iam put-role-policy --role-name CTRole --policy-name cloudtrail-policy --policy-document file://cloudtrail-policy.json
   

Kinesis Configuration Setup

Take the following steps to configure Kinesis. Note: The latest instructions can be found here.

1. Create an AWS Kinesis stream (if it does not exist). In this example, we'll create one called "fortisiem".
   

   aws kinesis create-stream --stream-name "fortisiem"

2. Create a file called "TrustPolicyForCWL.json" using the following code.
   

   {
     "Statement": {
       "Effect": "Allow",
       "Principal": { "Service": "logs.us-west-1.amazonaws.com" },
       "Action": "sts:AssumeRole"
     }
   }
   

3. Create an iam role with the defined policy, called "CWLtoKinesisRole" by running the following command.
   

   aws iam create-role --role-name CWLtoKinesisRole --assume-role-policy-document file://~/TrustPolicyForCWL.json
   

4. Create a permission file called "PermissionsForCWL.json" using the following code.
   

   {
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "kinesis:PutRecord",
         "Resource": "arn:aws:kinesis:us-west-1: 11111111111:stream/fortisiem"
       }
     ]
   }
   

5. Associate the above policy file with the role we created, by running the following command.
   

   aws iam put-role-policy --role-name CWLtoKinesisRole --policy-name Permissions-Policy-For-CWL --policy-document
   

6. Create the subscription to forward cloudwatch logs.

   Note: The following multiline command is for Windows, as ^ indicates the command continues on the next line. If using Linux, use the multiline command for Linux, which replaces the ^ with \. Paste all of it in as one command.

   Windows:

   //Create  CloudWatch Log Susbscription filter:
   aws logs put-subscription-filter ^
       --log-group-name "cloudwatch-group" ^
       --filter-name "fortisiem" ^
       --filter-pattern "" ^
       --destination-arn "arn:aws:kinesis:us-west-1:11111111111:stream/fortisiem " ^
       --role-arn "arn:aws:iam::11111111111:role/CWLtoKinesisRole "

   Linux:
   

   //Create  CloudWatch Log Susbscription filter:
   aws logs put-subscription-filter \
       --log-group-name "cloudwatch-group" \
       --filter-name "fortisiem" \
       --filter-pattern "" \
       --destination-arn "arn:aws:kinesis:us-west-1:11111111111:stream/fortisiem " \
       --role-arn "arn:aws:iam::11111111111:role/CWLtoKinesisRole "
   

7. Create an IAM user account that FortiSIEM can use to ingest Kinesis Data Streams. When assigning permissions for the IAM user, the following must be allowed.
   

   Consumer
   Actions	Resource	Purpose
   DescribeStream	Kinesis data stream	Before attempting to read records, the consumer checks if the stream exists and is active, and if the shards are contained in the stream.
   Get Records, Get ShardIterator	Kinesis data stream	Read records from a Kinesis Data Streams shard.
   CreateTable, DescribeTable, GetItem, PutItem, Scan, Update Item	Amazon DynamoDB table	If the consumer is developed using the Kinesis Client Library (KCL), it needs permissions to a DynamoDB table to track the processing state of the application. The first consumer started creates the table.
   Delete Item	Amazon DynamoDB table	For when the consumder performs split/merge operations on Kinesis Data Stream shards.
   PutMetricData	Amazon CloudWatch log	The KCL also uploads metrics to CloudWatch, which are useful for monitoring the application.

   
   

8. Navigate to IAM -> Users -> Add user
   Access Type: Programmatic Access
   Next Permission:

9. Under permissions, select Attach existing policies directly
   AmazonKinesisReadOnlyAccess
   AmazonDynamoDBFullAccess
   CloudWatchLogsFullAccess
   Note: If step 11 is followed, it replaces the more generic access policy set here in step 9, which would no longer be needed.
   

10. Click Next -> Review.
    You should see that under Permissions summary, the following policies attached to the user.
    

    Type	Name
    Managed policy	CloudWatchLogsFullAccess
    Managed policy	AmazonDynamoDBFullAccess
    Managed policy	AmazonKinesisReadOnlyAccess

11. Once the user is created, generate an access key and secret for this user. The following reference link demonstrates how to implement restrictive, minimum access policies to FortiSIEM (which is the consumer as referenced in the guide). - https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html
    Note: For advanced users, you can customize permissions policies to allow only access to a specific CloudWatch log group, specific Kinesis Stream, etc....
    
    

FortiSIEM Configuration Setup

Go to the ADMIN > Setup > Credentials tab.

In Step 1: Enter Credentials:

1. Follow the instructions in “Setting Credentials“ in the User's Guide to create a new credential.
2. Enter these settings in the Access Method Definition dialog box and click Save:

   Settings	Description
   Name	Enter a name for the credential, such as KinesisStreamIntegration
   Device Type	Amazon AWS Kinesis
   Access Protocol	AWS Kinesis Client Library
   Region	You can enter one or more regions separated by a space, for example, “us-east-2 us-west-1”. See Supported Regions in AWS for a list of valid regions.
   Log Stream Name	Enter "fortisiem" or the name of your stream.
   Password Config	Choose Manual, CyberArk SDK, CyberArk REST API, or RAX_Janus from the drop down list. For CyberArk SDK, see CyberArk SDK Password Configuration. For CyberArk REST API, see CyberArk REST API Password Configuration. For RAX_Janus, see RAX_Janus Password Configuration.
   Access Key	Access key for your AWS Kinesis instance. See Configuring AWS Kinesis.
   Secret Key	Your Secret key.
   Organization	The organization the device belongs to.
   Description	Description of the device.

3. In Step 2: Enter IP Range to Credential Associations, to the right of the Search... field, select the desired collector that will poll CloudTrail logs from the drop-down list if you have more than one collector. If only one collector is available, the drop-down list will not be available.
   
4. Click New.
   1. Select the CloudTrail credential you created earlier from the Credentials drop-down list. It should autofill IP/Host Name as destination "amazon.com". FortiSIEM will handle communication to the appropriate API.
   2. Click Save.
5. Select the new mapping and click the Test drop-down list and select Test Connectivity without Ping to start the polling.
6. To see the jobs associated with AWS Kinesis, select ADMIN > Setup > Pull Events.
7. To see the received events select ANALYTICS, then enter "AWS Kinesis" in the search box.

Configuring VPC Flow Logs through Kinesis Streams

The data flow for this setup is AWS VPC Flow Logs -> AWS CloudWatch -> AWS Kinesis Data Streams.

Taking the following steps to configure.

Prerequisite

• Obtain your access key and secret for a user account with administrator privileges.

• Ensure the AWS CLI is installed. Fore more information on AWS CLI, see here.

1. Run from cmd:
   aws configure

   Enter your access key, secret and default region.

   Example:

   Account ID: 11111111111

   Access Key ID: aaaaaaaaaaaaaaaaaaaaa

   Access Secret: 13452322222222222222

   Region: us-west-1

2. Configure AWS VPC flow logs to send to CloudWatch logs by following similar steps documented here.

Kinesis Configuration Setup

Take the following steps to configure Kinesis. Note: The latest instructions can be found here.

1. Create an AWS Kinesis stream (if it does not exist). In this example, we'll create one called "fortisiem".
   

   aws kinesis create-stream --stream-name "fortisiem"

2. Create a file called "TrustPolicyForCWL.json" using the following code.
   

   {
     "Statement": {
       "Effect": "Allow",
       "Principal": { "Service": "logs.us-west-1.amazonaws.com" },
       "Action": "sts:AssumeRole"
     }
   }
   

3. Create an iam role with the defined policy, called "CWLtoKinesisRole" by running the following command.
   

   aws iam create-role --role-name CWLtoKinesisRole --assume-role-policy-document file://~/TrustPolicyForCWL.json
   

4. Create a permission file called "PermissionsForCWL.json" using the following code.
   

   {
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "kinesis:PutRecord",
         "Resource": "arn:aws:kinesis:us-west-1: 11111111111:stream/fortisiem"
       }
     ]
   }
   

5. Associate the above policy file with the role we created, by running the following command.
   

   aws iam put-role-policy --role-name CWLtoKinesisRole --policy-name Permissions-Policy-For-CWL --policy-document
   

6. Create the subscription to forward cloudwatch logs.

   Note: The following multiline command is for Windows, as ^ indicates the command continues on the next line. If using Linux, use the multiline command for Linux, which replaces the ^ with \. Paste all of it in as one command.

   Windows:

   //Create  CloudWatch Log Susbscription filter:
   aws logs put-subscription-filter ^
       --log-group-name "cloudwatch-group" ^
       --filter-name "fortisiem" ^
       --filter-pattern "" ^
       --destination-arn "arn:aws:kinesis:us-west-1:11111111111:stream/fortisiem " ^
       --role-arn "arn:aws:iam::11111111111:role/CWLtoKinesisRole "

   Linux:
   

   //Create  CloudWatch Log Susbscription filter:
   aws logs put-subscription-filter \
       --log-group-name "cloudwatch-group" \
       --filter-name "fortisiem" \
       --filter-pattern "" \
       --destination-arn "arn:aws:kinesis:us-west-1:11111111111:stream/fortisiem " \
       --role-arn "arn:aws:iam::11111111111:role/CWLtoKinesisRole "
   

7. Create an IAM user account that FortiSIEM can use to ingest Kinesis Data Streams. When assigning permissions for the IAM user, the following must be allowed.
   

   Consumer
   Actions	Resource	Purpose
   DescribeStream	Kinesis data stream	Before attempting to read records, the consumer checks if the stream exists and is active, and if the shards are contained in the stream.
   Get Records, Get ShardIterator	Kinesis data stream	Read records from a Kinesis Data Streams shard.
   CreateTable, DescribeTable, GetItem, PutItem, Scan, Update Item	Amazon DynamoDB table	If the consumer is developed using the Kinesis Client Library (KCL), it needs permissions to a DynamoDB table to track the processing state of the application. The first consumer started creates the table.
   Delete Item	Amazon DynamoDB table	For when the consumder performs split/merge operations on Kinesis Data Stream shards.
   PutMetricData	Amazon CloudWatch log	The KCL also uploads metrics to CloudWatch, which are useful for monitoring the application.

8. Navigate to IAM -> Users -> Add user
   Access Type: Programmatic Access
   Next Permission:

9. Under permissions, select Attach existing policies directly
   AmazonKinesisReadOnlyAccess
   AmazonDynamoDBFullAccess
   CloudWatchLogsFullAccess
   Note: If step 11 is followed, it replaces the more generic access policy set here in step 9, which would no longer be needed.
   

10. Click Next -> Review.
    You should see that under Permissions summary, the following policies attached to the user.
    

    Type	Name
    Managed policy	CloudWatchLogsFullAccess
    Managed policy	AmazonDynamoDBFullAccess
    Managed policy	AmazonKinesisReadOnlyAccess

11. Once the user is created, generate an access key and secret for this user. The following reference link demonstrates how to implement restrictive, minimum access policies to FortiSIEM (which is the consumer as referenced in the guide). - https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html
    Note: For advanced users, you can customize permissions policies to allow only access to a specific CloudWatch log group, specific Kinesis Stream, etc....
    

FortiSIEM Configuration Setup

1. Go to the ADMIN > Setup > Credentials tab.

2. In Step 1: Enter Credentials:

   1. Follow the instructions in “Setting Credentials“ in the User's Guide to create a new credential.

   2. Enter these settings in the Access Method Definition dialog box and click Save:
      

      Settings	Description
      Name	Enter a name for the credential, such as KinesisStreamIntegration
      Device Type	Amazon AWS Kinesis
      Access Protocol	AWS Kinesis Client Library
      Region	You can enter one or more regions separated by a space, for example, “us-east-2 us-west-1”. See Supported Regions in AWS for a list of valid regions.
      Log Stream Name	Enter "fortisiem" or the name of your stream.
      Password Config	Choose Manual, CyberArk SDK, CyberArk REST API, or RAX_Janus from the drop down list. For CyberArk SDK, see CyberArk SDK Password Configuration. For CyberArk REST API, see CyberArk REST API Password Configuration. For RAX_Janus, see RAX_Janus Password Configuration.
      Access Key	Access key for your AWS Kinesis instance. See Configuring AWS Kinesis.
      Secret Key	Your Secret key.
      Organization	The organization the device belongs to.
      Description	Description of the device.

3. In Step 2: Enter IP Range to Credential Associations, to the right of the Search... field, select the desired collector that will poll CloudTrail logs from the drop-down list if you have more than one collector. If only one collector is available, the drop-down list will not be available.

4. Click New.

   1. Select the CloudTrail credential you created earlier from the Credentials drop-down list. It should autofill the IP/Host Name field as destination "amazon.com". FortiSIEM will handle communication to the appropriate API.

   2. Click Save.

5. Click the Test drop-down list and select Test Connectivity without Ping to start the polling.

6. To see the jobs associated with AWS Kinesis, select ADMIN > Setup > Pull Events.

7. To see the received events select ANALYTICS, then enter "AWS Kinesis" in the search box.

Sample Events

VPC Log Example

2021-04-27T08:40:50.00Z [FSM-AWSKinesis] 2 311111777194 eni-054c755e644dcf32d 103.114.104.68 172.31.9.216 22898 22717 6 1 40 1619538050 1619538102 REJECT OK

2021-04-27T09:14:45.00Z [FSM-AWSKinesis] 2 311111777194 eni-054c755e644dcf32d 172.31.9.216 222.187.239.109 22 43638 6 3 3996 1619540085 1619540142 ACCEPT OK

CloudTrail Log Example

2021-04-28T14:29:51.783Z [FSM-AWSKinesis] {"eventVersion":"1.08","userIdentity":{"type":"Root","principalId":"921045424829","arn":"arn:aws:iam::1234567890AB:root","accountId":"111115424829","accessKeyId":"ASIA1234567890ABCDEF","sessionContext":{"sessionIssuer":{},"webIdFederationData":{},"attributes":{"mfaAuthenticated":"false","creationDate":"2021-04-28T20:37:47Z"}}},"eventTime":"2021-04-28T21:13:07Z","eventSource":"health.amazonaws.com","eventName":"DescribeEventAggregates","awsRegion":"us-east-1","sourceIPAddress":"10.20.30.40","userAgent":"console.amazonaws.com","requestParameters":{"aggregateField":"eventTypeCategory","filter":{"eventStatusCodes":["open","upcoming"],"startTimes":[{"from":"Apr 21, 2021 9:13:07 PM"}]}},"responseElements":null,"requestID":"1a712381-62d1-4485-8ce5-109930945c62","eventID":"eb8c077f-321d-471a-b207-849f5089a428","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"eventCategory":"Management","recipientAccountId":"921045424829"}