Amazon Simple Storage Service (AWS S3)

Amazon Simple Storage Service (AWS S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can use it to store and protect any amount of data.

Support Added: FortiSIEM 6.5.0

Vendor Version Tested: Not Provided

Vendor: Amazon

Product: AWS S3

Product Information: https://aws.amazon.com/s3/

• Configuration
• Sample Events

Configuration

• Setup in AWS S3

• Setup in FortiSIEM

Setup in AWS S3

Complete these steps from AWS S3.

Generate a New Access Key

1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/

2. In the navigation pane, choose Users.

3. Click Create Users.

4. In the User name field, enter a user name.

5. For AWS access type, select Programmatic access - with an access key.

6. Click Next: Permissions.

7. Select the Attach existing policies directly tab.

8. Select AmazonS3ReadOnlyAccess and AmazonSQSFullAccess.

9. Click Next: Tag.

10. Click Next: review.

11. Click Create user.

12. Click Download Credentials.

13. Click on the Close button.

    The downloaded CSV file contains the Access Key ID (User Name) and Secret Access Key (Password) that will be used in the FortiSIEM configuration.

Enable Event Notifications

1. Open the Amazon S3 console at https://s3.console.aws.amazon.com/s3/

2. Select your bucket.

3. Click Properties.

4. Click Event notifications> Create event notification.

   1. In the Event name field, enter a event name.

   2. In the Prefix field, enter a prefix.
      

      Note: Leave this field blank to generate an event for all objects created in an S3 bucket. If you only want a specific subfolder ( terminology defined as prefix in S3 bucket) to generate events, then you can define it.

      Example: A prefix for /logs will only generate events when files are added under this prefix of the S3 bucket, but logs on the root prefix ‘/’ will not generate events.

      /logs/vpc_logs – Would generate an event when file created.

      /logs/my_custom_logs – Would generate an event when file created.

      /log1.gz – Would NOT generate an event when file created.

      /cloudtraillog.json.gz – Would NOT generate an event when file created.

   3. Under Event types, select All object create events.

   4. For Destination, select SQS queue.

   5. Select your SQS.

   6. Click Save changes.
      

      For your configuration, make sure of the following.

      • Ensure that there are no other servers that use this SQS.

      • Make sure the property Message retention period of SQS is set to 1 Days.

      • Make sure the property Default visibility timeout of SQS is set to 12 Hours.

      
      

Setup in FortiSIEM

Start a Pulling Job by taking the following steps.

1. Login to FortiSIEM.

2. Navigate to ADMIN > Device Support > Devices/Apps.

3. Click New to create a new device type.

4. In the Category drop-down list, select Device.

   In the Vendor field, enter the vendor name, e.g. "Amazon".

5. In the Model field, enter the device model, e.g. "AWS S3".

6. In the Version field, enter the version device.

7. In the Device/App Group drop-down list, expand Devices, and select a value, e.g. "Generic".

8. From the Access Protocol drop-down list, select AWS_S3_WITH_SQS.

9. Click Save.

Create a new Access Method Credential by taking the following steps.

1. Go to the ADMIN > Setup > Credentials tab.
2. In Step 1: Enter Credentials, click New to create a new credential.
   1. Follow the instructions in "Setting Credentials" in the User's Guide to create a new credential.
   2. Enter these settings in the Access Method Definition dialog box and click Save:

      Settings	Description
      Name	Enter a name for the credential
      Device Type	Select the device type you created earlier.
      Access Protocol	AWS_S3_WITH_SQS
      Bucket	Enter your AWS bucket.
      SQS Queue URL	Enter your SQS queue URL. You must enter the entire URL, for example: https://sqs.us-west-2.amazonaws.com/111111111111/sqsforloadblancer.
      User Name	Enter the username for your AWS S3 account. This will be the Access Key ID when you generated your access key. Note: Make sure no other devices uses the credential. Otherwise, events may be missed in this device.
      Password	Enter the password associated with your username. This will be the Secret Access Key when you generated your access key.
      Session Token	This can be left blank. It is not required for S3 integration.
      Log Keyword	By default, a log keyword is provided. Feel free to change this and enter a keyword that can be easily matched by a custom parser. This will simplify the creation of a parser for the logs collected from the AWS S3 bucket.
      Description	Description of the device.

3. In Step 2: Enter IP Range to Credential Associations, click New to create a mapping.
   1. Enter a host name, an IP, or an IP range in the IP/Host Name field.
   2. Select the name of your credential from the Credentials drop-down list.
   3. Click Save.
4. Click the Test drop-down list and select Test Connectivity to test the connection to the server.
5. Navigate to ADMIN > Setup > Pull Events to see the new job.
   
   Events can be queried from the ANALYTICS page.

Forwarding Logs to S3

Customers can forward any services logs to S3 if those services allow it to do so. Examples include AWS Elastic Load Balancing, Cisco Umbrella. Customer also can upload any logs file to S3 manually.

Forward ELB Access Logs to AWS S3

Please refer to “Setup in AWS” at https://docs.fortinet.com/document/fortisiem/6.6.5/external-systems-configuration-guide/470868/aws-elastic-load-balancer#AWS

Forward Cisco Umbrella Log to AWS S3

Please refer to https://docs.umbrella.com/deployment-umbrella/docs/setting-up-an-amazon-s3-bucket.

Upload Log File to AWS S3

Note: The file format currently can be any extension. (e.g. no extension, .txt, .log, .json, .log.gz, etc). The filetype is checked if it is zipped using gzip (regardless of extension .gz). If it is zipped using gzip, FortiSIEM will unzip the log.

Logs are processed as 1 line per log in the file. The entirety of a single log entry must exist on one line. Multiline JSON (JSON log entries spanning multiple lines in the log file) logs will be supported in an upcoming release.

e.g.

file1.txt contents:

{“Name” : “log1”}

{“Name” : “log2”}

{“Name” : “log3”}

Log4 text

Sample Logs

Note that in Credential Definition, you can change the text (Log Keyword = AWS_S3_LOG_KEYWORD_1) e.g. VPCFLOWLOGS if S3 bucket contains VPC flow logs. This is important for parsing the logs later in FortiSIEM.

AWS_S3_LOG_KEYWORD_1:phCustId=1,reptDevIpAddr=192.0.2.20,reptDevName=amazon.com,msg={“Name” : “log1”}

AWS_S3_LOG_KEYWORD_1:phCustId=1,reptDevIpAddr=192.0.2.20,reptDevName=amazon.com,msg={“Name” : “log2”}

AWS_S3_LOG_KEYWORD_1:phCustId=1,reptDevIpAddr=192.0.2.20,reptDevName=amazon.com,msg={“Name” : “log3”}

AWS_S3_LOG_KEYWORD_1:phCustId=1,reptDevIpAddr=192.0.2.20,reptDevName=amazon.com,msg=Log4 text

Take the following steps.

1. Open the Amazon S3 console at https://s3.console.aws.amazon.com/s3/.

2. Select your bucket.

3. Click Upload and then click Add files.

4. Select log files and click Upload.
   Notes: FortiSIEM only supports:

   • gz compression file, and uncompressed files (e.g. no extension, .txt, .log, .json, etc...)

   • “Single event per line” in log file

Sample Events

Logs from AWS ELB

AWS_S3_LOG_KEYWORD_1:phCustId=1,reptDevIpAddr=192.0.2.20,reptDevName=amazon.com,msg=http 2021-02-11T01:56:06.000372Z app/shashi-elb/061d492a88a60fb1 192.0.2.108:46938 - -1 -1 -1 503 - 500 337 "POST http://192.0.2.144:80/form/admin/formLogin HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" - - arn:aws:elasticloadbalancing:us-west-2:623885071509:targetgroup/shashi-tg/974fbb8764192573 "Root=1-60248eb5-01950dcf187ac3c244ab2231" "-" "-" 0 2021-02-11T01:56:05.999000Z "forward" "-" "-" "-" "-" "-" "-"

Logs from AWS Cloudtrail

AWS_S3_LOG_KEYWORD_1:phCustId=1,reptDevIpAddr=10.10.103.205,reptDevName=amazon.com,msg= {"Records":[{"eventVersion":"1.05","userIdentity":{"type":"IAMUser","principalId":"AIDAZCQTSWSKVN6P266F6","arn":"arn:aws:iam::623885071509:user/dusan","accountId":"623885071509","accessKeyId":"ASIAZCQTSWSKZ5NL34OB","userName":"dusan","sessionContext":{"sessionIssuer":{},"webIdFederationData":{},"attributes":{"mfaAuthenticated":"false","creationDate":"2019-09-05T09:05:25Z"}},"invokedBy":"signin.amazonaws.com"},"eventTime":"2019-09-05T11:16:21Z","eventSource":"lambda.amazonaws.com","eventName":"ListFunctions20150331","awsRegion":"us-west-1","sourceIPAddress":"85.241.114.212","userAgent":"signin.amazonaws.com","requestParameters":null,"responseElements":null,"requestID":"26057495-d109-4366-9961-77955807e508","eventID":"321c137a-8999-48a3-89f6-7168e4107ecb","eventType":"AwsApiCall","recipientAccountId":"623885071509"}]}