OnSight-based CounterMeasures

OnSight-based CounterMeasures allows CounterMeasures to be run on instances that cannot use the FortiMonitor Agent such as network devices and PaaS cloud services. OnSight-based CounterMeasures actions are executed from the OnSight and have the ability to target any number of remote targets corresponding to instances monitored from that OnSight.

Use cases

Outlined below are a few scenarios where OnSight-based CounterMeasures can be useful.

• Run CounterMeasures on network devices which do not run FortiMonitor Agents

• Trigger API calls to virtualization systems like VCenter, Nutanix, HyperV

• Trigger API calls to cloud providers (AWS, Azure, Google) for PaaS and SaaS services

Requirements

It is recommended that you upgrade to the latest version of the OnSight vCollector before proceeding. To verify that an OnSight vCollector is of appropriate version to run OnSight-based CounterMeasures, run the following:

From Root (using sudo -i, for example):

onsight status

Directory Structure

The following outlines the directory structure relevant to using OnSight Countermeasures.

• Root directory
  /usr/share/onsight/countermeasures

• Directory to install python packages required to run plugins
  /usr/share/onsight/countermeasures/dependencies/

• Directory to put countermeasure plugins
  /usr/share/onsight/countermeasures/plugins/

• Directory to put secret repository configurations
  /usr/share/onsight/countermeasures/secrets/

• Directory to put ssh private key files
  /usr/share/onsight/countermeasures/ssh-keys/

• Contains the countermeasure log files
  /var/log/countermeasures

Secret management framework

You can add SSH keys directly to your OnSight VM in the directory outlined above, but a more scalable solution exists in the form of secret management vaults. Currently, FortiMonitor only supports HashiCorp Vault while other vault technologies remain in development.

You will need to utilize the Vault URL, token, path, mount point, and version as shown in the example below.

Secret repository configuration

• Create a repository directory under /usr/share/onsight/countermeasures/secrets and insert the config.json file in it (use the repository name for the directory name).

Hashicorp Vault Key-Value engine example:

• Sample config.json (/usr/share/onsight/countermeasures/secrets/vault-kv/config.json)

{

"type": "hashicorp.vault.kv",

"url": "Your-Vault-URL,

"token": "Your-Vault-Token",

"path": "Your-Vault-Path",

"mount_point": "Your-Vault-MountPoint/Top-Level-Directory",

"kv_version": "v1"

}

Hashicorp Vault SSH engine example:

• Sample config.json (/usr/share/onsight/countermeasures/secrets/vault-ssh/config.json)

{

"type": "hashicorp.vault.ssh",

"url": "Your-Vault-URL,

"token": "Your-Vault-Token",

"path": "Your-Vault-Path",

"mount_point": "Your-Vault-MountPoint/Top-Level-Directory",

"api_version": "v1"

}

• The CA public key must be added to target host’s SSH configuration. Add the public key to all target host's SSH configuration.

  • Download the CA public key.
    curl -o /etc/ssh/trusted-user-ca-keys.pem http://URL:PORT/v1/ssh-client-signer/public_key

  • Edit SSH configuration.

    # /etc/ssh/sshd_config

    # ...

    TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
    

  • Restart the SSH service to pick up the changes.

Secret management helper functions for CounterMeasure plugins

The following functions have been pre-defined to assist in developing custom CounterMeasures.

def get_secret(self, repo, key=None, **kwargs):

"""

GET secret from Secret management framework

"""

def get_ssh_certificate(self, repo, username, **kwargs):

"""

GET ssh certificate from Secret management framework

"""

def execute_ssh_with_secret_pkey(self, cmd, hostname, username, repo, key):

"""

Use Spur/Paramiko to ssh to a remote server and run a command.

Get private key value in secret repo with a given key

Returns a tuple of (return code, stdout, stderr)

"""

def execute_ssh_with_secret_cert(self, cmd, hostname, username, repo):

"""

Use Spur/Paramiko to ssh to a remote server and run a command.

Get ssh cert file from secret repo

Returns a tuple of (return code, stdout, stderr)

"""

SSH access scenarios

Signed SSH certificate

The instructions below include details for each aspect of this setup.

• Configure the Vault

  • Set up Hashicorp Vault server and create SSH engine.

  • Configure CA for signing and create a named Vault role for signing client keys.

  • Signed SSH Certificates

• On the remote host

  • Add the public key to all target host's SSH configuration.

  • Download CA public key.
    curl -o /etc/ssh/trusted-user-ca-keys.pem http://URL:PORT/v1/ssh-client-signer/public_key

  • Edit ssh configuration
    # /etc/ssh/sshd_config
# ...
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

  • Restart the SSH service to pick up the changes.

• On the OnSight

  • Create config.json under countermeasures/secrets/vault-ssh

    {

    "type": "hashicorp.vault.ssh",

    "url": "Your-Vault-URL,

    "token": "Your-Vault-Token",

    "role": "ssh-client-signer/sign/my-role",

    "api_version": "v1"

    }

• Within the CounterMeasure plugin

  • Use execute_ssh_with_secret_cert() method to execute command on a remote host

    class VaultDemoCountermeasure(CountermeasurePlugin):

    name = "Vault demo countermeasure"

    textkey = "testing.vault"

    def run(self):

    server = self.metadata.get('server')

    hostname = server.get('fqdn')

    return_code, output, _ = self.execute_ssh_with_secret_cert("pwd", hostname, "vadmin", 'vault-ssh')

    self.save_output(output)

SSH key-pair access

• Store SSH private key in Vault KV engine and access remote Host with private key.

• Configure the Vault.

  • Set up Hashicorp Vault server and create KV engine.

  • Save private key in KV engine

    • For example: key=”mykey”, value=”-----BEGIN RSA PRIVATE KEY-----…”

• On the remote host

  • Copy the public key to the remote host.

• On the OnSight

  • Create config.json under countermeasures/secrets/vault-kv.

    {

    "type": "hashicorp.vault.kv",

    "url": "YOUR-VAULT-URL:PORT",

    "token": "YOUR-VAULT-TOKEN",

    "kv_version": "v1",

    "mount_point": "Your-Vault-MountPoint/Top-Level-Directory",

    "path": "Your-Vault-Path",

    }

• Within the CounterMeasure plugin

  • Use execute_ssh_with_secret_pkey() method to execute command on the remote host.

    class VaultDemoCountermeasure(CountermeasurePlugin):

    name = "Vault demo countermeasure"

    textkey = "testing.vault"

    def run(self):

    server = self.metadata.get('server')

    hostname = server.get('fqdn')

    return_code, output, _ = self.execute_ssh_with_secret_pkey("pwd", hostname, "myuser", 'vault-kv', 'mykey')

    self.save_output(output)

  • Sse get_secret() method to get private key, if you want to use your own code to access to a remote host.

    class VaultDemoCountermeasure(CountermeasurePlugin):

    name = "Vault demo countermeasure"

    textkey = "testing.vault"

    def run(self):

    server = self.metadata.get('server')

    hostname = server.get('fqdn')

    pkey = self.get_key('vault-kv', 'mykey')

    # access to a remote host with pkey

Development environment

A development environment is available to create and test CounterMeasures before deploying directly to an OnSight vCollector. This may be run within any environment supporting Python and does not need to be installed to an OnSight.

More information on creating custom countermeasures may be found here. Review the Implementation Reference section for details on the instance variables used within the countermeasure.

Installation

1. Download the script located here:

2. https://packages.panopta.com/install/onsight/onsight-dev.sh

2. Create a CounterMeasures directory and the requisite sub-directories(dependencies, logs, plugins, secrets, ssh-keys). See Directory Structure for more details on structure layout.

3. Add countermeasure script to the countermeasures/plugins directory

mkdir -p countermeasures/dependencies/

mkdir -p countermeasures/logs

mkdir -p countermeasures/plugins/

mkdir -p countermeasures/secrets/

mkdir -p countermeasures/ssh-keys/

Run test

Your may test a custom CounterMeasure with the option of no metadata, artificially created metadata, or using actual metadata from an incident that has already occurred within the FortiMonitor environment by using an API call. Logs are stored in /countermeasures/logs/countermeasure.log.

1. execute a plugin without metadata
   ./onsight-cm-dev.sh run --textkey=testing.getinterfaces

2. execute a plugin with metadata file
   ./onsight-cm-dev.sh run --textkey=testing.getinterfaces --metadata=metadata

3. execute a plugin with metadata from API
   ./onsight-cm-dev.sh run --textkey=testing.getinterfaces --api-key=YOUR-API-TOKEN --incident-id=-YOUR-INCIDENT-NUMBER
# you can specify api host like "--api-host=http://api-new.panopta-testing.com/"

4. dump incident metadata by calling API

   ./onsight-cm-dev.sh dump --api-key=YOUR-API-TOKEN --incident-id=-YOUR-INCIDENT-NUMBER

   # you can specify api host like "--api-host=http://api-new.panopta-testing.com/"

5. sample metadata file

{

"countermeasure_metadata": {

"resource": {

"item_type": "disk/usage.percent_used",

"resource_option": "/dev/xvda1 mounted at /",

"id": 65822241,

"name": "disk: disk % used - /dev/xvda1 mounted at /"

},

"metric": [

"disk: disk % used - /dev/xvda1 mounted at /"

],

"outage": {

"reasons": "disk: disk % used - /dev/xvda1 mounted at / greater than 95% for more than 5 minutes",

"timestamp": "2021-08-09 23:02:02 UTC",

"alert_label": "145014199",

"id": -145014199,

"severity": "critical"

},

"server": {

"server_key": "2kbs-bjhi-eggr-qxzy",

"Operating System": "Linux",

"Kernel Version": "4.14.225",

"AWS Instance Size": "t2.micro",

"partner_server_id": "",

"name": "OT-agent1",

"CPU Architecture": "x86_64",

"fqdn": "3.143.73.191",

"AWS Region": "us-east-2",

"Cloud Provider": "aws",

"CPU Core Count": "1",

"attributes": {

"Operating System": "Linux",

"Kernel Version": "4.14.225",

"AWS Instance Size": "t2.micro",

"CPU Architecture": "x86_64",

"AWS Region": "us-east-2",

"Cloud Provider": "aws",

"CPU Core Count": "1",

"AWS Availability Zone": "us-east-2c",

"Server Origin": "agent",

"AWS Image ID": "ami-05d72852800cbf29e"

},

"AWS Availability Zone": "us-east-2c",

"Server Origin": "agent",

"id": 6770198,

"tags": [

"Linux",

"apache",

"aws",

"mysql",

"t2.micro",

"us-east-2",

"us-east-2c"

],

"AWS Image ID": "ami-05d72852800cbf29e"

},

"services": [],

"metric_tags": []

}

}

See also

• Configuring a CounterMeasure

• Viewing CounterMeasure output

• CounterMeasure States

• Standard CounterMeasure actions

• Custom CounterMeasure actions