Alert

The Alert tab enables you to search through your existing alerts and set rules on how you receive alerts. You can also configure how you want to receive push notifications through various notification channels such as email, Slack, PagerDuty, WebHook, and more.

Notification channel alerts

You can add new ways of receiving alerts by adding a channel and specifying the channel type.

To create a notification channel with email

The following example shows how to set up the SMTP server and create an email notification channel.

1. Go to Services > Monitor > Configuration.
2. Enable SMTP.
3. In the SMTP Host field, enter the SMTP server address and SMTP port.
   The format is <SMTP server address>:<SMTP port number>. For example, smtp.gmail.com:587.
4. In the SMTP TLS Policy field, select TLS policy.
5. (Optional) Enable SMTP authentication if authentication is required.
6. In the SMTP Auth User and SMTP Auth Password fields, enter the username and password for authentication.
7. Click Save.
8. Go to the Instances tab and click Apply Config. The configuration changes take effect and triggers the Enable Smtp command.
   Wait for the command to finish running.
9. Go to Monitor > Health > Alert > Notification channels and click New channel.
10. In the Name field, enter a name for the channel.
11. In the Type field, select Email.
12. In the Addresses field, enter the destination email addresses for notifications. Separate multiple email addresses with a semi-colon (;).
13. Click Test and check if you can receive the test alert email.
14. Click Save once you have verified the email channel alert works.

To create a notification channel with Slack Incoming Webhook

The following example shows how to create a notification channel with Slack Incoming Webhook and set up an alert.

1. Go to Monitor > Health > Alert > Notification channels and click Add channel.
2. In the Name field, enter a name for the channel.
3. In the Type field, select Slack.
4. You can choose how you want to configure your alert.
   In this example, enable the Include image toggle so a snapshot of your Slack chart can be sent with the alert.
5. In the URL field, enter your Slack Incoming Webbook URL.
   For instructions on how to create a Slack Incoming Hook, refer to the Slack documentation.
6. In the Token field, enter the in the Slack “Bot User OAuth Access Token” in order to allow the generated image to be uploaded via Slack’s file.upload API method.
7. In Slack, invite the bot to the channel you want to send notifications to and add the Slack channel name to the Recipient field.
8. Click Send Test and check if you can see the test message in your Slack channel with the Webhook hooked.
9. Once you have verified that the channel alert works, click Save.
   

Custom alert rules

You can create custom alert rules from Dashboard panels and have it be sent to a specified notification channel.

To create a custom alert for a notification channel

The following example shows how to create custom alert rule that can be sent directly to the example Slack notification channel.

1. Go to Monitor > Dashboard and select a panel you want to create an alert for.
2. Click the panel title and click Edit.
   
   The panel's detailed view loads.
3. Click Alert to access the Alert view and click Create Alert to specify conditions that trigger the alert.
4. You can create conditions through two different methods:
   • By making queries in the Conditions section.

     

   • By dragging the threshold bar in the graph to indicate an allowable threshold level.

     

5. After you've defined your condition, select the Notification Channel and click Test Rule to test the alert rule.
6. Click Save to save your settings.
   If your conditions are configured correctly, you should receive an alert with snapshot resembling the following: