Fortinet Document Library

Version:


Table of Contents

User Guide

21.4.0
Copy Link

Troubleshooting Windows Agent Heartbeat Incident

An agent heartbeat incident occurs when the FortiMonitor agent is unable to sync to our platform. The following troubleshooting steps should be performed on Windows servers experiencing heartbeat incidents.

Determine if the Server Running the Agent is Able To Reach Our Platform

  • Method 1: Checking through Windows PowerShell (Useful for Windows Server Core)

    • In a Windows PowerShell command line, run the following command:
      invoke-webrequest https://aggregator2.panopta.com/v2/hello

    • You should receive a response similar to the following:

      StatusCode        : 200
      StatusDescription : OK
      Content           : {"hello": "world", "timestamp": 1617805078}
      RawContent        : HTTP/1.1 200 OK
                          X-Frame-Options: deny
                          Alt-Svc: clear
                          Content-Length: 43
                          Content-Type: application/json
                          Date: Wed, 07 Apr 2021 14:17:58 GMT
                          Via: 1.1 google
       
                          {"hello": "world", "timestamp": 1617...
      Forms             : {}
      Headers           : {[X-Frame-Options, deny], [Alt-Svc, clear], [Content-Length, 43], [Content-Type,
                          application/json]...}
      Images            : {}
      InputFields       : {}
      Links             : {}
      ParsedHtml        : System.__ComObject
      RawContentLength  : 43

  • Method 2: Checking through Web Browser

    • In a web browser on the target server, enter the following web address into the browser https://aggregator2.panopta.com/v2/hello

    • If successful, you should receive output similar to the following in the web browser

      Navigating to endpoint from browser.

If you do not receive a 200 status code similar to what is shown in the method 1 example above, or if method 2 fails, this indicates that this server is unable to reach our platform and is unable to sync. If this server cannot reach our platform, the agent will experience an agent heartbeat incident. Please check your network configuration to determine why the server cannot reach our platform.

Outbound traffic requirements can be found in this article.

Verify That the FortiMonitor Agent Service Is Running

  • Method 1: Checking through PowerShell (Useful for Windows Server Core)

    • In a Windows PowerShell command line, run the following command:
      Get-Service PanoptaAgent

    • You should receive output to show that the service is “Running”.

      Status   Name               DisplayName
      ------   ----               -----------
      Running  PanoptaAgent       FortiMonitor Agent Service
    • If the PanoptaAgent service status is shown as “Stopped”, you can run the below command to start it.
      Start-Service PanoptaAgent

  • Method 2: Checking through Windows Service Application

    • Open the Windows services application. You can find this by searching for “Services” in the Windows Start Menu.

    • In the Windows services application, confirm that the PanoptaAgent service status is “Running”. If it is not, start the application.

The FortiMonitor Agent Windows service must be running in order to sync. If the service is stopped, an agent heartbeat incident will occur.

Confirm the Agent Server Key Matches the Server Key of the Affected FortiMonitor Instance

Every agent instance has a server key. The server key of the device you are troubleshooting, should match the server key of the instance shown in the FortiMonitor Control Panel.

  • Method 1: Getting Server Key through Windows PowerShell

    • The agent server key can always be found in the agent.config file for Windows. To get the server key through PowerShell, you can run the following command:
      Select-String -Path "C:\Program Files (x86)\PanoptaAgent\Agent.config" -Pattern "ServerKey"

    • After running the above command you should receive output showing the server key. In the below example, xmft-bi65-xdx5-vqeg is the server key.

      PS C:\Users\Administrator> Select-String -Path "C:\Program Files (x86)\PanoptaAgent\Agent.config" -Pattern "ServerKey"C:\Program Files (x86)\PanoptaAgent\Agent.config:5:    <add key="ServerKey" value="xmft-bi65-xdx5-vqeg" />

Once you have the agent server key, confirm that this key matches the key that you are seeing in the FortiMonitor Control Panel. If the server keys do not match, confirm you are looking at the right server.

If you are looking at the correct server, then you will need to change the server key on the instance to match the key that is found in the agent.config file.

Server Key Example as Shown on Server Instance:

Server Key Shown In Panopta Control Panel

  • Method 2: Getting the Server Key Directly From the agent.config File

    • Navigate to the following directory in Windows Explorer: C:\Program Files (x86)\PanoptaAgent

    • In this directory, you should find the agent.config file. Open that file in a text editor to find the server key.

    • The server key should be found towards the top of the file (example below for reference).

      <agent>  <service>    <add key="AggregatorUrl" value="https://aggregator2.panopta.com" />    <add key="ServerKey" value="xmft-bi65-xdx5-vqeg" />    <add key="MaintenanceDuration" value="" />    <add key="ShowPublicStatus" value="" />    <add key="EnableCounterMeasures" value="" />    <add key="CounterMeasuresRemotePlugins" value="" />    <add key="CounterMeasuresRefreshPlugins" value="" />    <add key="MetricIncomingDirectory" value="" />    <!--

Once you have the agent server key, confirm that this key matches the key that you are seeing in the FortiMonitor Control Panel. If the server keys do not match, confirm you are looking at the right server.

If you are looking at the correct server, then you will need to change the server key on the instance to match the key that is found in the agent.config file.

Server Key Example as Shown on Server Instance:

Server Key Shown in Panopta Control Panel

Verify That Storage Disk is Not at Capacity

  • Check that the storage capacity on the server to verify that there is still space available. If the disk is full, the agent will not continue to operate.

If the steps above do not identify the cause of the agent heartbeat problem, please send us a copy of your agent logs. Our team will look at them and follow-up with you.

FortiMonitor Agent Log Location: C:\Program Files (x86)\PanoptaAgent\logs

Troubleshooting Windows Agent Heartbeat Incident

An agent heartbeat incident occurs when the FortiMonitor agent is unable to sync to our platform. The following troubleshooting steps should be performed on Windows servers experiencing heartbeat incidents.

Determine if the Server Running the Agent is Able To Reach Our Platform

  • Method 1: Checking through Windows PowerShell (Useful for Windows Server Core)

    • In a Windows PowerShell command line, run the following command:
      invoke-webrequest https://aggregator2.panopta.com/v2/hello

    • You should receive a response similar to the following:

      StatusCode        : 200
      StatusDescription : OK
      Content           : {"hello": "world", "timestamp": 1617805078}
      RawContent        : HTTP/1.1 200 OK
                          X-Frame-Options: deny
                          Alt-Svc: clear
                          Content-Length: 43
                          Content-Type: application/json
                          Date: Wed, 07 Apr 2021 14:17:58 GMT
                          Via: 1.1 google
       
                          {"hello": "world", "timestamp": 1617...
      Forms             : {}
      Headers           : {[X-Frame-Options, deny], [Alt-Svc, clear], [Content-Length, 43], [Content-Type,
                          application/json]...}
      Images            : {}
      InputFields       : {}
      Links             : {}
      ParsedHtml        : System.__ComObject
      RawContentLength  : 43

  • Method 2: Checking through Web Browser

    • In a web browser on the target server, enter the following web address into the browser https://aggregator2.panopta.com/v2/hello

    • If successful, you should receive output similar to the following in the web browser

      Navigating to endpoint from browser.

If you do not receive a 200 status code similar to what is shown in the method 1 example above, or if method 2 fails, this indicates that this server is unable to reach our platform and is unable to sync. If this server cannot reach our platform, the agent will experience an agent heartbeat incident. Please check your network configuration to determine why the server cannot reach our platform.

Outbound traffic requirements can be found in this article.

Verify That the FortiMonitor Agent Service Is Running

  • Method 1: Checking through PowerShell (Useful for Windows Server Core)

    • In a Windows PowerShell command line, run the following command:
      Get-Service PanoptaAgent

    • You should receive output to show that the service is “Running”.

      Status   Name               DisplayName
      ------   ----               -----------
      Running  PanoptaAgent       FortiMonitor Agent Service
    • If the PanoptaAgent service status is shown as “Stopped”, you can run the below command to start it.
      Start-Service PanoptaAgent

  • Method 2: Checking through Windows Service Application

    • Open the Windows services application. You can find this by searching for “Services” in the Windows Start Menu.

    • In the Windows services application, confirm that the PanoptaAgent service status is “Running”. If it is not, start the application.

The FortiMonitor Agent Windows service must be running in order to sync. If the service is stopped, an agent heartbeat incident will occur.

Confirm the Agent Server Key Matches the Server Key of the Affected FortiMonitor Instance

Every agent instance has a server key. The server key of the device you are troubleshooting, should match the server key of the instance shown in the FortiMonitor Control Panel.

  • Method 1: Getting Server Key through Windows PowerShell

    • The agent server key can always be found in the agent.config file for Windows. To get the server key through PowerShell, you can run the following command:
      Select-String -Path "C:\Program Files (x86)\PanoptaAgent\Agent.config" -Pattern "ServerKey"

    • After running the above command you should receive output showing the server key. In the below example, xmft-bi65-xdx5-vqeg is the server key.

      PS C:\Users\Administrator> Select-String -Path "C:\Program Files (x86)\PanoptaAgent\Agent.config" -Pattern "ServerKey"C:\Program Files (x86)\PanoptaAgent\Agent.config:5:    <add key="ServerKey" value="xmft-bi65-xdx5-vqeg" />

Once you have the agent server key, confirm that this key matches the key that you are seeing in the FortiMonitor Control Panel. If the server keys do not match, confirm you are looking at the right server.

If you are looking at the correct server, then you will need to change the server key on the instance to match the key that is found in the agent.config file.

Server Key Example as Shown on Server Instance:

Server Key Shown In Panopta Control Panel

  • Method 2: Getting the Server Key Directly From the agent.config File

    • Navigate to the following directory in Windows Explorer: C:\Program Files (x86)\PanoptaAgent

    • In this directory, you should find the agent.config file. Open that file in a text editor to find the server key.

    • The server key should be found towards the top of the file (example below for reference).

      <agent>  <service>    <add key="AggregatorUrl" value="https://aggregator2.panopta.com" />    <add key="ServerKey" value="xmft-bi65-xdx5-vqeg" />    <add key="MaintenanceDuration" value="" />    <add key="ShowPublicStatus" value="" />    <add key="EnableCounterMeasures" value="" />    <add key="CounterMeasuresRemotePlugins" value="" />    <add key="CounterMeasuresRefreshPlugins" value="" />    <add key="MetricIncomingDirectory" value="" />    <!--

Once you have the agent server key, confirm that this key matches the key that you are seeing in the FortiMonitor Control Panel. If the server keys do not match, confirm you are looking at the right server.

If you are looking at the correct server, then you will need to change the server key on the instance to match the key that is found in the agent.config file.

Server Key Example as Shown on Server Instance:

Server Key Shown in Panopta Control Panel

Verify That Storage Disk is Not at Capacity

  • Check that the storage capacity on the server to verify that there is still space available. If the disk is full, the agent will not continue to operate.

If the steps above do not identify the cause of the agent heartbeat problem, please send us a copy of your agent logs. Our team will look at them and follow-up with you.

FortiMonitor Agent Log Location: C:\Program Files (x86)\PanoptaAgent\logs