Test case 4
Starting position:
-
Root CA certificate DigiCert Global Root G2 installed on both FortiAnalyzers.
-
Deleted the Microsoft Azure TLS Issuing CA 06 CA certificate.
-
Rebooted both FortiAnalyzers.
-
FAZHA001 is the primary.
-
FAZHA002 is the secondary.
-
lynx001 continuous ping targeting the VIP 10.0.10.10.
-----BEGIN CERTIFICATE-----
MIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH
MjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI
2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx
1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ
q2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz
tCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ
vIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP
BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV
5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY
1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4
NeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG
Fdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91
8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe
pLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl
MrY=
-----END CERTIFICATE-----
openssl x509 -inform der -in DigiCertGlobalRootG2.crt -text ─╯
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
xxxxxxx
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
Validity
Not Before: Aug 1 12:00:00 2013 GMT
Not After : Jan 15 12:00:00 2038 GMT
Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:bb:37:cd:34:dc:7b:6b:c9:b2:68:90:ad:4a:75:
ff:46:ba:21:0a:08:8d:f5:19:54:c9:fb:88:db:f3:
ae:f2:3a:89:91:3c:7a:e6:ab:06:1a:6b:cf:ac:2d:
e8:5e:09:24:44:ba:62:9a:7e:d6:a3:a8:7e:e0:54:
75:20:05:ac:50:b7:9c:63:1a:6c:30:dc:da:1f:19:
b1:d7:1e:de:fd:d7:e0:cb:94:83:37:ae:ec:1f:43:
4e:dd:7b:2c:d2:bd:2e:a5:2f:e4:a9:b8:ad:3a:d4:
99:a4:b6:25:e9:9b:6b:00:60:92:60:ff:4f:21:49:
18:f7:67:90:ab:61:06:9c:8f:f2:ba:e9:b4:e9:92:
32:6b:b5:f3:57:e8:5d:1b:cd:8c:1d:ab:95:04:95:
49:f3:35:2d:96:e3:49:6d:dd:77:e3:fb:49:4b:b4:
ac:55:07:a9:8f:95:b3:b4:23:bb:4c:6d:45:f0:f6:
a9:b2:95:30:b4:fd:4c:55:8c:27:4a:57:14:7c:82:
9d:cd:73:92:d3:16:4a:06:0c:8c:50:d1:8f:1e:09:
be:17:a1:e6:21:ca:fd:83:e5:10:bc:83:a5:0a:c4:
67:28:f6:73:14:14:3d:46:76:c3:87:14:89:21:34:
4d:af:0f:45:0c:a6:49:a1:ba:bb:9c:c5:b1:33:83:
29:85
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
X509v3 Subject Key Identifier:
4E:22:54:20:18:95:E6:E3:6E:E6:0F:FA:FA:B9:12:ED:06:17:8F:39
Signature Algorithm: sha256WithRSAEncryption
60:67:28:94:6f:0e:48:63:eb:31:dd:ea:67:18:d5:89:7d:3c:
c5:8b:4a:7f:e9:be:db:2b:17:df:b0:5f:73:77:2a:32:13:39:
81:67:42:84:23:f2:45:67:35:ec:88:bf:f8:8f:b0:61:0c:34:
a4:ae:20:4c:84:c6:db:f8:35:e1:76:d9:df:a6:42:bb:c7:44:
08:86:7f:36:74:24:5a:da:6c:0d:14:59:35:bd:f2:49:dd:b6:
1f:c9:b3:0d:47:2a:3d:99:2f:bb:5c:bb:b5:d4:20:e1:99:5f:
53:46:15:db:68:9b:f0:f3:30:d5:3e:31:e2:8d:84:9e:e3:8a:
da:da:96:3e:35:13:a5:5f:f0:f9:70:50:70:47:41:11:57:19:
4e:c0:8f:ae:06:c4:95:13:17:2f:1b:25:9f:75:f2:b1:8e:99:
a1:6f:13:b1:41:71:fe:88:2a:c8:4f:10:20:55:d7:f3:14:45:
e5:e0:44:f4:ea:87:95:32:93:0e:fe:53:46:fa:2c:9d:ff:8b:
22:b9:4b:d9:09:45:a4:de:a4:b8:9a:58:dd:1b:7d:52:9f:8e:
59:43:88:81:a4:9e:26:d5:6f:ad:dd:0d:c6:37:7d:ed:03:92:
1b:e5:77:5f:76:ee:3c:8d:c4:5d:56:5b:a2:d9:66:6e:b3:35:
37:e5:32:b6
FAZHA001 DigiCert Global Root G2 installed:
config system certificate ca
edit "DigiCertGlobalRootG2"
set ca "-----BEGIN CERTIFICATE-----
MIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH
MjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI
2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx
1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ
q2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz
tCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ
vIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP
BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV
5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY
1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4
NeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG
Fdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91
8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe
pLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl
MrY=
-----END CERTIFICATE-----"
set comment "Created by CA certificate"
next
end
FAZHA002 DigiCert Global Root G2 installed:
config system certificate ca
edit "DigiCertGlobalRootG2"
set ca "-----BEGIN CERTIFICATE-----
MIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH
MjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI
2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx
1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ
q2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz
tCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ
vIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP
BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV
5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY
1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4
NeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG
Fdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91
8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe
pLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl
MrY=
-----END CERTIFICATE-----"
set comment "Created by CA certificate"
next
end
Result:
After executing diagnose ha failover on the primary (FAZHA001), the ping to the VIP was stocked. FortiAnalyzer HA worked as expected and the new primary is now FAZHA002 (see below). After ~2min 30sec the ping starts woking again and the VIP IP address is transferred to the new primary FAZHA002.
lynx001 ping:
azureuser@lynx01:~$ ping 10.0.10.10
PING 10.0.10.10 (10.0.10.10) 56(84) bytes of data.
64 bytes from 10.0.10.10: icmp_seq=1 ttl=64 time=1.19 ms
64 bytes from 10.0.10.10: icmp_seq=2 ttl=64 time=0.600 ms
64 bytes from 10.0.10.10: icmp_seq=3 ttl=64 time=8.11 ms
From 10.0.10.6 icmp_seq=82 Destination Host Unreachable
From 10.0.10.6 icmp_seq=83 Destination Host Unreachable
From 10.0.10.6 icmp_seq=84 Destination Host Unreachable
...
From 10.0.10.6 icmp_seq=136 Destination Host Unreachable
From 10.0.10.6 icmp_seq=137 Destination Host Unreachable
From 10.0.10.6 icmp_seq=138 Destination Host Unreachable
64 bytes from 10.0.10.10: icmp_seq=141 ttl=64 time=4.91 ms
64 bytes from 10.0.10.10: icmp_seq=142 ttl=64 time=0.772 ms
64 bytes from 10.0.10.10: icmp_seq=143 ttl=64 time=0.759 ms
FAZHA001 diagnose ha status:
HA-Status: Secondary
up-time: 4m7.962s
config-sync: Allow
serial-no: FAZVMSTMxxxxxxx3
fazuid: 3433169053
hostname: fazha001
load balance status: 0x8
HA-Primary fazha@10.0.10.5 FAZVMSTMxxxxxxx4
ip: 10.0.10.5
serial-no: FAZVMSTMxxxxxxx4
fazuid: 4148610926
hostname: fazha002
conn-st: up
up/down-time: 4m7.962s
conn-msg:
cfgsync-st: up, 3m49.153s
data-init-sync-st: done, 4m4.160s
FAZHA001 checking HA log:
bash$ tail -f /var/private/clusterd/faz-ha.log
2023/02/03 00:23:15 <1946> main: -> BACKUP
2023/02/03 00:23:15 <1946> to_BACKUP: -> BACKUP
2023/02/03 00:23:28 <2578> check_empty: platform=FAZVM64-AZURE
2023/02/03 00:23:28 <2578> main: #0: 10.0.10.10 port1 00:0d:3a:bb:fc:4a
2023/02/03 00:23:28 <2578> main: -> MASTER
2023/02/03 00:23:29 [add-ip] 10.0.10.10 is private IP
2023/02/03 00:23:29 [add-ip] secondary IP 10.0.10.10 not attached to any NIC
2023/02/03 00:23:29 [add-ip] lookupVNetByNIC() CMM_Vnet_FAZ
2023/02/03 00:23:29 [add-ip] lookupSubnetIDByIP() /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CMM_RG_FAZHA/providers/Microsoft.Network/virtualNetworks/CMM_Vnet_FAZ/subnets/cmm_vnet_dc01
2023/02/03 00:23:29 [add-ip] Update NIC fazha001-NIC0, add secondary IP 10.0.10.10
2023/02/03 00:25:53 <3707> check_empty: platform=FAZVM64-AZURE
2023/02/03 00:25:53 <3707> main: #0: 10.0.10.10 port1 00:0d:3a:bb:fc:4a
2023/02/03 00:25:53 <3707> main: -> STOP
2023/02/03 00:25:53 <3707> main: -> STOP
2023/02/03 00:25:54 <3720> check_empty: platform=FAZVM64-AZURE
2023/02/03 00:25:54 <3720> main: #0: 10.0.10.10 port1 00:0d:3a:bb:fc:4a
2023/02/03 00:25:54 <3720> main: -> BACKUP
2023/02/03 00:25:54 <3720> to_BACKUP: -> BACKUP
FAZHA002 diagnose ha status:
HA-Status: Primary
up-time: 3m55.667s
config-sync: Allow
serial-no: FAZVMSTMxxxxxxx4
fazuid: 4148610926
hostname: fazha002
load balance status: 0x0
HA-Secondary fazha@10.0.10.4 FAZVMSTMxxxxxxx3
ip: 10.0.10.4
serial-no: FAZVMSTMxxxxxxx3
fazuid: 3433169053
hostname: fazha001
conn-st: up
up/down-time: 3m55.067s
conn-msg:
cfgsync-st: up, 3m34.942s
data-init-sync-st: done, 3m46.947s
FAZHA002 checking HA log:
bash$ tail -f /var/private/clusterd/faz-ha.log
2023/02/03 00:21:53 <14597> main: -> MASTER
2023/02/03 00:21:54 [add-ip] 10.0.10.10 is private IP
2023/02/03 00:21:57 <14630> check_empty: platform=FAZVM64-AZURE
2023/02/03 00:21:57 <14630> main: #0: 10.0.10.10 port1 60:45:bd:f5:2b:32
2023/02/03 00:21:57 <14630> main: -> STOP
2023/02/03 00:21:57 <14630> main: -> STOP
2023/02/03 00:23:20 <1953> check_empty: platform=FAZVM64-AZURE
2023/02/03 00:23:20 <1953> main: #0: 10.0.10.10 port1 60:45:bd:f5:2b:32
2023/02/03 00:23:20 <1953> main: -> BACKUP
2023/02/03 00:23:20 <1953> to_BACKUP: -> BACKUP
2023/02/03 00:25:54 <4212> check_empty: platform=FAZVM64-AZURE
2023/02/03 00:25:54 <4212> main: #0: 10.0.10.10 port1 60:45:bd:f5:2b:32
2023/02/03 00:25:54 <4212> main: -> MASTER
2023/02/03 00:25:55 [add-ip] 10.0.10.10 is private IP
2023/02/03 00:27:26 [add-ip] removed ip 10.0.10.10 from NIC fazha001-NIC0
2023/02/03 00:27:26 [add-ip] removed ip 10.0.10.10 in subnet /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/CMM_RG_FAZHA/providers/Microsoft.Network/virtualNetworks/CMM_Vnet_FAZ/subnets/cmm_vnet_dc01
2023/02/03 00:27:26 [add-ip] Update NIC fazha002-NIC0, add secondary IP 10.0.10.10