Troubleshooting
Related KB Articles
Manual High Availability (HA) restore in a L3 environment via CLI
Occasional Database Replication Error Alarms
Database replication Error in a FortiNAC HA setup
named.conf Not Replicating in L3 High Availability (HA) Environment
Determine Which Appliance Has the Shared IP (Layer 2 HA)
Enter ip addr sh dev eth0 at the command prompt and look at the output to determine which eth0 interface has the Shared IP Address (eth0 of the primary or eth0 of the secondary). In the example below, the Shared IP Address is 192.168.8.25. The eth0 on the primary has the Shared IP Address.
Primary Server
> ip addr sh dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:2e:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.8.23/24 brd 192.168.8.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.8.25/24 scope global secondary eth0 Shared IP
valid_lft forever preferred_lft forever
Secondary Server
> ip addr sh dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:08:4e brd ff:ff:ff:ff:ff:ff
inet 192.168.8.26/24 brd 192.168.8.255 scope global eth0
valid_lft forever preferred_lft forever
Shared IP Missing After Network Service Restart
Shared IP association to eth0 is not persistent through network service restarts on the appliance in control. Example after running “service network restart” on the Primary Server:
> ip addr sh dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:30:48:f9:9e:b6 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.208/16 brd 10.1.255.255 scope global eth0
inet6 fe80::230:48ff:fef9:9eb6/64 scope link
valid_lft forever preferred_lft forever
If network services are restarted on the appliance in control, re-add the Shared IP.
Re-Add Shared IP Address
Use one of the options below to re-add the Shared IP address.
Option 1 (Does not require a restart of FortiNAC services)
Log in to the CLI as root of the appliance in control and type
hsIP ADD <Virtual IP address> <mask CIDR format> eth0
Example:
> hsIP ADD 192.168.8.25 24 eth0
Option 2: Restart FortiNAC services on the appliance in control
Log in to the appliance CLI as root and type
shutdownNAC
<wait 30 seconds>
startupNAC
Option 3: Reboot Appliance via UI
-
In the Administration UI, navigate to System > Settings > System Management > Power Management.
-
Select the appliance in control and click Reboot.
Verify Shared IP
Confirm the Shared IP entry has been re-added to eth0 via the CLI. Type
ip addr sh dev eth0
Example:
> ip addr sh dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:2e:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.8.23/24 brd 192.168.8.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.8.25/24 scope global secondary eth0 Shared IP
valid_lft forever preferred_lft forever
Verify License Key Configuration
The High Availability feature is included in BASE, PLUS and PRO licenses.
For more information on licensing, refer to the License Upgrade Guide in the Document Library.
License Entitlements
The license can be verified using the command licensetool.
Example:
> licensetool
EFFECTIVE:
serial = xxxxx
type = NetworkControlApplicationServer
level = PRO
count = 100000
expiration = 31622400000
expired = false
mac = 00:50:56:98:34:73
uuid = 4218e64a-d8f1-39e3-471f-46e2c5f027df
certificates = [xxxx]
> licensetool
EFFECTIVE:
serial = xxxxxx
type = NetworkControlApplicationServer
level = PRO
count = 2000
expiration = 31622400000
expired = false
mac = 00:50:56:98:5E:B3
uuid = 4218c883-093b-5e28-f895-bee88bc3202d
certificates = [xxxx]
To view both Primary and Secondary Server licenses at once, log in to the Secondary Server CLI and type
licensetool -key APPLIANCE -key PRIMARY
Example (Output of system with Primary Server in control):
> licensetool -key EFFECTIVE -key APPLIANCE -key PRIMARY -key MANAGER
EFFECTIVE: <--- Key of server in control (Primary Server)
serial = xxxxx
type = NetworkControlApplicationServer
level = PRO
count = 100000
expiration = 31622400000
expired = false
mac = 00:50:56:98:34:73
uuid = 4218e64a-d8f1-39e3-471f-46e2c5f027df
certificates = [xxxx]
APPLIANCE: <--- Secondary Server
serial = xxxxxx
type = NetworkControlApplicationServer
level = PRO
count = 100000
expiration = 31622400000
expired = false
mac = 00:50:56:98:5E:B3
uuid = 4218c883-093b-5e28-f895-bee88bc3202d
certificates = [xxxx]
PRIMARY: <--- Primary Server
serial = xxxxx
type = NetworkControlApplicationServer
level = PRO
count = 100000
expiration = 31622400000
expired = false
mac = 00:50:56:98:34:73
uuid = 4218e64a-d8f1-39e3-471f-46e2c5f027df
certificates = [xxxx]
Validating Processes – CLI
CampusManager - Management Process that runs on all appliances regardless of control status.
Yams - Loader that runs when the appliance status is “Running - In Control.”
To verify if these processes are running, use the “jps” command.
> jps
3828 Yams
2885 CampusManager
4055 Yams
7976 Jps
1400 TomcatAdmin
1548 TomcatPortal
Log in to each appliance as root and type
tail –F /bsc/logs/output.processManager | grep "In Control Idle"
The following message indicates Primary is in control:
Primary Server: (Primary) Primary In Control Idle(false)
Secondary Server: (Secondary) Primary In Control Idle(false)