Troubleshooting High Availability

• Adding Primary Follower

• Follower becomes Leader

Adding Primary Follower

To troubleshoot adding a Primary Follower, take the following steps.

1. In the upper right GUI, click on the Jobs/Errors icon.

2. In the Search bar, enter "Add Super Follower" to search for "Add Super Follower".
   

   In the Status column, the job must be Done, and in the Progress column, the progress must be 100%.

   

   The following logs (with keyword "670-Cluster-AddSuperFollower") can be found in /opt/phoenix/logs/phoenix_config.txt. For the sample logs below, 10.65.20.71 is the Leader and 10.65.20.78 is the Follower.

   Sample Logs

   2022-12-27T10:57:13.976710-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=48,[phLogDetail]=670-Cluster-AddSuperFollower: Started (<task createTime="1672167429762" id="967181" type="AddSuperFollower"><custId>1</custId><parameters>10.65.20.71,10.65.20.78,</parameters></task>)
   
   2022-12-27T10:57:14.007764-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=51,[phLogDetail]=670-Cluster-AddSuperFollower: Check license
   
   2022-12-27T10:57:14.037766-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=64,[phLogDetail]=670-Cluster-AddSuperFollower: Run script phinitleader 
   
   2022-12-27T10:57:15.149636-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=76,[phLogDetail]=670-Cluster-AddSuperFollower: Get followers IPs
   
   2022-12-27T10:57:15.203904-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=93,[phLogDetail]=670-Cluster-AddSuperFollower: Send AddSuperFollower task to super follower 10.65.20.78
   
   2022-12-27T10:58:56.377972-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=102,[phLogDetail]=670-Cluster-AddSuperFollower: AddSuperFollower task of super follower 10.65.20.78 is Done
   
   2022-12-27T10:58:56.428121-08:00 sp2071 phMonitorSupervisor[57363]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phSuperFollowerManager.cpp,[lineNumber]=145,[phLogDetail]=670-Cluster-AddSuperFollower: Finished
   
   

Follower becomes Leader

After you have executed the failover steps, take the following steps to confirm the switch of Follower becoming Leader.

1. In the upper right GUI, click on the Jobs/Errors icon.

2. In the Search bar, enter "Switch Follower DB" to search for "Switch Follower DB".
   

   In the Status column, the job must be Done, and in the Progress column, the progress must be 100%.

   

The following logs (with keyword "670-Cluster-SwitchFollowerDB") can be found in /opt/phoenix/logs/phoenix_config.txt. For the sample logs below, 10.65.20.71 is the Leader, 10.65.20.78 follows 10.65.20.71, 10.65.20.77 follows 10.65.20.78 and 10.65.20.79 follows 10.65.20.77.

Sample Logs

2022-12-27T12:28:10.835766-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8217,[phLogDetail]=670-Cluster-SwitchFollowerDB: task parameter: 10.65.20.78,

2022-12-27T12:37:51.095162-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8303,[phLogDetail]=670-Cluster-SwitchFollowerDB: Notifying 10.65.20.77

2022-12-27T12:37:51.095410-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8303,[phLogDetail]=670-Cluster-SwitchFollowerDB: Notifying 10.65.20.79

2022-12-27T12:38:08.292191-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8313,[phLogDetail]=670-Cluster-SwitchFollowerDB: Successfully notified 10.65.20.77

2022-12-27T12:38:22.231424-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8269,[phLogDetail]=670-Cluster-SwitchFollowerDB: Started threads to notify

2022-12-27T12:40:46.987212-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8313,[phLogDetail]=670-Cluster-SwitchFollowerDB: Successfully notified 10.65.20.79

2022-12-27T12:40:46.987823-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8273,[phLogDetail]=670-Cluster-SwitchFollowerDB: All threads have completed

2022-12-27T12:40:46.987867-08:00 b2078 phMonitorSupervisor[5610]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8285,[phLogDetail]=670-Cluster-SwitchFollowerDB: Successfully notified all active Supervisors: 10.65.20.79 10.65.20.77