Fortinet white logo
Fortinet docs logo DOCUMENT LIBRARY
Fortinet white logo
Fortinet docs logo DOCUMENT LIBRARY
Products Best Practices Hardware Guides Products A-Z
Summary
By Solution
By 4D Pillars
By Cloud
Secure Networking
Unified SASE
Security Operations
Secure SD-WAN
Secure Access Service Edge (SASE)
ZTNA
LAN Edge
Identity and Access Management
Next Generation Firewall
Public Cloud
Private Cloud
FortiCloud
Secure Networking
  • Hybrid Mesh Firewall
  • NOC Management
  • LAN
  • WAN
    • More >>
    Unified SASE
  • Single Vendor SASE
  • Cloud Network Security
  • Secure Endpoint Connectivity
  • Web Application / API Protection
    • More >>
    Security Operations
  • Security Operations Automation
  • Identity
  • Early Detection & Prevention
    • More >>
    Secure Networking
  • Hybrid Mesh Firewall
  • NOC Management
  • LAN
  • WAN
  • Communication & Surveillance
    • Unified SASE
  • Single Vendor SASE
  • Secure Endpoint Connectivity
  • Cloud Network Security
  • Cloud-Native Security
  • Web Application / API Protection
    • Security Operations
  • Security Operations Automation
  • Endpoint
  • Data Protection
  • Identity
  • Email
  • Early Detection & Prevention
  • Expert Services
  • Edge Firewall
  • Orchestration & management
  • SD Branch
  • Application Delivery
  • Single Vendor SASE
  • Secure Endpoint Connectivity
  • Secure Private Access
  • Thin Edge
  • Identity
  • Application Gateway
  • Enterprise Asset Management
  • Endpoint Agent
  • Agentless Security Posture
  • Identity
  • Wireless
  • Switching
  • Identity
  • Privilege Acccess Management
  • Next Generation Firewall
  • Orchestration & management
  • Expert Services
  • All
  • All
  • Account Management
  • SAAS Management
  • SAAS Application Security
  • Managed Services
  • Platform as a service (PAAS)
  • Other SAAS Services
    • 4D Resources
    Solution Hubs
  • 4D Pillars
  • Cloud
  • Popular Solutions

    • High Availability and Disaster Recovery Procedures - ClickHouse

    Home FortiSIEM 6.7.1 High Availability and Disaster Recovery Procedures - ClickHouse
    6.7.1
    7.2.4
    7.2.3
    7.2.2
    7.2.1
    7.2.0
    7.1.7
    7.1.6
    7.1.5
    7.1.4
    7.1.3
    7.1.2
    7.1.1
    7.1.0
    7.0.3
    7.0.2
    7.0.1
    7.0.0
    6.7.9
    6.7.8
    6.7.7
    6.7.6
    6.7.5
    6.7.4
    6.7.3
    6.7.2
    6.7.1
    6.7.0

    Troubleshooting Disaster Recovery

    Troubleshooting Disaster Recovery

    GUI

    While Replication is being set up, you can see the progress in the GUI in two locations.

    1. ADMIN > License > Nodes > Add DR
      or

    2. Jobs/Errors > Jobs > filter on 6.3.1-DR

    Backend Logs

    On both the Primary and Secondary nodes, use the cat command to view the backend logs (/opt/phoenix/log/phoenix.log):

    cat /opt/phoenix/config/phoenix.log

    Successful Enablement of Disaster Recovery of the Primary node

    2021-08-20T16:26:22.186396-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7398,[phLogDetail]=631-DR, Step 1: check command type

    2021-08-20T16:26:22.186451-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7407,[phLogDetail]=631-DR, Step 2: check command data

    2021-08-20T16:26:22.186475-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7414,[phLogDetail]=631-DR, Step 3: load replication setting

    2021-08-20T16:26:22.222539-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7444,[phLogDetail]=631-DR-init-primary, Step 1: Saving Disaster Recovery Settings on Primary

    2021-08-20T16:26:22.252896-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7457,[phLogDetail]=631-DR-init-primary, Step 2: Saving peer SSH public key on Primary

    2021-08-20T16:26:22.291143-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7471,[phLogDetail]=631-DR-init-primary, Step 3: Updating SSH Configuration on Primary

    2021-08-20T16:26:22.328987-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7486,[phLogDetail]=631-DR-init-primary, Step 4: Updating SSH Known Hosts on Primary

    2021-08-20T16:26:22.503675-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7508,[phLogDetail]=631-DR-init-primary, Step 5: Initializing Disaster Recovery

    2021-08-20T16:26:22.577513-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7567,[phLogDetail]=631-DR-init-primary, Step 5.1: Running phinitprimary script on Primary

    2021-08-20T16:26:23.521939-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7582,[phLogDetail]=631-DR-init-primary, Step 5.2: Sending Task to Secondary

    2021-08-20T16:26:23.579504-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7619,[phLogDetail]=631-DR-init-primary, Step 5.3: Initializing Secondary

    2021-08-20T16:26:23.620795-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 1

    2021-08-20T16:26:23.644367-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 20

    2021-08-20T16:26:23.671393-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 30

    2021-08-20T16:26:23.709927-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 40

    2021-08-20T16:26:23.740996-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 50

    2021-08-20T16:29:01.385371-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 80

    2021-08-20T16:29:01.454035-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7634,[phLogDetail]=631-DR-init-primary, Step 5.5: Secondary Setup complete

    2021-08-20T16:29:01.454250-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7515,[phLogDetail]=631-DR-init-primary, Step 6: Waiting for App Server to come back up on Primary

    2021-08-20T16:29:01.509276-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7523,[phLogDetail]=631-DR-init-primary, Step 7: Waiting for Secondary App Server to update Service Passwords

    2021-08-20T16:29:01.797224-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8187,[phLogDetail]=631-DR-init-primary, Step 7.1: get sevice user

    2021-08-20T16:29:01.797257-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8195,[phLogDetail]=631-DR-init-primary, Step 7.2: get secondary host

    2021-08-20T16:29:01.797440-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8214,[phLogDetail]=631-DR-init-primary, Step 7.3: send task updating Svc password to secondary

    2021-08-20T16:29:01.882229-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8224,[phLogDetail]=631-DR-init-primary, Step 7.4: finish updating Svc password on secondary

    2021-08-20T16:29:01.884157-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7541,[phLogDetail]=631-DR-init-primary, Step 8: Restarting processes on Primary

    2021-08-20T16:29:01.923315-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7549,[phLogDetail]=631-DR-init-primary, Step 9: Broadcasting DR tasks to all processes on Primary

    2021-08-20T16:29:01.962740-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7559,[phLogDetail]=631-DR-init-primary, Step 10: Disaster Recovery setup complete

    2021-08-20T16:29:02.169745-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7432,[phLogDetail]=631-DR, Step 4: initializing DR finished.

    2021-08-20T17:03:54.103262-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7712,[phLogDetail]=631-DR-init-secondary, Step 1: Initalize secondary From Primary server

    2021-08-20T17:03:54.103449-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7733,[phLogDetail]=631-DR-init-secondary, Step 2: parse task info

    2021-08-20T17:03:54.103689-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7757,[phLogDetail]=631-DR-init-secondary, Step 3: update SSH keys of primary on secondary

    2021-08-20T17:03:54.115290-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7775,[phLogDetail]=631-DR-init-secondary, Step 4: update SSH configurations on secondary

    2021-08-20T17:03:54.126466-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7793,[phLogDetail]=631-DR-init-secondary, Step 5: update SSH known hosts for primary on secondary

    2021-08-20T17:03:54.148876-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7817,[phLogDetail]=631-DR-init-secondary, Step 6: run phinitsecondary script

    2021-08-20T17:06:29.865977-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7881,[phLogDetail]=631-DR-init-secondary, Step 10: finish initialize secondary

    Successful Enablement of Disaster Recovery of the Secondary node

    2021-08-20T16:29:29.146469-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7712,[phLogDetail]=631-DR-init-secondary, Step 1: Initalize secondary From Primary server

    2021-08-20T16:29:29.146829-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7733,[phLogDetail]=631-DR-init-secondary, Step 2: parse task info

    2021-08-20T16:29:29.147537-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7757,[phLogDetail]=631-DR-init-secondary, Step 3: update SSH keys of primary on secondary

    2021-08-20T16:29:29.160266-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7775,[phLogDetail]=631-DR-init-secondary, Step 4: update SSH configurations on secondary

    2021-08-20T16:29:29.172964-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7793,[phLogDetail]=631-DR-init-secondary, Step 5: update SSH known hosts for primary on secondary

    2021-08-20T16:29:29.287039-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7817,[phLogDetail]=631-DR-init-secondary, Step 6: run phinitsecondary script

    2021-08-20T16:32:06.952364-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7881,[phLogDetail]=631-DR-init-secondary, Step 10: finish initialize secondary

    2021-08-20T17:06:58.153075-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7398,[phLogDetail]=631-DR, Step 1: check command type

    2021-08-20T17:06:58.153099-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7407,[phLogDetail]=631-DR, Step 2: check command data

    2021-08-20T17:06:58.153113-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7414,[phLogDetail]=631-DR, Step 3: load replication setting

    2021-08-20T17:06:58.188916-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7444,[phLogDetail]=631-DR-init-primary, Step 1: Saving Disaster Recovery Settings on Primary

    2021-08-20T17:06:58.217059-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7457,[phLogDetail]=631-DR-init-primary, Step 2: Saving peer SSH public key on Primary

    2021-08-20T17:06:58.260626-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7471,[phLogDetail]=631-DR-init-primary, Step 3: Updating SSH Configuration on Primary

    2021-08-20T17:06:58.306178-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7486,[phLogDetail]=631-DR-init-primary, Step 4: Updating SSH Known Hosts on Primary

    2021-08-20T17:06:58.353953-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7508,[phLogDetail]=631-DR-init-primary, Step 5: Initializing Disaster Recovery

    2021-08-20T17:06:58.422293-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7567,[phLogDetail]=631-DR-init-primary, Step 5.1: Running phinitprimary script on Primary

    2021-08-20T17:06:59.039355-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7582,[phLogDetail]=631-DR-init-primary, Step 5.2: Sending Task to Secondary

    2021-08-20T17:06:59.144875-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7619,[phLogDetail]=631-DR-init-primary, Step 5.3: Initializing Secondary

    2021-08-20T17:06:59.188592-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 1

    2021-08-20T17:06:59.220850-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 20

    2021-08-20T17:06:59.250523-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 30

    2021-08-20T17:06:59.284640-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 40

    2021-08-20T17:06:59.310058-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 50

    2021-08-20T17:09:34.909429-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 80

    2021-08-20T17:09:34.987417-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7634,[phLogDetail]=631-DR-init-primary, Step 5.5: Secondary Setup complete

    2021-08-20T17:09:34.987610-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7515,[phLogDetail]=631-DR-init-primary, Step 6: Waiting for App Server to come back up on Primary

    2021-08-20T17:09:35.042360-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7523,[phLogDetail]=631-DR-init-primary, Step 7: Waiting for Secondary App Server to update Service Passwords

    2021-08-20T17:09:35.106214-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8187,[phLogDetail]=631-DR-init-primary, Step 7.1: get sevice user

    2021-08-20T17:09:35.106241-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8195,[phLogDetail]=631-DR-init-primary, Step 7.2: get secondary host

    2021-08-20T17:09:35.106352-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8214,[phLogDetail]=631-DR-init-primary, Step 7.3: send task updating Svc password to secondary

    2021-08-20T17:09:35.180197-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8224,[phLogDetail]=631-DR-init-primary, Step 7.4: finish updating Svc password on secondary

    2021-08-20T17:09:35.182966-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7541,[phLogDetail]=631-DR-init-primary, Step 8: Restarting processes on Primary

    2021-08-20T17:09:35.222909-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7549,[phLogDetail]=631-DR-init-primary, Step 9: Broadcasting DR tasks to all processes on Primary

    2021-08-20T17:09:35.253335-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7559,[phLogDetail]=631-DR-init-primary, Step 10: Disaster Recovery setup complete

    2021-08-20T17:09:35.276445-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7432,[phLogDetail]=631-DR, Step 4: initializing DR finished.

    Failure to Connect to Secondary

    If the message "Failed to connect secondary" appears, after enabling Disaster Recovery, try restarting phMonitor on the Secondary.

    Backend Processes in DR Device are Down

    If the backend processes in DR device are down and the response code of RESP APIs in /etc/httpd/logs/ssl_access_log is 401, then take the following steps.

    1. Get the new service password in the Primary server by running the following command:

      phLicenseTool --showServicePassword

    2. Send Rest API by Curl to the DR App server to update the password in the license file by running the following:

      curl -X PUT -u '1:<ServicePassword>' -H "Content-Type: text/xml" -d '<request><password>ServicePassword</password></request>' -k https://<DR_SUPER_IP>:443/phoenix/rest/custMgmt/service/changePwd

    Previous
    Next

    Troubleshooting Disaster Recovery

    Troubleshooting Disaster Recovery

    GUI

    While Replication is being set up, you can see the progress in the GUI in two locations.

    1. ADMIN > License > Nodes > Add DR
      or

    2. Jobs/Errors > Jobs > filter on 6.3.1-DR

    Backend Logs

    On both the Primary and Secondary nodes, use the cat command to view the backend logs (/opt/phoenix/log/phoenix.log):

    cat /opt/phoenix/config/phoenix.log

    Successful Enablement of Disaster Recovery of the Primary node

    2021-08-20T16:26:22.186396-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7398,[phLogDetail]=631-DR, Step 1: check command type

    2021-08-20T16:26:22.186451-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7407,[phLogDetail]=631-DR, Step 2: check command data

    2021-08-20T16:26:22.186475-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7414,[phLogDetail]=631-DR, Step 3: load replication setting

    2021-08-20T16:26:22.222539-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7444,[phLogDetail]=631-DR-init-primary, Step 1: Saving Disaster Recovery Settings on Primary

    2021-08-20T16:26:22.252896-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7457,[phLogDetail]=631-DR-init-primary, Step 2: Saving peer SSH public key on Primary

    2021-08-20T16:26:22.291143-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7471,[phLogDetail]=631-DR-init-primary, Step 3: Updating SSH Configuration on Primary

    2021-08-20T16:26:22.328987-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7486,[phLogDetail]=631-DR-init-primary, Step 4: Updating SSH Known Hosts on Primary

    2021-08-20T16:26:22.503675-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7508,[phLogDetail]=631-DR-init-primary, Step 5: Initializing Disaster Recovery

    2021-08-20T16:26:22.577513-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7567,[phLogDetail]=631-DR-init-primary, Step 5.1: Running phinitprimary script on Primary

    2021-08-20T16:26:23.521939-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7582,[phLogDetail]=631-DR-init-primary, Step 5.2: Sending Task to Secondary

    2021-08-20T16:26:23.579504-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7619,[phLogDetail]=631-DR-init-primary, Step 5.3: Initializing Secondary

    2021-08-20T16:26:23.620795-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 1

    2021-08-20T16:26:23.644367-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 20

    2021-08-20T16:26:23.671393-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 30

    2021-08-20T16:26:23.709927-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 40

    2021-08-20T16:26:23.740996-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 50

    2021-08-20T16:29:01.385371-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 80

    2021-08-20T16:29:01.454035-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7634,[phLogDetail]=631-DR-init-primary, Step 5.5: Secondary Setup complete

    2021-08-20T16:29:01.454250-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7515,[phLogDetail]=631-DR-init-primary, Step 6: Waiting for App Server to come back up on Primary

    2021-08-20T16:29:01.509276-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7523,[phLogDetail]=631-DR-init-primary, Step 7: Waiting for Secondary App Server to update Service Passwords

    2021-08-20T16:29:01.797224-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8187,[phLogDetail]=631-DR-init-primary, Step 7.1: get sevice user

    2021-08-20T16:29:01.797257-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8195,[phLogDetail]=631-DR-init-primary, Step 7.2: get secondary host

    2021-08-20T16:29:01.797440-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8214,[phLogDetail]=631-DR-init-primary, Step 7.3: send task updating Svc password to secondary

    2021-08-20T16:29:01.882229-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8224,[phLogDetail]=631-DR-init-primary, Step 7.4: finish updating Svc password on secondary

    2021-08-20T16:29:01.884157-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7541,[phLogDetail]=631-DR-init-primary, Step 8: Restarting processes on Primary

    2021-08-20T16:29:01.923315-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7549,[phLogDetail]=631-DR-init-primary, Step 9: Broadcasting DR tasks to all processes on Primary

    2021-08-20T16:29:01.962740-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7559,[phLogDetail]=631-DR-init-primary, Step 10: Disaster Recovery setup complete

    2021-08-20T16:29:02.169745-07:00 va3005 phMonitorSupervisor[11908]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7432,[phLogDetail]=631-DR, Step 4: initializing DR finished.

    2021-08-20T17:03:54.103262-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7712,[phLogDetail]=631-DR-init-secondary, Step 1: Initalize secondary From Primary server

    2021-08-20T17:03:54.103449-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7733,[phLogDetail]=631-DR-init-secondary, Step 2: parse task info

    2021-08-20T17:03:54.103689-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7757,[phLogDetail]=631-DR-init-secondary, Step 3: update SSH keys of primary on secondary

    2021-08-20T17:03:54.115290-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7775,[phLogDetail]=631-DR-init-secondary, Step 4: update SSH configurations on secondary

    2021-08-20T17:03:54.126466-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7793,[phLogDetail]=631-DR-init-secondary, Step 5: update SSH known hosts for primary on secondary

    2021-08-20T17:03:54.148876-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7817,[phLogDetail]=631-DR-init-secondary, Step 6: run phinitsecondary script

    2021-08-20T17:06:29.865977-07:00 va3005 phMonitorSupervisor[7384]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7881,[phLogDetail]=631-DR-init-secondary, Step 10: finish initialize secondary

    Successful Enablement of Disaster Recovery of the Secondary node

    2021-08-20T16:29:29.146469-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7712,[phLogDetail]=631-DR-init-secondary, Step 1: Initalize secondary From Primary server

    2021-08-20T16:29:29.146829-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7733,[phLogDetail]=631-DR-init-secondary, Step 2: parse task info

    2021-08-20T16:29:29.147537-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7757,[phLogDetail]=631-DR-init-secondary, Step 3: update SSH keys of primary on secondary

    2021-08-20T16:29:29.160266-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7775,[phLogDetail]=631-DR-init-secondary, Step 4: update SSH configurations on secondary

    2021-08-20T16:29:29.172964-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7793,[phLogDetail]=631-DR-init-secondary, Step 5: update SSH known hosts for primary on secondary

    2021-08-20T16:29:29.287039-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7817,[phLogDetail]=631-DR-init-secondary, Step 6: run phinitsecondary script

    2021-08-20T16:32:06.952364-07:00 andy-super-ipv6 phMonitorSupervisor[11936]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7881,[phLogDetail]=631-DR-init-secondary, Step 10: finish initialize secondary

    2021-08-20T17:06:58.153075-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7398,[phLogDetail]=631-DR, Step 1: check command type

    2021-08-20T17:06:58.153099-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7407,[phLogDetail]=631-DR, Step 2: check command data

    2021-08-20T17:06:58.153113-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7414,[phLogDetail]=631-DR, Step 3: load replication setting

    2021-08-20T17:06:58.188916-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7444,[phLogDetail]=631-DR-init-primary, Step 1: Saving Disaster Recovery Settings on Primary

    2021-08-20T17:06:58.217059-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7457,[phLogDetail]=631-DR-init-primary, Step 2: Saving peer SSH public key on Primary

    2021-08-20T17:06:58.260626-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7471,[phLogDetail]=631-DR-init-primary, Step 3: Updating SSH Configuration on Primary

    2021-08-20T17:06:58.306178-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7486,[phLogDetail]=631-DR-init-primary, Step 4: Updating SSH Known Hosts on Primary

    2021-08-20T17:06:58.353953-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7508,[phLogDetail]=631-DR-init-primary, Step 5: Initializing Disaster Recovery

    2021-08-20T17:06:58.422293-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7567,[phLogDetail]=631-DR-init-primary, Step 5.1: Running phinitprimary script on Primary

    2021-08-20T17:06:59.039355-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7582,[phLogDetail]=631-DR-init-primary, Step 5.2: Sending Task to Secondary

    2021-08-20T17:06:59.144875-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7619,[phLogDetail]=631-DR-init-primary, Step 5.3: Initializing Secondary

    2021-08-20T17:06:59.188592-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 1

    2021-08-20T17:06:59.220850-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 20

    2021-08-20T17:06:59.250523-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 30

    2021-08-20T17:06:59.284640-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 40

    2021-08-20T17:06:59.310058-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 50

    2021-08-20T17:09:34.909429-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7629,[phLogDetail]=631-DR-init-primary, Step 5.4: Setting up Secondary: 80

    2021-08-20T17:09:34.987417-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7634,[phLogDetail]=631-DR-init-primary, Step 5.5: Secondary Setup complete

    2021-08-20T17:09:34.987610-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7515,[phLogDetail]=631-DR-init-primary, Step 6: Waiting for App Server to come back up on Primary

    2021-08-20T17:09:35.042360-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7523,[phLogDetail]=631-DR-init-primary, Step 7: Waiting for Secondary App Server to update Service Passwords

    2021-08-20T17:09:35.106214-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8187,[phLogDetail]=631-DR-init-primary, Step 7.1: get sevice user

    2021-08-20T17:09:35.106241-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8195,[phLogDetail]=631-DR-init-primary, Step 7.2: get secondary host

    2021-08-20T17:09:35.106352-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8214,[phLogDetail]=631-DR-init-primary, Step 7.3: send task updating Svc password to secondary

    2021-08-20T17:09:35.180197-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=8224,[phLogDetail]=631-DR-init-primary, Step 7.4: finish updating Svc password on secondary

    2021-08-20T17:09:35.182966-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7541,[phLogDetail]=631-DR-init-primary, Step 8: Restarting processes on Primary

    2021-08-20T17:09:35.222909-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7549,[phLogDetail]=631-DR-init-primary, Step 9: Broadcasting DR tasks to all processes on Primary

    2021-08-20T17:09:35.253335-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7559,[phLogDetail]=631-DR-init-primary, Step 10: Disaster Recovery setup complete

    2021-08-20T17:09:35.276445-07:00 andy-super-ipv6 phMonitorSupervisor[38314]: [PH_GENERIC_INFO]:[eventSeverity]=PHL_INFO,[procName]=phMonitorSupervisor,[fileName]=phMonitorProcess.cpp,[lineNumber]=7432,[phLogDetail]=631-DR, Step 4: initializing DR finished.

    Failure to Connect to Secondary

    If the message "Failed to connect secondary" appears, after enabling Disaster Recovery, try restarting phMonitor on the Secondary.

    Backend Processes in DR Device are Down

    If the backend processes in DR device are down and the response code of RESP APIs in /etc/httpd/logs/ssl_access_log is 401, then take the following steps.

    1. Get the new service password in the Primary server by running the following command:

      phLicenseTool --showServicePassword

    2. Send Rest API by Curl to the DR App server to update the password in the license file by running the following:

      curl -X PUT -u '1:<ServicePassword>' -H "Content-Type: text/xml" -d '<request><password>ServicePassword</password></request>' -k https://<DR_SUPER_IP>:443/phoenix/rest/custMgmt/service/changePwd

    Previous
    Next