For troubleshooting the distributed managed service provider model for multi-tenancy issues that you might face, you can use the
postman.log located at:
You can change the logging levels for the postman.log by using the following command:
# vi /opt/cyops-routing-agent/postman/config.ini and set the required logging level. By default, the logging level is set to WARN.
You can set the following logging levels in the log files:
- DEBUG: Low-level system information for debugging purposes.
- INFO: General system information.
- WARN: Information describing a minor problem that has occurred.
- ERROR: Information describing a major problem that has occurred.
- CRITICAL: Information describing a critical problem that has occurred.
You can continue to retry to connect to the secure message exchange for 10 minutes. A long retry has been specified to ensure that you do not have to restart the service every time the network breaks since all consumers stop after a long network break.
You can configure the time for which you will continue retrying to establish a connection with the secure message exchange using the
config file, located at
heartbeat = 120 connection_attempts = 50 retry_delay = 10
This issue can occur due to two reasons:
- Incorrect secure message exchange configuration.
- Incorrect selection of secure message exchange while configuring the tenant on the master node.
Correct the secure message exchange configuration or selection and click the Retry button.
This issue can occur if you have specified the wrong password or port while configuring the master on the tenant node.
On the tenant node, click Master Configuration > Edit Configuration. On the
Configure Master dialog and correct the password or TCP port number that you have specified.
If you yet see that the Enabled button is in the NO state, and you want to enable data replication, toggle the Enabled button to YES.
Records created at the tenant node do not replicate to the master node even when data replication is turned on
In this case, the FortiSOAR logs contain the following message:
Error message on master "Error: Bad Request for url: <URL> when a field is non-mandatory on tenant and mandatory on master
This issue occurs if you have switched off replication of a required record field on a tenant node, or the field does not exist on the same module defined at the tenant node. This leads to a failure of record creation on the master node.
You must ensure that replication is switched on for all required record fields, and you should also note that if you have done any schema changes in a module at the master node, then you must ensure that the required fields are marked as required across all tenants that are replicating that module.
Any user who is created in a multi-tenant environment and who requires to view records must have a minimum of read-only access for the
Tenants module in their respective role.
Therefore, ensure that you have assigned the user a role that has a minimum of read-only access for the
You might not see any modules of a dedicated tenant node at the master node in Tenant Manager > Manage Modules after the initial configuration (addition) of the tenant node if there are errors while configuring the tenant.
This issue occurs due to the tenant being in the "Verification Failed" state at the time of configuration. To make the MMD of the tenant visible at the master node, restart the "cyops-postman" service at the tenant node.
In case of clustering of secure message exchanges, FortiSOAR is unable to connect to primary secure message exchange even after the primary node has come back online after a failure
If you have your secure message exchanges setup as a cluster for high availability, and you face a failure, and the primary secure message exchange node comes back online and yet the FortiSOAR node is still not able to connect to the secure message exchange. You will see the following error in the postman logs:
2018-11-20 10:36:21,022 140437372753664 59b195ab94e35ef70e28ed129c58c804 ERROR pika.callback callback process(): Calling <bound method BlockingChannel._on_channel_closed of <BlockingChannel impl=<Channel number=1 CLOSED conn=<SelectConnection OPEN socket=(‘<xxx.xxx.xx.xxx’, 53378)->(‘xxx.xxx.xx.xxx’, 52011) params=>>>> for “1:Channel.Close” failed Traceback (most recent call last): File “/opt/cyops-routing-agent/.env/lib/python3.4/site-packages/pika/callback.py”, line 236, in process callback(*args, **keywords) File “/opt/cyops-routing-agent/.env/lib/python3.4/site-packages/pika/adapters/blocking_connection.py”, line 1358, in _on_channel_closed method.reply_text) pika.exceptions.ChannelClosed: (404, “NOT_FOUND - home node ‘rabbit@’ of durable queue ‘queue.postman.data.remoterequest.805553bfece2f6a5895c8db8c54b9ae0’ in vhost ‘vhost_59b195ab94e35ef70e28ed129c58c804’ is down or inaccessible”)
Stop and start the rabbitmq app on the secure message exchange node using the following commands:
# rabbitmqctl stop_app
# rabbitmqctl start_app
Shifting the Secure Message Exchange of tenants leads to MMDs not being pushed to the tenants that have been shifted to the new Secure Message Exchange
If you have shifted the Secure Message Exchange of any tenant in your multi-tenant configuration to a new Secure Message Exchange, then the remote mmd management does not work since the
uwsgi service keeps the Secure Message Exchange details in memory and uses it to publish to the Secure Message Exchange. However, when the Secure Message Exchange is changed for any tenant the change gets notified using "rabbitmq publish" and the updated Secure Message Exchange setting is available only to the
postman service; the
uwsgi still has old Secure Message Exchange details.
After you have shifted any tenant to a new Secure Message Exchange, then you must update the router settings in the
uwsgi service by restarting the
uwsgi services on the master node using the following commands:
# systemctl restart uwsgi
# systemctl restart cyops-postman
You also need to restart the
postman service on the tenant node.
Tenant with basic authentication is stuck in the "Awaiting Remote Node Connection" state when a user removes and re-adds certificates
If a user deletes the certificate or key file by mistake and then re-add the certificate on an 'Agent', the status of the agent changes to "Configuration Failed" and then to "Awaiting Remote Node Connection". In this case if the user configures multitenancy again by exporting the latest master configuration from master and importing the configuration on a tenant node, then the record of the tenant on master might remain stuck in the "Awaiting Remote Node Connection"; however, the master configuration on the tenant node shows that the remote node is connected.
- Open the Tenant node and navigate to Setting > Master Configuration.
- Disable and again enable the master configuration. This updates the state of the tenant record on master and displays the correct "Remote Node Connected" state.
Once you have updated your MSSP setup to 7.3.0 or later from a release prior to 7.3.0, the status of the tenant displays "
Remote Node Unreachable"
cyops-integrations-agent service on the tenant node.