Troubleshooting FortiSOAR Issues

Troubleshooting issues occurring in FortiSOAR due to insufficient space

You can face deployment or upgrade failures due to insufficient space. If you have limited partition size for /dev/mapper/vgapp-csapps, then FortiSOAR upgrade might fail. Therefore, before you upgrade your FortiSOAR system, you must ensure that you increase the partition size to a minimum of 4 GB for /dev/mapper/vgapp-csapps to prevent potential loss of backups.

You might also experience any of the following symptoms when the disk space of the database on which FortiSOAR is running gets full:

• Users are unable to log into FortiSOAR.
• All FortiSOAR services might stop working, as they cannot write to their respective log files.
  For example, the PostgreSQL service fails to start when the PostgreSQL database disk is full.

Insufficient space in FortiSOAR can be caused due to a number of reasons, some of them are as follows:

• Increase in the number of log files in /var/log/ and /var/log/cyops
• /home drive is full
• Increase in the data in the database
• PostgreSQL database disk is full

You can fix the insufficient space issue using the csadm system disk expand-lv command to extend a logical volume to occupy space that is available in its own volume group or if a new disk is attached, then a single partition is created and the logical volume is expanded to occupy that partition based on the size (GB) you have specified. For additional information on this command, see the FortiSOAR Admin CLI chapter in the "Administration Guide."

You can also use the following methods to fix this issue:

• Resolution 1: Extend the disk space by adding a new disk and then extending the logical volume in the new disk.
  NOTE: To support disk sizes larger than 2 TB, FortiSOAR OVAs starting with the 7.5.0 release come pre-configured with a GPT-based disk layout. Previously, FortiSOAR OVAs were shipped with an MBR-based disk layout, which limited disk management to a size of 2TB. If you already have a FortiSOAR instance and need a partition larger than 2 TB, we recommend creating a new FortiSOAR VM on release 7.5.0 or later and utilizing the Export and Import wizards to migrate your data from the old instance to the new one. This is required as FortiSOAR does not support a combination of MBR and GPT partitions.
• Resolution 2: Extend the logical volume by using the free space that is already available in the volume group.
• Resolution 3: Extend the logical volume on the existing disk without adding a new disk, if sufficient unallocated space is available on the existing disk.
• Resolution 4: Procedure to be followed when the Postgres service has failed which could be due to the PostgreSQL database disk is full.
  

Note: Perform the following procedures as a root user using the sudo su command.

Resolution 1

Perform the following steps to extend your disk space by adding a new disk and then extending the logical volume (LVM) in the new disk.

1. Stop all FortiSOAR services using the following command:
   # csadm services --stop
2. Add the new disk drive with the required size on the instance.
3. Run the following command to check the size of the newly added and unpartitioned disk.
   # lsblk
   This command displays the size of newly added and unpartitioned disk. In this example, sde is the newly added disk:
   NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 8G 0 disk
'-sda1 8:1 0 8G 0 part /
sdb 8:16 0 100G 0 disk
'-sdb1 8:17 0 100G 0 part
sdc 8:32 0 300G 0 disk
'-sdc1 8:33 0 300G 0 part
'-vg_repo-lvol0 253:0 0 500G 0 lvm /repos
sdd 8:48 0 200G 0 disk
'-vg_repo-lvol0 253:0 0 500G 0 lvm /repos
sde 8:64 0 200G 0 disk #This is the new attached partition
   sr0 11:0 1 1024M 0 rom
   
4. Use the csadm system disk expand-lv command to extend a logical volume by the specified size. It is recommended that you use the --validate option to validate the inputs you have passed to the csadm system disk expand-lv command. For example:
   # csadm system disk expand-lv --validate --logical-volume <LVM_name> --disk <disk_name> --size <disk_size>
   The --validate option provides a summary of changes that will be made after running this command. This summary displays the current lvm size, the current free space on the disk, and the expected lvm size following the execution of the command, and also displays an appropriate error if the requested disk space for expansion is less than the free space that is available.
   The --size option is optional and must be specified in GBs; if not specified then the whole disk will be attached to that logical volume.
   For example, if you want to expand the logical volume, pgsql, by 10 GB, you can use the following command with the validate option:
   # csadm system disk expand-lv --validate --logical-volume pgsql --disk sda --size 10
   If the summary displayed is acceptable, then run the command without the --validate option:
   # csadm system disk expand-lv --logical-volume pgsql --disk sda --size 10
   Note: You must extend the disk size to less than or equal to the total size of the volume group.
5. Run the following command to check if the volume size is extended:
   # df -h /dev/mapper/<LVM_name>
6. Start all FortiSOAR services using the following command:
   # csadm services --start

Resolution 2

Perform the following steps to extend the logical volume by using the free space that is already available in the volume group:

1. Stop all FortiSOAR services using the following command:
   # csadm services --stop
2. Use the csadm system disk expand-lv command to extend a logical volume by the specified size. It is recommended that you use the --validate option to validate the inputs you have passed to the csadm system disk expand-lv command. For example:
   # csadm system disk expand-lv --validate --logical-volume <LVM_name> --use-vg --size <disk_size>
   The --validate option provides a summary of changes that will be made after running this command. This summary displays the current lvm size, the current free space on the disk, and the expected lvm size following the execution of the command, and also displays an appropriate error if the requested disk space for expansion is less than the free space that is available.
   The --size option is optional and must be specified in GBs; if not specified then the all the free space in the volume group will be assigned to that logical volume.
   For example, if you want to expand the logical volume, pgsql, by 10 GB, you can use the following command with the validate option:
   # csadm system disk expand-lv --validate --logical-volume pgsql --use-vg --size 10
   If the summary displayed is acceptable, then run the command without the --validate option:
   # csadm system disk expand-lv --logical-volume pgsql --use-vg --size 10
   Note: You must extend the disk size to less than or equal to the total size of the volume group.
3. Run the following command to check if the volume size is extended:
   # df -h /dev/mapper/<LVM_name>
4. Start all FortiSOAR services using the following command:
   # csadm services --start

Resolution 3

Perform the following steps to extend the logical volume on the existing disk without adding a new disk, if sufficient unallocated space is available on the existing disk. Before proceeding further, it is recommended that you find out whether sufficient unallocated space is available on existing disk using the following command:
# parted /dev/<disk_name> print free

For example, running the # parted /dev/sdb print free command will display the following:

Model: VMware Virtual disk (scsi)
	Disk /dev/sdb: 215GB
	Sector size (logical/physical): 512B/512B
	Partition Table: msdos
	Disk Flags:
	Number  Start   End     Size    Type     File system  Flags

	32.3kB  1049kB  1016kB  Free Space
	1      1049kB  161GB   161GB   primary               lvm
	161GB   215GB   53.7GB  Free Space

The “Free Space” in above the output refers to unallocated space. If the mentioned size is sufficient, then perform the following steps:

1. Stop all FortiSOAR services using the following command:
   # csadm services --stop
2. Create a partition on the existing disk using the following command:
   # fdisk /dev/<disk_name>
3. Enter n to create a new partition.
4. Enter p to choose the newly created partition as the primary partition.
5. Select the Partition number. The operating system will calculate this on its own, you just require to press Enter.
6. Select defaults for the First Cylinder/Sector and Last Cylinder/Sector values.
7. Enter w to write the changes to the partition table.
8. Run the following command to rewrite the filesystem:
   # partprobe /dev/<disk_name>
9. After the VM restarts, stop all FortiSOAR services using the following command:
   # csadm services --stop
10. Run the following command to see the new partition:
    # fdisk -l /dev/<disk_name>
    This command displays the properties of the newly attached partition. For this example, /dev/sda3 is the newly added partition.
11. Create a physical volume in the new partition using the following command:
    # pvcreate
    For example, # pvcreate /dev/sda3
12. Extend the volume group using the following command:
    # vgextend
    For example, # vgextend <volume_group_name> /dev/sda3
13. Extend the logical volume by the size you specify using the following command:
    # lvextend -L+<size>G <LVM_name>
    For example, to extend the logical volume by 20 G use the following command:
    # lvextend -L+20G /dev/mapper/cyops-relations
    Alternatively, you can also use the csadm system disk expand-lv command to extend a logical volume by the specified size. It is recommended that you use the --validate option to validate the inputs you have passed to the csadm system disk expand-lv command. For example:
    # csadm system disk expand-lv --validate --logical-volume <LVM_name> --use-vg --size <disk_size>
    The --validate option provides a summary of changes that will be made after running this command. This summary displays the current lvm size, the current free space on the disk, and the expected lvm size following the execution of the command, and also displays an appropriate error if the requested disk space for expansion is less than the free space that is available.
    For example, if you want to expand the logical volume, pgsql, by 10 GB, you can use the following command with the validate option:
    # csadm system disk expand-lv --validate --logical-volume pgsql --use-vg --size 10
    If the summary displayed is acceptable, then run the command without the --validate option:
    # csadm system disk expand-lv --logical-volume pgsql --use-vg --size 10
14. Increase the file system size using the following command:
    # xfs_growfs
    For example, # xfs_growfs /dev/mapper/cyops-relations
15. Start all FortiSOAR services using the following command:
    # csadm services --start

Resolution 4

Perform the following steps when you notice that your Postgres service has failed which could be due to the PostgreSQL database disk is full.

1. Run the following command and check the disk space in /var/lib/pgsql/:
   # df -h /var/lib/pgsql/
   If there is no space or less than 1 GB space left in /var/lib/pgsql/, then the psql service will not start due to the space issue.
2. # cd /var/lib/pgsql/
# ls -lrth
   You will observe that there is a file named resv_space that is taking up 1GB of space.
3. Stop all the services using the following command:
   csadm services --stop
4. Move the resv_space file from /var/lib/pgsql/ to /home/csadmin using the following command:
   # mv /var/lib/pgsql/resv_space /home/csadmin
5. Check the disk space in /var/lib/pgsql/ again:
   # df -h /var/lib/pgsql/
   You will observe that 1GB of space has been freed.
   
6. Start all the services again using the following command:
   csadm services --start
7. If this is a development or staging instance and you cannot extend the disk space, then the disk space can also be freed up by purging workflow logs. For information on purging workflow logs, see the System Configuration chapter in the "Administration Guide."
8. Move the resv_space file from /home/csadmin back to /var/lib/pgsql/:
   # mv /home/csadmin/resv_space /var/lib/pgsql/

Increasing the disk space for record storage in case of AWS AMI deployment

If you are deploying a fresh instance of FortiSOAR in AWS with AMI, and you require larger disk space for record storage, do the following:

1. Increase the size Elastic and PostgreSQL disks, for example /dev/sdg (Elastic) and /dev/sdf (postgresql).
2. Provision your FortiSOAR instance and complete running the FortiSOAR VM Configuration Wizard.
3. Start an SSH session as a root user and check the allocated disk space using the df -h command.
4. If the newly increased disk space that is allocated is sufficient, then no changes are required, else you can use the csadm system command to increase the partition size and allocate unused space. For more information on the csadm command, see the FortiSOAR Admin CLI chapter in the "Administration Guide."

Troubleshooting Deployment Issues

The FortiSOAR Virtual Appliance deployment on ESX is failing

Resolution:

1. Verify that FortiSOAR Virtual Appliance file that you have downloaded is not corrupted by running a # md5sum command for the FortiSOAR Virtual Appliance.
2. Check that the ESX server has fulfilled all prerequisites specified for the VM. Refer to the Planning section for details.
3. If both points 1 and 2 are ok, contact VMWare support.

Cannot access the FortiSOAR portal

Resolution:

1. Check the ESX network to which FortiSOAR VM is connected.
2. Check if the IP address is assigned to your FortiSOAR VM, in the case of DHCP or static IP addresses.
   Refer to the Editing the VM configuration section for more information on Setting a static IP and Determining your DHCP IP address.

Cannot login to the FortiSOAR platform

Resolution:

1. Check if you are using the correct credentials that have been provided to you by FortiSOAR Customer Support.
2. ssh to the VM where you have deployed FortiSOAR to check the status of cyops-auth service. The cyops-auth service must be running.
3. If both points 1 and 2 are ok, and the cyops-auth service is running, contact FortiSOAR support.

Getting a 502 error when you click on the Reports tab

Resolution:

1. ssh to the VM where you have installed FortiSOAR.
2. Log in using the ssh credentials.
3. Run the $ sudo su command.
   Enter your FortiSOAR password.
4. Run the $ sudo systemctl restart tomcat command.
5. Run the $ sudo systemctl restart nginx command.

If the issue yet does not get resolved, contact FortiSOAR support.

Troubleshooting Upgrade Issues

For the procedure on how to upgrade to FortiSOAR 7.5.0, see the Upgrading a FortiSOAR enterprise instance to 7.5.0 section in the “Upgrade Guide.”

Post license renewal you cannot log into FortiSOAR

If you have requested for a license from FortiSOAR with lesser number of users than your existing users, you cannot log onto FortiSOAR post upgrade.

For example, when you had requested a license from FortiSOAR, you had requested for a license for 10 users; however, you have 15 users existing in your system, you will not be able to log onto FortiSOAR post-upgrade. So, it is very important for you to provide the correct number of users while requesting for a license from FortiSOAR.

Resolution:

Contact FortiSOAR Support to generate a new license for you with the correct number of users.

Failure to upgrade FortiSOAR

In case you face a failure while trying to upgrade FortiSOAR, then perform the following steps:

Resolution:

1. To gather logs and send them to FortiSOAR Support, do the following:
   1. ssh to machine as a root user and type the following command:
      # csadm log --collect
   2. Specify the path where you want to collect the logs. By default, the logs are collected in the /tmp/ folder.
      A file named fortisoar-logs.tar.gz.gpg gets generated in the path you have specified. Send this file to FortiSOAR Support.
2. Revert the snapshot of your system to the latest working state. You must take a snapshot of your system before you attempt to upgrade FortiSOAR on your system.
   Follow the steps mentioned in the documentation of your platform for taking a snapshot and reverting to the current snapshot.

Post-upgrade your playbooks fail to execute, and the playbooks are also not listed in the executed playbooks log

If you have not cleaned the workflow history prior to upgrading your FortiSOAR version, and if you have a large number of records in the workflow history (> 50000), then the overall upgrade time would increase, and this issue might occur. For the method to clean the workflow execution run history, see the "Setting up auto-cleanup of workflow execution history" topic in the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."

Resolution

Perform the following steps:

1. Check the install logs to see if errors such as the following are present:
   psycopg2.InternalError: missing chunk number 0 for toast value 1502832 in pg_toast_17046
2. Connect to the postgres db and run the following command:
   # REINDEX table pg_toast.pg_toast_XXXXX;
   (where XXXXX is the number from the error message present in step 1).
3. Run the following commands:
   # cd /opt/cyops-workflow/sealab
$ sudo -u nginx /opt/cyops-workflow/.env/bin/python3 manage.py migrate
   Note: If manage.py fails again for pg_toast value (error mentioned in step 1), then you must execute steps 2 and 3 again with the new pg_toast value mentioned in the error, till the manage.py executes successfully.

Login and logout events are not audited after you have upgraded your FortiSOAR version

After you have upgraded FortiSOAR on your system, you observe the following error in the auditlog log file located at /var/log/cyops/cyops-gateway/auditlog.log :

ERROR c.c.a.service.RecordLogService.processRecordLogs - 500 Internal Server Error, and you also do not see any Login and Logout events in Audit Logs on FortiSOAR UI, then perform the steps mentioned in the resolution.

Resolution

To resolve this issue and include login and login events in audit logs, run the following commands on your FortiSOAR VM as a root user:

yes | cp /opt/cyops-workflow/sealab/.envdir/APPLIANCE_PUBLIC_KEY /etc/cyops/APPLIANCE_PUBLIC_KEY
yes | cp /opt/cyops-workflow/sealab/.envdir/APPLIANCE_PRIVATE_KEY /etc/cyops/APPLIANCE_PRIVATE_KEY
chmod -R 644 /etc/cyops/APPLIANCE_*_KEY
chown tomcat:tomcat /etc/cyops/APPLIANCE_*_KEY
systemctl restart tomcat

Issues occurring when you have restored data on a FortiSOAR 6.0+ system with data backed up from a system prior to 6.0.0

If you have backed up a FortiSOAR system whose version is prior to 6.0.0 and restored this system on a fresh installation of FortiSOAR 6.0.0 or later, then you might face issues in creating records etc. since the system will not be able to find your record ID sequence.

Resolution

To keep your record ID sequence and avoid issues on the FortiSOAR 6.0+ instance, you must run the following commands as a root user to keep your record ID sequence:

1. In the /opt/cyops-api/config/parameters_prod.yaml file set the id_per_module parameter to false.
2. cd /opt/cyops-api
sudo -u nginx php bin/console cache:clear --env=prod