Post-7.1.1 Upgrade ClickHouse IP Index Rebuilding

If you are upgrading ClickHouse based deployment from pre-7.1.1, then after upgrade has completed, you need to run the following command to rebuild ClickHouse indices. If you are running a version later than 7.1.1, and have already executed the rebuilding steps, then nothing more needs to be done.

/opt/phoenix/phscripts/clickhouse/rebuild_ip_bloom_filter.sh -h

Usage: ./script [-h for help] [-p for 10 oldest partitions] [-m manual commands] [-r <index id> index to be rebuilt]

Here are indexes available for rebuild. Please choose one and specify the id in -r option.

-r <index id> option is mandatory.

1. reptDevIpAddr
2. srcIpAddr
3. destIpAddr
4. hostIpAddr

For example, to rebuild reptDevIpAddr index, issue the following command.

$ ./rebuild_ip_bloom_filter.sh -r 1

Notes before running the script: The script should be run on only one node per shard. Only one instance of the script can be run at one time on a host. This is checked and enforced by the script. The rebuilding of bloom filter index is a table-wide operation and it could take some time to complete, based on the amount of data and resources available. Since the script could be long-running, please use screen or tmux sessions to launch the script. During the rebuilding of bloom filter index, there will be a temporary spike in storage usage as ClickHouse goes through the existing data and rebuilds indexes for the whole table. The rebuilding of bloom filter index is a non-intrusive operation. After the DROP, ADD, MATERIALIZE requests are submitted to ClickHouse, ClickHouse will schedule the various tasks in an optimized way. The user has no control over the process. All nodes in the target shard should have at least 10% free space on /data-clickhouse-hot-1. The script will also check the 10% space requirement on all nodes of the shard and suggests partitions that should be moved to a different tier to create space. If there are no more tiers to move the partitions to, please consult with Fortinet support to identify a reasonable solution. The script will show the remaining parts to be processed for both DROP and MATERIALIZE phase. If the "parts to be processed" is stuck at a specific count for more than 24 hours, please contact Fortinet support to check if there are any underlying conditions that would have prevented the DROP or MATERIALIZE command to proceed further.

Detailed Steps are as follows:

1. Go to root shell by running the following command:
   sudo -s
2. Run the following command:
   /opt/phoenix/phscripts/clickhouse/rebuild_ip_bloom_filter.sh -r 1
   The script will produce output similar to the following:

   [root@server]# /opt/phoenix/phscripts/clickhouse/rebuild_ip_bloom_filter.sh -r 1
   current attr is reptDevIpAddr
   Materialize in progress "false" for reptDevIpAddr with 0 parts to process
   Materialize in progress "false" for srcIpAddr with 0 parts to process
   Materialize in progress "false" for destIpAddr with 0 parts to process
   Materialize in progress "false" for hostIpAddr with 0 parts to process
   drop in progress "false" for reptDevIpAddr
   drop in progress "false" for srcIpAddr
   drop in progress "false" for destIpAddr
   drop in progress "false" for hostIpAddr
   Processing index index_reptDevIpAddr_bloom_filter
   Trying to rebuild index_reptDevIpAddr_bloom_filter
   drop in progress "false"
   There is enough storage on host 172.30.56.206.
   done checking storage for index index_reptDevIpAddr_bloom_filter
   dropping index index_reptDevIpAddr_bloom_filter
   dropping index command issued
   Start monitoring index index_reptDevIpAddr_bloom_filter for drop operation
   drop index_reptDevIpAddr_bloom_filter done. Total time taken: 0 seconds
   Total time to drop index_reptDevIpAddr_bloom_filter is 1 seconds
   adding index_reptDevIpAddr_bloom_filter
   Start monitoring index index_reptDevIpAddr_bloom_filter for materialize operation
   parts left to materialize reptDevIpAddr index : 6 - time taken 0 seconds
   materialize index_reptDevIpAddr_bloom_filter done. Total time taken: 10 seconds
   Rebuilding index for reptDevIpAddr done. Total time taken for rebuild 13 seconds
   

3. Run the following command:
   /opt/phoenix/phscripts/clickhouse/rebuild_ip_bloom_filter.sh -r 2
4. Run the following command:
   /opt/phoenix/phscripts/clickhouse/rebuild_ip_bloom_filter.sh -r 3
5. Run the following command:
   /opt/phoenix/phscripts/clickhouse/rebuild_ip_bloom_filter.sh -r 4