Merging Small Elasticsearch Indices into a Big Index

In Elasticsearch, you may see older indices with few documents. You may want to merge these smaller indices into a bigger index and create an alias for them, by following these steps.

Elasticsearch reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.13/docs-reindex.html

Notes:

1. Don't merge indices that belong to different organizations together.
2. The naming format for event index is: fortisiem-event-<Year>.<Month>.<Date>-<OrgId>-<SeqNo>
3. When merging indices from different days together, make sure to create aliases for the different days to point to the merged index

Steps

1. Create one new index.
   

   curl -XPUT '172.30.56.182:9200/fortisiem-event-2021.07.30-3-000001-merged?pretty' -H 'Content-Type: application/json' -d'
   {
       "settings" : {
           "index" : {
              "number_of_shards" : 1
           }
       }
   }
   '
   
   

2. Merge the smaller indices into the new index created in Step 1.
   

   curl -XPOST '172.30.56.182:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
   {
       "conflicts": "proceed",
       "source": {
           "index": "fortisiem-event-2021.07.30-3-000001,fortisiem-event-2021.07.29-3-000001"
       },
       "dest": {
           "index": "fortisiem-event-2021.07.30-3-000001-merged",
           "op_type": "create"
       }
   }
   '
   
   

3. Create aliases for all (newly created) merged indices.
   

   curl -X POST 'http://172.30.56.182:9200/_aliases' -H 'Content-Type: application/json' -d'
   {
      "actions":[
         {
            "add":{
               "index":"fortisiem-event-2021.07.30-3-000001-merged",
               "alias":"fortisiem-event-2021.07.30-3"
            }
         }
      ]
   }
   '
   
   

4. Delete all old indices.
   

   curl -XDELETE http://172.30.56.182:9200/fortisiem-event-2021.07.30-3-000001
   curl -XDELETE http://172.30.56.182:9200/fortisiem-event-2021.07.29-3-000001