ELK Backup and Restore Operation

Introduction

Hi all, in this blog we are going to perform Elasticsearch backup and restore operation using Snapshots.

Pre-requisite:-

1. Install Elasticsearch and configuration

2. Install Kibana and its configuration

3. Install Logstash and create Logstash.conf file

   

If you are not aware about how to install above three components please watch video on YouTube regarding this.

Once you install and configure all above components you are ready to take Backup and restore operation using.

Make sure your Elasticsearch, Kibana and Logstash is up and running.

Snapshot Backup :-

Step1:-

1. Login into the Kibana portal and create repository.

2. To create repository go to the Management --> Stack Management --> Data --> Snapshot and Restore and then click and register a repository

3. Once repository in created integrate it with AWS S3. Open Elasticsearch bin folder and open command prompt.

4. Use this command and enter your access_key "elasticsearch-keystore add s3.client.default.access_key"

5. User this command and enter your secret_key "elasticsearch-keystore add s3.client.default.secret_key"

6. You need to add S3 plugin using the command "elasticsearch-plugin install repository-s3"

7. Once this done please restart your Elasticsearch service using command prompt.

Now connect to the Elasticsearch. Just go the Elasticsearch bin folder and run command "elasticsearch.bat" This is how you can integrate your AWS account to Elasticsearch

8. Go the Kibana bin folder path and open command prompt and run command "kinaba.bat". Login in to the Kibana portal

9. In Kibana go to the Stack Management option and click on Snapshot and Restore option

10. Create repository over there and then create policy and while creating the policy only you have to mentioned Cron expression

Step2:-

1. Now let's try to create policy.

2. To create policy go to the Management --> Stack Management --> Data --> Snapshot and Restore and then click and create a policy

Below snapshot is created automatically as per policy time.

Now look into your AWS S3 bucket if Snapshot is backed up or not

Successfully snapshot backed up into our AWS S3 bucket.

Snapshot Restore :-

Sometimes because of hardware failure your ELK server is stop and that time you are not able to access your logs.

So, in this scenario we can restore the data from the snapshot which is already backed up in AWS S3 bucket.

You can restore your data into new index. Please follow the below process for restore operation.

To restore the snapshot data into the new index run the below line of code in Dev tool option in Kibana. "Kibana --> Dev Tool --> "

Please use below script to restore your data into new index.

POST snapshot/repository-name/snapshot-name/restore

{

"indices" : "your-current-index-name",

"rename_pattern" : "your-current-index-name",

"rename_replacement" : "New-index-name",

"include_global_state" : true,

"ignore_unavailable" : true

}

Using above code, you are able to restore particular snapshot data into the new index.

My current Index management page is look like this

Now I'm trying to restore my snapshot data into another index (my_index2)

Once I will run this above query my snapshot data will restore into new index (my_index2)

See all data which is present in my_index1 is successfully restored into my new index (my_index2).

This is how you can perform Snapshot backup and restore operation.

Hope you all are getting this. Thank you.