S3 Snapshots for Elasticsearch via Kibana
4 min read

S3 Snapshots for Elasticsearch via Kibana

How to setup automatic weekly snapshots for Elasticsearch via S3

I'm terrified of racking up big bills with AWS so I don't work a lot with it, aside from Amazon Simple Email Service (SES). For my ElasticSearch 7.14 cluster that I run alongside with Kibana, I wanted to setup Snapshot and Restore.

My understanding was that with Snapshots, you obtain a copy of your cluster data at that point and you can restore it when you would like to. I already had an AWS account setup before starting with this. For this, you needed to setup a storage repository and then configure via Kibana.

This is how I went about it.

  • Create a S3 Bucket (this is the storage repository we will use with Elasticsearch)
  • Create an AWS policy to allow the IAM User to access the S3 Bucket
  • Create an IAM User and attach the policy we created
  • Install the S3 Repository Plugin
  • Register the repository with Kibana
  • Setup an automatic policy with Kibana / Elasticsearch so it backs up data via cron

Create a S3 Bucket

  • Open AWS Console  -> S3 -> Create Bucket
  • Enter a bucket name: elasticsearch-s3-bucket-snapshot, other defaults are good - make sure you select "Block all public access" and then go ahead with creating the bucket (or saving)
Enter a bucket name: elasticsearch-s3-bucket-snapshot, other defaults are good - make sure you select "Block all public access

Create an AWS policy

  • Open AWS Console -> IAM -> Policies
  • Click on "Create Policy"
  • Switch to JSON editor and paste the following policy (that Elasticsearch recommends for S3 permissions)
{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
  • Click on "Next: Tags", tags can be default so click next again
  • Enter a policy name elasticsearch-s3-policy-snapshot and hit "Create policy"
Enter a policy name elasticsearch-s3-policy-snapshot and hit "Create policy"

Create an IAM User

  • Open AWS Console -> IAM -> Access Management -> Users
  • Click on "Add Users"
  • Enter an user name elasticsearch-s3-user (customizable), check Programmatic access and click Next: Permissions
Enter an user name elasticsearch-s3-user (customizable), check Programmatic access and click Next: Permissions
  • Switch to "Attach existing policies directly" , search and select "elasticsearch-s3-policy-snapshot"
  • Click "Next: Tags", defaults are fine. Review the details and click Create User.
Review screen for Elasticsearch IAM user
  • This will generate a "Access Key ID" and "Secret access key" - both of which we will need soon to setup repository in Kibana, so keep them handy. There's a CSV option to download.
Credentials for IAM User

Install the S3 Repository Plugin

  • On your machine or server, go to where the Elasticsearch bin folder is. Mine was at /usr/share/elasticsearch
  • Run sudo bin/elasticsearch-plugin install repository-s3, answer with y or Y when asked about permissions
  • Restart your cluster!
  • We will configure the client settins for the plugin. Using elasticsearch-keystore, set the credentials for your IAM User
bin/elasticsearch-keystore add s3.client.default.access_key
# enter your "Access Key ID", hit enter
bin/elasticsearch-keystore add s3.client.default.access_key
# enter your "Secret Access Key", hit enter

Register the repository with Kibana

  • Go to your Kibana dashboard -> Management -> Stack Management -> Snapshot & Restore -> Repositories ->  "Register a repository"
Snapshot & Restore on Kibana
  • Name it as elasticsearch-repository-snapshot, select AWS S3 as repository type, click Next. If you don't see AWS S3, make sure you restarted your cluster!
Repository name & Type
  • Client name is default (this is part of your keystore secrets s3.client.default.access_key) and your S3 Bucket Name. Review other defaults and hit Register.
Repository client name & S3 Bucket name
  • Once this is done, it pops up a details screen for the repository. Click on Verify repository
Verify Repository to make sure everything is good

Setup an automatic policy with Kibana

  • On the same screen, switch to Policies -> Create Policy
  • Enter these details
    • elasticsearch-weekly-snapshot as the name
    • <elasticsearch-weekly-snapshot-{now/d}> as the snapshot name, appends the date at the end
    • Select elasticsearch-repository-snapshot
    • Set frequency to every week
    • Click Next
Details for Policy - Cron
  • Click Next, and on review screen, hit Create Policy
Review Policy
  • You can run the policy and when it's done, you will see the snapshot
Run now for the policy
  • You should see something like this once snapshot is complete
snapshot is complete