Tag Archives: aws

Importing/restoring elastic search snapshot to AWS Elastic Search Service

Took me a long time to find out how to do this.

A few people have re-posted a lot of this AWS article but missed out some crucial details: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-snapshots

The general idea is:

  1. Create an AWS bucket and put the snapshot files into it (don’t use a subdirectory, the .dat files should be in the bucket root).  No need to change permissions on the bucket or anything.
  2. Create an IAM role and policy as per the documentation in the AWS docs link above.  When creating the role using web management console you need to choose EC2 role type and manually modify the trust relationship after creating it.
  3. Run a python script (can find this in the docs link above) using the boto library to register the bucket as a snapshot repository in ES.  You need to sign the request regardless of the ES access policy you are using.  HOWEVER set `is_secure` to `True`.  Without this I was getting `<html></html>` returned instead of any error messages.
  4. Use curl to do the restore (no need to sign restore/backup requests if your access policy is open / IP-based).  Again check the doc for the exact curl command, but as above use https instead of http to get real error messages.

Mount docker socket inside AWS container

  1. To your container, add a new volume
  2. Name: ‘docker_sock’, source path: ‘/var/run/docker.sock’
  3. In Storage and Logging section, add new mount point
  4. Select ‘docker_sock’, container path: ‘/var/run/docker.sock’

And that’s it. No need to give privileged access, and if you run docker commands directly from inside the container there’s no need to change IAM policy.

Associating EC2 instances with an ECS cluster

The EC2 instance is associated with a Container Service cluster using the /etc/ecs/ecs.config file on the instance, in the format ECSCLUSTER=yourcluster_name.

The EC2 instance must also have the ECS agent installed. If you create the instance using the ECS AMI this will be pre-installed (search for AMI called amazon-ecs-optimized).

This configuration can be put in the User Data field:

echo ECS_CLUSTER=your_cluster_name >>/etc/ecs/ecs.config

To find the setting on an instance that already exists: Actions -> Instance Settings -> View/Change User Data

Exact instructions for setting up the EC2 instance properly can be found here: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/launchcontainerinstance.html