Quickly deploy elasticsearch with docker

UPDATE: there is a new version that was released in March 2022 and you can read about it here

I needed a way to quickly stand up various versions of elasticsearch for testing. Sometimes just 1 instance was needed and at other time a small cluster and configure it with TLS and security settings. I spent way too much time installing and reinstalling the software on my vm and knew that this will not work well for me. So I set out to look for a light weight, portable, fast, re-produce-able solution that I can quickly stand up and delete and do it over and over.

I decided to tackle this challange using docker-compose and developed a script that will automate elasticsearch and kibana install, VERSION 1. Then I started to test additional items.. metricbeats collections using metricbeats to collect stack monitoring and filebeats to collect elasticsearch and kibana logs, so comes VERSION 2. Later on, I decided I need to test snapshots so I added minio server and configured a snapshot repository, VERSION 3. I will most likely continue to develop this script to add more features.

Features

• Support all modern versions of elasticsearch. 6.x-7.x is supported for the basic stack feature. Various other features such as SLM and metricbeats collection monitoring will only be enabled for when it was enabled on elasticsearch. For example metricbeat collections only started on elasticsearch 6.5+
• TLS Encription on all elasticsearch and kibana endpoints. Only 2 endpoints are exposed on the network after the default stack install: HTTP endpoint for the first instance of elasticsearch and HTTP endpoint for kibana. Both are serviced with https. Rest of the endpoints are internal to the docker network only.
• Certificate Authority that is used to generate all the certificates is copied out to the {version} directory so you can use it for logstash and beats or even import to your browser.
• temp directory is also created and shared amoung all the containers as /temp so that if you need to share or transfer file between containers and host it will be easiler.
• Both elasticsearch.yml and kibana.yml will be created locally so that you can add/remove features and configurations.
• Configurable JVM heap. HEAP settings are toward the top of the script so that it can easily be adjusted.
• Enables security and changes the passwords for all built in users and save it in the notes file
• Node attributes are set so that you can test various ILM policies and shard allocation awareness
• Only takes 2-5 minutes to stand up.. pending your network speed!
• Do not need to be root to run the script! just be part of the docker group

Reqirements

• This script was tested in linux, macOS, WSL. SPECIAL NOTES if you are on macOS please make sure to incrase your cpu/memory for docker via docker desktop or else the containers will keep bootlooping.
• Script is based on docker and docker-compose.
• Default of 512m JVM heap is used per elasticsearch instance and there are 3, so make sure you have enough memory.
• The instance names are: es01, es02, es03, kibana, metricbeat, filebeat, minio01 so you can not have other containers with the same names on your system.
• Due to exposing ports you can only run 1 deployment at a time.
• Ports 9200 & 5601 (also 9000 if minio01 is stood up) is exposed to the host and you can browse to it via localhost or IP.
• Creates a cleanup script so that your deployment can be easily cleaned up and removed. Now cleanup is built into the script as a command

Installation

$ cd   # cd to your home directory
$ curl -fsSL https://raw.githubusercontent.com/jlim0930/scripts/master/deploy-elastic.sh -o deploy-elastic.sh
$ chmod a+x deploy-elastic.sh

Commands & How to use

Commands

• stack|build|start {version} – Stands up 3 elasticsearch & 1 kibana nodes and configure TLS communication.
• monitor {version} – Deploys elasticsearch and kibana if its not deployed and adds metricbeat monitoring to the stack. Only 6.5+ versions are supported. metricbeat & filebeat containers will be stood up to collect monitoring data. NOTE: you can also skip this and just do legacy stack monitoring from kibana.
• snapshot {version} – Deploys elasticsearch and kibana if its not deployed and adds minio container named minio01 and adds repository-s3 plugin to elasticsearch containers and configure the snapshot repository called minio01.
• full {version} – Will deploy the full stack + monitoring + snapshots in 1 run.
• NEW apm {version} – Deploys elasticsearch and kibana if its not deployed already and adds apm to the stack
• NEW fullapm {version} – will deploy full stack + monitoring + snapshot + apm
• NEW entsearch {version} – Deploys elasticsearch and kibana if its not deployed and adds enterprise search. Enterprise Search is only availiable on versions 7.7+
• NEW fullentsearch {version} – will deploy full stack + monitoring + snapshot + enterprise search
• NEW all {version} – deploys full stack + monitoring + snapshot + apm + enterprise search
  

  How to use and examples

  $ ./deploy-elastic.sh stack 7.10.2 # will create a elasticstack directory in your homedir and deploy everything inside of it.
  $ ./deploy-elastic.sh full 7.10.2 # will create a full stack with all the features

  To cleanup a deployment run ./deploy-elastic.sh cleanup

  $ ./deploy-elastic.sh cleanup # this will take down all the containers and remove containers/network/volumes

  If you are having issues with your deployment and want to look at logs, you can look at the container logs to see why the cluster is not coming up.

  $ docker ps
  $ docker logs -f <container name>

Notes

• Once a deployment is stood up, all built in users passwords are stored in the notes file inside of the deployment directory.
• elasticsearch endpoint will be https://localhost:9200 or https://IPofServer:9200
• kibana endpoint will be https://localhost:5601 or https://IPofServer:5601
• minio endpoint will be http://localhost:9000 or https://IPofServer:9000
• node attributes added:

Use & Test Cases

• legacy collections monitoring can be enabled and tested.
• metricbeats collections monitoring.
• snapshot & restore.
• SAML/SSO/LDAP/AD authentication
• logstash/beats ingestions/pipelines
• watchers/alerts
• Centralized pipelines
• Centralized beats
• fleet
• various API calls
• ILM with node attributes
• etc etc etc! if you have a good use case please make a comment.

Future enhancements

• Add App Search
• Add Workplace Search
• Add APM
• Add local SSO server
• Please make suggestions!!