PLEASE use THIS POST instead, as many items were updated and changed.
Configuring snapshots for ECE(Elasticsearch Cloud Enterprise) or deployments for ECE from a S3 object storage or on a s3 compliant storage with publicly trusted certificates are easy but how do you configure the snapshot repository if you are using self-signed or internally signed certificates ?
There are 2 ways of configuring snapshot repository and snapshots on ECE.
First you can configure the snapshot repository on ECE. Once you configure the snapshot repository in the Admin UI, you can configure found-snapshots for each deployment via the Admin UI. Unfortunately, for now unless your endpoint is configured with a publicly trusted certificate or you are using a non TLS endpoint it is not possible to configure the repository in the ECE Admin UI. There are some "hacks" that uses apache/nginx to provide http endpoint for your https none public trusted S3 endpoint but we will not cover it here.
Second method is to configure a snapshot repository per deployment. Again if your s3 endpoint is not TLS or is signed by a publicly trusted CA then the steps are easy however if it is using a self-signed certificate or internal certificate then you would need to create a custom JVM trust store bundle and add it to the deployment so that the certificate can be trusted.
Part 1 – I will configure a minio server with SSL encrypted endpoint, configure s3cmd(mc alternative – since I am more used to s3cmd than mc) to work with my minio server. I will create a custom JVM trust store bundle and host the custom bundle on a web server.
Part 2 – I will configure my deployment to use the custom bundle and configure the snapshot repository
Lets get started
My ECE environment is a 3 node cluster where all the nodes have all roles. I will install the minio server and the httpd server onto the 3rd node. My hosts are running CentOS.
Part 1
Configure minio server
Lets export some variables into our environment
export MINIO_ACCESS_KEY=minio export MINIO_SECRET_KEY=minio123 export MINIO_BUCKET_NAME=elastic
Lets install docker-compose and some other needed tools
sudo yum install docker-compose unzip zip wget -y
Create a /tmp/minio directory – if you have limited /tmp you can create the directory anywhere just make sure to update the path as you go
mkdir /tmp/minio cd /tmp/minio
Create the docker-compose.yml in /tmp/minio/ – we are running minio on port 9001 since port 9000 is taken already by ECE.
version: "2"
services:
minio:
image: minio/minio:latest
ports:
- "9001:9000"
volumes:
- ./data/minio/data:/export
- ./data/minio/config:/root/.minio
environment:
- "MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY"
- "MINIO_SECRET_KEY=$MINIO_SECRET_KEY"
command: server /export
createbuckets:
image: minio/mc
depends_on:
- minio
entrypoint: >
/bin/sh -c "
until (/usr/bin/mc config host add myminio http://minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY) do echo '...waiting...' && sleep 1; done;
/usr/bin/mc mb myminio/$MINIO_BUCKET_NAME;
/usr/bin/mc policy download myminio/$MINIO_BUCKET_NAME;
exit 0;
"
Start the container – from /tmp/minio
$ docker-compose up -d Creating network "a_default" with the default driver Pulling minio (minio/minio:latest)... latest: Pulling from minio/minio d46336f50433: Pull complete be961ec68663: Pull complete 2d8918585761: Pull complete 234a02bcc461: Pull complete 40c7306e8d99: Pull complete 296dea563960: Pull complete fae88b72ea9f: Pull complete Digest: sha256:3755b00abde3ec763272d3fed1f4f4130bcbd56bc1fddc6e134249413d267fd8 Status: Downloaded newer image for minio/minio:latest Pulling createbuckets (minio/mc:latest)... latest: Pulling from minio/mc d46336f50433: Already exists be961ec68663: Already exists 929afd096525: Pull complete d9ca8ede5805: Pull complete 8643f6c7725d: Pull complete Digest: sha256:c3943bffa32e18a463a9283e053a9fcde66dd02ad1017224b01f632d2c3efb78 Creating minio_minio_1 ... done Creating minio_minio_1 ... Creating a_createbuckets_1 ... done $ docker ps | grep minio a4789174b406 minio/mc "/bin/sh -c ' until …" 6 seconds ago Up 5 seconds a_createbuckets_1 98ce8a201610 minio/minio:latest "/usr/bin/docker-ent…" 6 seconds ago Up 5 seconds 0.0.0.0:9001->9000/tcp minio_minio_1
Configure s3cmd to use with my minio server
Install & configure s3cmd
sudo yum install -y s3cmd vi ~/.s3cfg
Fill with the following
host_base = localhost:9001 host_bucket = localhost:9001 use_https = False access_key = minio secret_key = minio123 signature_v2 = False
Lets take s3cmd for a spin
$ s3cmd ls 2021-12-06 20:44 s3://elastic $ s3cmd ls s3://elastic $
We can see that our bucket was already created and the contents of the bucket is empty
Configure TLS for the minio server
We will use the certutil to create the CA and CA signed certificate and apply it to our minio server. In a separate directory. For the IP I am using my host’s internal IP address(GCP environment with an internal and external IP address).
$ cd /tmp
$ mkdir certs
$ chmod 777 /tmp/certs
$ cd /tmp/certs
$ export IMG=$(docker images docker.elastic.co/cloud-assets/elasticsearch --format "{{.Repository}}:{{.Tag}}"| head -n1)
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} /usr/share/elasticsearch/bin/elasticsearch-certutil -s ca --pem --out /tmp/certs/ca.zip
$ sudo chmod 777 ca.zip
$ unzip ca.zip
$ ls -lah ca
total 8.0K
drwxrwxr-x. 2 jlim jlim 34 Dec 7 02:59 .
drwxrwxrwx. 4 jlim jlim 62 Dec 7 03:09 ..
-rw-rw-r--. 1 jlim jlim 1.2K Dec 7 02:59 ca.crt
-rw-rw-r--. 1 jlim jlim 1.7K Dec 7 02:59 ca.key
$ export DOMAIN=ece.local
$ export IP=172.16.0.61
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} /usr/share/elasticsearch/bin/elasticsearch-certutil cert -s --pem --name server --dns "localhost,${DOMAIN},*.${DOMAIN}" --ip "127.0.0.1,${IP}" --out /tmp/certs/server.zip --ca-cert /tmp/certs/ca/ca.crt --ca-key /tmp/certs/ca/ca.key
$ sudo chmod 777 server.zip
$ unzip server.zip
$ ls -lah server
total 8.0K
drwxrwxr-x. 2 jlim jlim 42 Dec 7 03:26 .
drwxrwxrwx. 4 jlim jlim 62 Dec 7 03:26 ..
-rw-rw-r--. 1 jlim jlim 1.3K Dec 7 03:26 server.crt
-rw-rw-r--. 1 jlim jlim 1.7K Dec 7 03:26 server.key
Lets copy server.crt to /tmp/minio/data/minio/config/certs/public.crt & copy server.key to /tmp/minio/data/minio/config/certs/private.key
Take down minio to re-configure it
$ cd /tmp/minio $ docker-compose down
Edit docker-compose.yml line command: server /export to command: server /export --certs-dir /root/.minio/certs/
restart minio
$ cd /tmp/minio $ docker-compose up -d
Now lets reconfigure our s3cmd so that we can access the minio server. Edit .s3cfg to
host_base = localhost:9001 host_bucket = localhost:9001 use_https = True access_key = minio secret_key = minio123 signature_v2 = False ca_certs_file=/tmp/certs/ca/ca.crt
and now your s3cmd should work again.
Create a custom JVM trust store
The instructions to create a custom JVM trust store is located here
$ openssl s_client -connect localhost:9001 -showcerts
CONNECTED(00000003)
depth=0 CN = server
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=server
i:/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDXDCCAkSgAwIBAgIUMCsBYVtbOd7oyxvHsGJBJ3yeYyMwDQYJKoZIhvcNAQEL
BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
cmF0ZWQgQ0EwHhcNMjExMjA3MDMwODQ5WhcNMjQxMjA2MDMwODQ5WjARMQ8wDQYD
VQQDEwZzZXJ2ZXIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCC9Tzv
lKUeT/YeEzxQJjksYGRyqA39hkQoOLm8IHEg3eTctnKsPy/Du2wraRmrxoSTooYJ
mt3/a/9Z8+bmGoS9Q/nckd/jxvT4cTgP0yqrbjRGNaEu+9IdHmzpcxtnV57lrX9l
pJ5dxIHXKVnZSK960leYLYII8dzVDibRGvgUmwfUFMiPG+V2VtEs1RsVWcgEHMzn
Rip+kaLFsSPuEDHjgj6VzTfQjuyCHb6CpDZjDwDGiXR5GucQSBy36sxLrj3xASnj
xZmDFxCZoiPMCFgYcI8ehKonAgMC+oCLEr80iCeQ2W8LyzHXa/fIpCiGRsmxFvFQ
Rqlod35OoIPOTk7FAgMBAAGjgYgwgYUwHQYDVR0OBBYEFDIyiLI+6w3XNy1vUeg8
O0W/Xj3HMB8GA1UdIwQYMBaAFJjHjVISVMTwK/t9n9/becaxJZILMDgGA1UdEQQx
MC+CCWVjZS5sb2NhbIILKi5lY2UubG9jYWyCCWxvY2FsaG9zdIcEfwAAAYcErAAA
ATAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQA5f4BPg1VjdYaQDir46B6+
1h2e7qPMlecCsiI/LgC6fDx9A2VDYDCs36tPKxjvW1N3kHhO3zBap9ANtQCrXTIS
2LiyKihR2Ok/4UNzVlrpBT//Axuf2ZhvSF57w3CpzKiWGfZjSw13byddbiZ1AzmE
7kmQCrZXB23IN/bCSoVnAC+BzLxH0RmEG4wlCbwrIagLJl7AYqF7i7wuV7EgSl4m
dL73AErLB2YaTFb6Ao/P2iNHLNKHYRP/iULpDhbMcgLblizKMPGdzmOo1ub4SOei
p8TL26oEGLwZu3JEmkJhEwmDpm816xCGxwqZswtAns21At8ulA1cPxHA/kXxRlGe
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=server
issuer=/CN=Elastic Certificate Tool Autogenerated CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1485 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: FED030D7D4685EE4010FE3D1FD0AFEB890EC8A9EB8E23D3C21A717120FA52D11
Session-ID-ctx:
Master-Key: DB99744CF8D56955F3C646052893F939A7FBB47A6C31C6BC4F0A1E6504F3591A7CF3BCF8E0D1BCC2EC6F095DD03D3DD2
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket:
0000 - 17 4c 8b d6 3f c5 90 a6-aa b0 ea 23 5b 48 ca e3 .L..?......#[H..
0010 - f5 5c f0 a7 07 78 fb ee-f4 f9 a9 5c dd 20 da f1 .\...x.....\. ..
0020 - 40 f4 4e 8b c4 34 d3 e0-b5 b7 92 c2 28 5f 63 24 @.N..4......(_c$
0030 - be cb ab 7a 78 86 a5 82-ac a5 f4 f3 b3 65 d4 a5 ...zx........e..
0040 - a0 30 17 16 9a 4f 84 86-e5 5e c5 b5 d2 51 9d 1f .0...O...^...Q..
0050 - d4 2a fe 94 a0 0d 53 b5-ed 27 27 42 e3 8e 86 ae .*....S..''B....
0060 - 3d a7 45 a9 bd a3 9b d2-4d 4c 5e 17 4f 24 e3 fe =.E.....ML^.O$..
0070 - 3b db 18 76 75 eb 82 6c-3d 9a ca 0d 14 c9 93 87 ;..vu..l=.......
0080 - 44 D
Start Time: 1638847788
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
I am only going to grab lines from and including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- so it will be
-----BEGIN CERTIFICATE----- MIIDXDCCAkSgAwIBAgIUMCsBYVtbOd7oyxvHsGJBJ3yeYyMwDQYJKoZIhvcNAQEL BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l cmF0ZWQgQ0EwHhcNMjExMjA3MDMwODQ5WhcNMjQxMjA2MDMwODQ5WjARMQ8wDQYD VQQDEwZzZXJ2ZXIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCC9Tzv lKUeT/YeEzxQJjksYGRyqA39hkQoOLm8IHEg3eTctnKsPy/Du2wraRmrxoSTooYJ mt3/a/9Z8+bmGoS9Q/nckd/jxvT4cTgP0yqrbjRGNaEu+9IdHmzpcxtnV57lrX9l pJ5dxIHXKVnZSK960leYLYII8dzVDibRGvgUmwfUFMiPG+V2VtEs1RsVWcgEHMzn Rip+kaLFsSPuEDHjgj6VzTfQjuyCHb6CpDZjDwDGiXR5GucQSBy36sxLrj3xASnj xZmDFxCZoiPMCFgYcI8ehKonAgMC+oCLEr80iCeQ2W8LyzHXa/fIpCiGRsmxFvFQ Rqlod35OoIPOTk7FAgMBAAGjgYgwgYUwHQYDVR0OBBYEFDIyiLI+6w3XNy1vUeg8 O0W/Xj3HMB8GA1UdIwQYMBaAFJjHjVISVMTwK/t9n9/becaxJZILMDgGA1UdEQQx MC+CCWVjZS5sb2NhbIILKi5lY2UubG9jYWyCCWxvY2FsaG9zdIcEfwAAAYcErAAA ATAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQA5f4BPg1VjdYaQDir46B6+ 1h2e7qPMlecCsiI/LgC6fDx9A2VDYDCs36tPKxjvW1N3kHhO3zBap9ANtQCrXTIS 2LiyKihR2Ok/4UNzVlrpBT//Axuf2ZhvSF57w3CpzKiWGfZjSw13byddbiZ1AzmE 7kmQCrZXB23IN/bCSoVnAC+BzLxH0RmEG4wlCbwrIagLJl7AYqF7i7wuV7EgSl4m dL73AErLB2YaTFb6Ao/P2iNHLNKHYRP/iULpDhbMcgLblizKMPGdzmOo1ub4SOei p8TL26oEGLwZu3JEmkJhEwmDpm816xCGxwqZswtAns21At8ulA1cPxHA/kXxRlGe -----END CERTIFICATE-----
Now lets take a look at this certificate you will see that it is the same file as your server.crt, so ideally you can skip the above and just use your server.crt for this exercise.
$ cd /tmp/certs
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} cp /usr/share/elasticsearch/jdk/lib/security/cacerts /tmp/certs/cacerts
$ sudo chmod 777 cacerts
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} /usr/share/elasticsearch/jdk/bin/keytool -keystore cacerts -storepass changeit -noprompt -importcert -file server/server.crt -alias minio
$ zip cacerts.zip cacerts
Lets get the cacerts.zip staged for your httpd
mkdir /tmp/htdocs cd /tmp/htdocs cp /tmp/certs/cacerts.zip ./
Host the custom bundle on a web server
Lets now host a simple http server that will serve our cacerts.zip
$ docker run -dit --name apache -p 9002:80 -v /tmp/htdocs:/usr/local/apache2/htdocs/ httpd:2.4 $ docker ps | grep http 515e05b0a759 httpd:2.4 "httpd-foreground" About an hour ago Up About an hour 0.0.0.0:9002->80/tcp apache
To test lets try to grab the cacerts file
$ wget http://localhost:9002/cacerts.zip --2021-12-07 00:29:25-- http://localhost:9002/cacerts.zip Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:9002... connected. HTTP request sent, awaiting response... 200 OK Length: 111398 (109K) [application/zip] Saving to: ‘cacerts.zip’ 100%[===============================================================================================================================================>] 111,398 --.-K/s in 0s 2021-12-07 00:29:25 (360 MB/s) - ‘cacerts.zip’ saved [111398/111398]
Now all the preparation is complete!
Part 2
We will stand up a deployment and configure custom bundles and configure the snapshot
Lets create a simple deployment named test. We will use the default elasticsearch deployment with 2 AZ.
After the deployment is created lets goto Edit -> Advanced Edit -> Deployment configuration and around line 56 look for
"elasticsearch": {
"version": "7.11.1"
},
and change it to
"elasticsearch": {
"version": "7.11.1",
"user_bundles": [
{
"name": "custom-ca-certs",
"url": "http://172.16.0.61:9002/cacerts.zip",
"elasticsearch_version": "*"
}
]
},
Please change 172.16.0.61 to the IP of your http server that we stood up in Part 1. I am using GCP instances so I used the internal IP of the GCP instance. IF you are on prem you can use the IP of the ECE host.
Click on SAVE – this will lead to a change plan and your cluster should restart. Once the change is complete goto the devtools in kibana for the test deployment
Lets create the snapshot repository
PUT _snapshot/minio01
{
"type": "s3",
"settings": {
"bucket": "elastic",
"endpoint": "172.16.0.61:9001",
"protocol": "https",
"path_style_access": "true",
"access_key": "minio",
"secret_key": "minio123"
}
}
#! [access_key] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
#! [secret_key] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
#! Using s3 access/secret key from repository settings. Instead store these in named clients and the elasticsearch keystore for secure settings.
{
"acknowledged" : true
}
Verify the repository
POST /_snapshot/minio01/_verify
{
"nodes" : {
"eFEJL77STGyv7PU0a8OtSw" : {
"name" : "instance-0000000001"
},
"ORg5jorNT2-9jdeYcqEC-Q" : {
"name" : "instance-0000000000"
}
}
}
You can now go to Stack Management -> Snapshot and restore -> Policy -> create or even create a new SLM policy from devtools.
Just for testing lets run a quick snapshot and verify it
PUT _snapshot/minio01/snapshot_1?wait_for_completion=true GET _cat/snapshots/minio01?v id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards snapshot_1 SUCCESS 1638823923 20:52:03 1638823924 20:52:04 1s 9 9 0 9
One Comment