2

New elastic kubernetes script – deploy-elastick8s.sh

The old deploy-eck.sh script have grown and matured a bit and now it encompasses helm charts and native installs. I renamed the script to be more inclusive of its features. All of the older articles will reference this post.

The new script is located deploy-elastick8s.sh

This script was designed to easily stand up elastic and its stack on kubernetes in various modes to easily test and replicate issues for troubleshooting and to setup examples for various workloads. The deployments that this script creates is not for production use but to be used as lab or test environments.

This post will be about how to use the script with examples to easily stand up various elastic configurations on kubernetes using eck, helm charts, natively run elastic on kubernetes.

All yaml files, certificates, any files that are related to the deployment is stored in <HOMEDIR>/elastick8s so that if you need to test changes, or additional items you can easily have access to these files and make changes.

Requirements

This script does have some requirements:

  1. kubernetes environment. I wrote the script using GKE and minikube but it should work with AKS, EKS, & openshift. For GKE, I do have a script that can be used to stand up your GKE environment gke.sh and it is encouraged for you to use this script.
  2. jq
  3. kubectl
  4. openssl
  5. docker and docker service running on the same host
  6. access to your home directory. The script writes files and configurations into your <HOMEDIR>/elastick8s

Limitations

As mentioned above please do not use the environments stood up with this script for production. Various parts of the script is limited to a set of versions of the ECK operator and elasticstack versions. This was done to make things easier and the script shorter.

CommandVersion *
operatorECK operator 1.0.0+
stack/start/build/eckECK operator 1.4.0+ & STACK 7.10.0+
eckldapECK operator 1.4.0+ & STACK 7.10.0+
dedicatedECK operator 1.4.0+ & STACK 7.10.0+
beatsECK operator 1.4.0+ & STACK 7.10.0+
monitor1ECK operator 1.4.0+ & STACK 7.10.0+
monitor2ECK operator 1.7.0+ & STACK 7.14.0+
fleetECK operator 1.7.0+ & STACK 7.14.0+
helmSTACK 7.14.0+ & < 8.0.0
helmldapSTACK 7.14.0+ & < 8.0.0
helmbeatsSTACK 7.14.0+ & < 8.0.0
helmlogstashbeatsSTACK 7.14.0+ & < 8.0.0
helmmonitorSTACK 7.14.0+ & < 8.0.0
nativeSTACK 7.3.0+
nativebeatsSTACK 7.3.0+
nativemonitorSTACK 7.5.0+

* pending your kubernetes version

Commands:

help

  • print help and exit

summary

  • If a deployment exists it will print the endpoints and elastic users password

cleanup

  • delete resources and delete ~/elastick8s directory
> ./deploy-elastick8s.sh cleanup

********** Cleaning up **********

[DEBUG] DELETING Resources for: /Users/jlim/elastick8s/license.yaml
[DEBUG] DELETING Resources for: /Users/jlim/elastick8s/operator.yaml
[DEBUG] DELETING Resources for: /Users/jlim/elastick8s/crds.yaml

[DEBUG] All cleanedup

ECK Commands:

operator

  • Installs the ECK operator and the trial license. This command deploys the ECK operator version that was specified & a trial ECK license so that you can stand up your own ECK operator based environment in kubernetes.
> ./deploy-elastick8s.sh operator 1.8.0
 ********** Deploying ECK 1.8.0 OPERATOR **************

[DEBUG] ECK 1.8.0 downloading crds: crds.yaml
[DEBUG] ECK 1.8.0 downloading operator: operator.yaml
[DEBUG] ECK Operator is starting.  Checking again in 10 seconds.  If the operator does not goto Running status in few minutes something is wrong. CTRL-C please

[DEBUG] ECK 1.8.0 OPERATOR is HEALTHY

                     NAME                     READY   STATUS    RESTARTS   AGE
                     pod/elastic-operator-0   1/1     Running   0          11s

                     NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
                     service/elastic-webhook-server   ClusterIP   10.88.15.118   <none>        443/TCP   11s

                     NAME                                READY   AGE
                     statefulset.apps/elastic-operator   1/1     11s

[DEBUG] ECK 1.8.0 Creating license.yaml
[DEBUG] ECK 1.8.0 Applying trial license
~ > ls elastick8s
ECK           ECKVERSION    crds.yaml     license.yaml  operator.yaml temp

stack|start|build

  • Installs the ECK operator with the trial license and install elasticsearch & kibana. The deployment is named eck-lab. After the script is done, it will display the elasticsearch & kibana endpoints that you can access it by and also the elastic users password. All of this information is stored in ~/elastick8s/notes in case it was forgotten. ca.crt used to sign the server certificates will also be stored in ~/elastick8s/ directory. The elasticsearch will have 3 nodes marked as default and 1 instance of kibana.
> ./deploy-elastick8s.sh stack 8.4.1 2.3.0
[DEBUG] ECK 2.3.0 version validated.
[DEBUG] This might take a while.  In another window you can watch -n2 kubectl get all or kubectl get events -w to watch the stack being stood up

 ********** Deploying ECK 2.3.0 OPERATOR **************

[DEBUG] ECK 2.3.0 downloading crds: crds.yaml
[DEBUG] ECK 2.3.0 downloading operator: operator.yaml
[DEBUG] ECK Operator is starting.  Checking again in 10 seconds.  If the operator does not goto Running status in few minutes something is wrong. CTRL-C please

[DEBUG] ECK 2.3.0 OPERATOR is HEALTHY

                     NAME                     READY   STATUS    RESTARTS   AGE
                     pod/elastic-operator-0   1/1     Running   0          11s

                     NAME                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
                     service/elastic-webhook-server   ClusterIP   10.88.5.189   <none>        443/TCP   11s

                     NAME                                READY   AGE
                     statefulset.apps/elastic-operator   1/1     12s

[DEBUG] ECK 2.3.0 Creating license.yaml
[DEBUG] ECK 2.3.0 Applying trial license

 ********** Deploying ECK 2.3.0 STACK 8.4.1 CLUSTER eck-lab **************

[DEBUG] ECK 2.3.0 STACK 8.4.1 CLUSTER eck-lab Creating elasticsearch.yaml
[DEBUG] ECK 2.3.0 STACK 8.4.1 CLUSTER eck-lab Starting elasticsearch cluster.
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-es-default sts is showing 3 replicas ready

                     NAME                 READY   AGE
                     eck-lab-es-default   3/3     93s

[DEBUG] ECK 2.3.0 STACK 8.4.1 CLUSTER  eck-lab Creating kibana.yaml
[DEBUG] ECK 2.3.0 STACK 8.4.1 CLUSTER  eck-lab Starting kibana.
[DEBUG] eck-lab-kb is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-kb is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-kb is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-kb is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-kb is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] eck-lab-kb deployment is showing 1 replicas ready

                     NAME         READY   UP-TO-DATE   AVAILABLE   AGE
                     eck-lab-kb   1/1     1            1           61s

[DEBUG] Grabbing elastic password for eck-lab: 42QD77H17244taDOmW7YxHPa
[DEBUG] Grabbing elasticsearch endpoint for  eck-lab: https://34.171.41.227:9200
[DEBUG] Grabbing kibana endpoint for eck-lab: https://34.70.145.200:5601

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                              READY   STATUS    RESTARTS   AGE
                     pod/eck-lab-es-default-0          1/1     Running   0          2m43s
                     pod/eck-lab-es-default-1          1/1     Running   0          2m43s
                     pod/eck-lab-es-default-2          1/1     Running   0          2m43s
                     pod/eck-lab-kb-59b6b8fcdc-sl9z6   1/1     Running   0          69s
                     pod/eck-lab-kb-6c6d98546c-jhgrr   0/1     Running   0          41s

                     NAME                               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-default         ClusterIP      None           <none>          9200/TCP         2m43s
                     service/eck-lab-es-http            LoadBalancer   10.88.1.83     34.171.41.227   9200:30910/TCP   2m45s
                     service/eck-lab-es-internal-http   ClusterIP      10.88.0.184    <none>          9200/TCP         2m45s
                     service/eck-lab-es-transport       ClusterIP      None           <none>          9300/TCP         2m45s
                     service/eck-lab-kb-http            LoadBalancer   10.88.14.203   34.70.145.200   5601:31550/TCP   69s
                     service/kubernetes                 ClusterIP      10.88.0.1      <none>          443/TCP          16m

                     NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb   1/1     1            1           69s

                     NAME                                    DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-59b6b8fcdc   1         1         1       69s
                     replicaset.apps/eck-lab-kb-6c6d98546c   1         1         0       41s

                     NAME                                  READY   AGE
                     statefulset.apps/eck-lab-es-default   3/3     2m43s

[SUMMARY] STACK INFO:
eck-lab elastic password: 42QD77H17244taDOmW7YxHPa
eck-lab elasticsearch endpoint: https://34.171.41.227:9200
eck-lab kibana endpoint: https://34.70.145.200:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

eckldap

  • SAME as the default build but adds an openldap server and configures elasticsearch to add an ldap realm for authentication. The ldap server comes with 2 users user01:password01 & user02:password02. After the deployment is complete, you can login with the ldap users. The ldap realm has superuser role configured. NOTE: reconfiguring the elasticsearch pods does take some time so please ensure that all 3 elasticsearchs have restarted before trying to login using ldap users.
> ./deploy-elastick8s.sh eckldap 8.4.1 2.3.0

.....SNIPPED....

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                             READY   STATUS        RESTARTS   AGE
                     pod/eck-lab-es-default-0         1/1     Running       0          2m20s
                     pod/eck-lab-es-default-1         1/1     Running       0          2m20s
                     pod/eck-lab-es-default-2         1/1     Terminating   0          2m20s
                     pod/eck-lab-kb-cbfc59d95-2cs8c   1/1     Running       0          29s
                     pod/openldap-7f878f4bcc-6cfdr    1/1     Running       0          7s

                     NAME                               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-default         ClusterIP      None           <none>          9200/TCP         2m20s
                     service/eck-lab-es-http            LoadBalancer   10.88.14.209   104.154.19.8    9200:31506/TCP   2m22s
                     service/eck-lab-es-internal-http   ClusterIP      10.88.12.187   <none>          9200/TCP         2m22s
                     service/eck-lab-es-transport       ClusterIP      None           <none>          9300/TCP         2m22s
                     service/eck-lab-kb-http            LoadBalancer   10.88.4.238    34.66.121.117   5601:30793/TCP   57s
                     service/kubernetes                 ClusterIP      10.88.0.1      <none>          443/TCP          22m
                     service/openldap                   ClusterIP      10.88.6.185    <none>          1636/TCP         7s

                     NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb   1/1     1            1           56s
                     deployment.apps/openldap     1/1     1            1           7s

                     NAME                                   DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-cbfc59d95   1         1         1       29s
                     replicaset.apps/openldap-7f878f4bcc    1         1         1       7s

                     NAME                                  READY   AGE
                     statefulset.apps/eck-lab-es-default   3/3     2m20s

[SUMMARY] STACK INFO:
eck-lab elastic password: 9F3WF98MtbNw3463y8piyo5D
eck-lab elasticsearch endpoint: https://104.154.19.8:9200
eck-lab kibana endpoint: https://34.66.121.117:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

~ > curl --cacert ~/elastick8s/ca.crt -u "user01:password01" https://104.154.19.8:9200
{
  "name" : "eck-lab-es-default-2",
  "cluster_name" : "eck-lab",
  "cluster_uuid" : "qRoqvrt6RxmhX-2blfdmMg",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c",
    "build_date" : "2022-08-26T12:11:43.232597118Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

dedicated

  • Deploys dedicated elasticsearch instances. 3 masters & 3 data nodes.
> ./deploy-elastick8s.sh dedicated 8.4.1 2.3.0

.....SNIPPED....

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                              READY   STATUS    RESTARTS   AGE
                     pod/eck-lab-es-data-0             1/1     Running   0          2m31s
                     pod/eck-lab-es-data-1             1/1     Running   0          2m31s
                     pod/eck-lab-es-data-2             1/1     Running   0          2m31s
                     pod/eck-lab-es-master-0           1/1     Running   0          2m31s
                     pod/eck-lab-es-master-1           1/1     Running   0          2m31s
                     pod/eck-lab-es-master-2           1/1     Running   0          2m31s
                     pod/eck-lab-kb-5dccbb7b68-l8b2w   0/1     Running   0          10s
                     pod/eck-lab-kb-7bb9b95d78-8bzt6   1/1     Running   0          38s

                     NAME                               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-data            ClusterIP      None           <none>          9200/TCP         2m31s
                     service/eck-lab-es-http            LoadBalancer   10.88.3.149    34.134.74.193   9200:31713/TCP   2m33s
                     service/eck-lab-es-internal-http   ClusterIP      10.88.10.143   <none>          9200/TCP         2m33s
                     service/eck-lab-es-master          ClusterIP      None           <none>          9200/TCP         2m31s
                     service/eck-lab-es-transport       ClusterIP      None           <none>          9300/TCP         2m33s
                     service/eck-lab-kb-http            LoadBalancer   10.88.5.11     104.198.33.66   5601:31747/TCP   39s
                     service/kubernetes                 ClusterIP      10.88.0.1      <none>          443/TCP          34m

                     NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb   1/1     1            1           39s

                     NAME                                    DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-5dccbb7b68   1         1         0       10s
                     replicaset.apps/eck-lab-kb-7bb9b95d78   1         1         1       39s

                     NAME                                 READY   AGE
                     statefulset.apps/eck-lab-es-data     3/3     2m31s
                     statefulset.apps/eck-lab-es-master   3/3     2m31s

[SUMMARY] STACK INFO:
eck-lab elastic password: n17Ijyh6cpr24Ou2eq9k215h
eck-lab elasticsearch endpoint: https://34.134.74.193:9200
eck-lab kibana endpoint: https://104.198.33.66:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

beats

  • Installs the default stack and installs filebeat, metricbeat, heartbeat, packetbeat for the kubernetes environment. NOTE: beats may restart a few times and that is normal.
> ./deploy-elastick8s.sh beats 8.4.1 2.3.0

.....SNIPPED....

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                                            READY   STATUS    RESTARTS        AGE
                     pod/eck-lab-es-default-0                        1/1     Running   0               4m52s
                     pod/eck-lab-es-default-1                        1/1     Running   0               4m52s
                     pod/eck-lab-es-default-2                        1/1     Running   0               4m52s
                     pod/eck-lab-kb-6dcbbf7cdf-gg6zz                 1/1     Running   0               2m57s
                     pod/filebeat-beat-filebeat-dflcz                1/1     Running   0               2m46s
                     pod/filebeat-beat-filebeat-fn6xs                1/1     Running   0               2m46s
                     pod/filebeat-beat-filebeat-j8nzp                1/1     Running   0               2m46s
                     pod/heartbeat-beat-heartbeat-756dc6cb98-vvkd6   1/1     Running   0               2m48s
                     pod/metricbeat-beat-metricbeat-8vdp5            1/1     Running   0               2m48s
                     pod/metricbeat-beat-metricbeat-h5jwj            1/1     Running   0               2m48s
                     pod/metricbeat-beat-metricbeat-s5q47            1/1     Running   0               2m48s
                     pod/packetbeat-beat-packetbeat-52q6v            1/1     Running   2 (2m6s ago)    2m46s
                     pod/packetbeat-beat-packetbeat-826m5            1/1     Running   2 (2m18s ago)   2m46s
                     pod/packetbeat-beat-packetbeat-mwqj4            1/1     Running   0               2m47s

                     NAME                               TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-default         ClusterIP      None          <none>          9200/TCP         4m53s
                     service/eck-lab-es-http            LoadBalancer   10.88.3.197   34.132.208.41   9200:30510/TCP   4m54s
                     service/eck-lab-es-internal-http   ClusterIP      10.88.6.21    <none>          9200/TCP         4m54s
                     service/eck-lab-es-transport       ClusterIP      None          <none>          9300/TCP         4m54s
                     service/eck-lab-kb-http            LoadBalancer   10.88.14.87   35.222.68.152   5601:30959/TCP   3m28s
                     service/kubernetes                 ClusterIP      10.88.0.1     <none>          443/TCP          47m

                     NAME                                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
                     daemonset.apps/filebeat-beat-filebeat       3         3         3       3            3           <none>          2m46s
                     daemonset.apps/metricbeat-beat-metricbeat   3         3         3       3            3           <none>          2m48s
                     daemonset.apps/packetbeat-beat-packetbeat   3         3         3       3            3           <none>          2m47s

                     NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb                 1/1     1            1           3m28s
                     deployment.apps/heartbeat-beat-heartbeat   1/1     1            1           2m49s

                     NAME                                                  DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-6dcbbf7cdf                 1         1         1       2m58s
                     replicaset.apps/heartbeat-beat-heartbeat-756dc6cb98   1         1         1       2m49s

                     NAME                                  READY   AGE
                     statefulset.apps/eck-lab-es-default   3/3     4m54s

[SUMMARY] STACK INFO:
eck-lab elastic password: 0VZRS2SvR79B30kZ20TQq79d
eck-lab elasticsearch endpoint: https://34.132.208.41:9200
eck-lab kibana endpoint: https://35.222.68.152:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

monitor1

  • Stands up a standard eck-lab stack & also another stack named eck-lab-monitor along with filebeat & metricbeat as a pod to stack monitor eck-lab into eck-lab-monitor It will display the login info for both clusters. You can log into eck-lab-monitor to view the stack monitoring for eck-lab. NOTE: beats may restart a few times and that is normal.
> ./deploy-elastick8s.sh monitor1 8.4.1 2.3.0

.....SNIPPED....

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                                              READY   STATUS    RESTARTS      AGE
                     pod/eck-lab-es-default-0                          1/1     Running   0             6m21s
                     pod/eck-lab-es-default-1                          1/1     Running   0             6m21s
                     pod/eck-lab-es-default-2                          1/1     Running   0             6m21s
                     pod/eck-lab-kb-6dd7b79876-95h44                   1/1     Running   0             4m29s
                     pod/eck-lab-monitor-es-default-0                  1/1     Running   0             4m21s
                     pod/eck-lab-monitor-es-default-1                  1/1     Running   0             4m21s
                     pod/eck-lab-monitor-es-default-2                  1/1     Running   0             4m21s
                     pod/eck-lab-monitor-kb-7bc8bd5f7c-lqm4d           1/1     Running   0             2m29s
                     pod/filebeat-beat-filebeat-44qt5                  1/1     Running   4 (60s ago)   2m5s
                     pod/filebeat-beat-filebeat-4h2rl                  1/1     Running   4 (66s ago)   2m5s
                     pod/filebeat-beat-filebeat-smlvr                  1/1     Running   4 (66s ago)   2m5s
                     pod/metricbeat-beat-metricbeat-65877875d4-6jg2b   1/1     Running   4 (56s ago)   2m6s

                     NAME                                       TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-default                 ClusterIP      None          <none>          9200/TCP         6m21s
                     service/eck-lab-es-http                    LoadBalancer   10.88.1.194   35.223.44.173   9200:31164/TCP   6m23s
                     service/eck-lab-es-internal-http           ClusterIP      10.88.4.220   <none>          9200/TCP         6m23s
                     service/eck-lab-es-transport               ClusterIP      None          <none>          9300/TCP         6m23s
                     service/eck-lab-kb-http                    LoadBalancer   10.88.2.72    34.171.41.227   5601:32163/TCP   4m58s
                     service/eck-lab-monitor-es-default         ClusterIP      None          <none>          9200/TCP         4m21s
                     service/eck-lab-monitor-es-http            LoadBalancer   10.88.7.124   34.70.145.200   9200:30356/TCP   4m22s
                     service/eck-lab-monitor-es-internal-http   ClusterIP      10.88.10.15   <none>          9200/TCP         4m22s
                     service/eck-lab-monitor-es-transport       ClusterIP      None          <none>          9300/TCP         4m22s
                     service/eck-lab-monitor-kb-http            LoadBalancer   10.88.8.178   34.66.121.117   5601:32713/TCP   2m57s
                     service/kubernetes                         ClusterIP      10.88.0.1     <none>          443/TCP          57m

                     NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
                     daemonset.apps/filebeat-beat-filebeat   3         3         3       3            3           <none>          2m5s

                     NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb                   1/1     1            1           4m57s
                     deployment.apps/eck-lab-monitor-kb           1/1     1            1           2m57s
                     deployment.apps/metricbeat-beat-metricbeat   1/1     1            1           2m6s

                     NAME                                                    DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-6dd7b79876                   1         1         1       4m29s
                     replicaset.apps/eck-lab-monitor-kb-7bc8bd5f7c           1         1         1       2m29s
                     replicaset.apps/metricbeat-beat-metricbeat-65877875d4   1         1         1       2m6s

                     NAME                                          READY   AGE
                     statefulset.apps/eck-lab-es-default           3/3     6m21s
                     statefulset.apps/eck-lab-monitor-es-default   3/3     4m21s

[SUMMARY] STACK INFO:
eck-lab elastic password: I1Q21G2wmjUrU1WL3sV05E03
eck-lab elasticsearch endpoint: https://35.223.44.173:9200
eck-lab kibana endpoint: https://34.171.41.227:5601
eck-lab-monitor elastic password: 30CUUsR9blRY2cnb41L1O225
eck-lab-monitor elasticsearch endpoint: https://34.70.145.200:9200
eck-lab-monitor kibana endpoint: https://34.66.121.117:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

monitor2

  • Similar to monitor1 however starting ECK 1.7.0+ & STACK 7.14.0+ you can enable sidecar containers instead of pods for filebeat and metricbeats to monitor your cluster.
> ./deploy-elastick8s.sh monitor2 8.4.1 2.3.0

.....SNIPPED....

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                                      READY   STATUS    RESTARTS      AGE
                     pod/eck-lab-es-default-0                  3/3     Running   0             104s
                     pod/eck-lab-es-default-1                  3/3     Running   0             104s
                     pod/eck-lab-es-default-2                  3/3     Running   0             104s
                     pod/eck-lab-kb-7bb7545c9b-xx62g           3/3     Running   3 (52s ago)   76s
                     pod/eck-lab-monitor-es-default-0          1/1     Running   0             3m49s
                     pod/eck-lab-monitor-es-default-1          1/1     Running   0             3m49s
                     pod/eck-lab-monitor-es-default-2          1/1     Running   0             3m49s
                     pod/eck-lab-monitor-kb-75c79d64fd-b7j8d   1/1     Running   0             117s

                     NAME                                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-default                 ClusterIP      None           <none>          9200/TCP         104s
                     service/eck-lab-es-http                    LoadBalancer   10.88.15.192   104.198.33.66   9200:30662/TCP   109s
                     service/eck-lab-es-internal-http           ClusterIP      10.88.3.121    <none>          9200/TCP         109s
                     service/eck-lab-es-transport               ClusterIP      None           <none>          9300/TCP         109s
                     service/eck-lab-kb-http                    LoadBalancer   10.88.12.248   35.222.68.152   5601:30422/TCP   103s
                     service/eck-lab-monitor-es-default         ClusterIP      None           <none>          9200/TCP         3m49s
                     service/eck-lab-monitor-es-http            LoadBalancer   10.88.10.96    104.154.19.8    9200:30314/TCP   3m51s
                     service/eck-lab-monitor-es-internal-http   ClusterIP      10.88.0.133    <none>          9200/TCP         3m51s
                     service/eck-lab-monitor-es-transport       ClusterIP      None           <none>          9300/TCP         3m51s
                     service/eck-lab-monitor-kb-http            LoadBalancer   10.88.0.146    34.134.74.193   5601:31864/TCP   2m25s
                     service/kubernetes                         ClusterIP      10.88.0.1      <none>          443/TCP          64m

                     NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb           1/1     1            1           100s
                     deployment.apps/eck-lab-monitor-kb   1/1     1            1           2m24s

                     NAME                                            DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-7bb7545c9b           1         1         1       76s
                     replicaset.apps/eck-lab-monitor-kb-75c79d64fd   1         1         1       117s

                     NAME                                          READY   AGE
                     statefulset.apps/eck-lab-es-default           3/3     104s
                     statefulset.apps/eck-lab-monitor-es-default   3/3     3m49s

[SUMMARY] STACK INFO:
eck-lab-monitor elastic password: q417Xjw86x0i9y5c3BYnk6XJ
eck-lab-monitor elasticsearch endpoint: https://104.154.19.8:9200
eck-lab-monitor kibana endpoint: https://34.134.74.193:5601
eck-lab elastic password: OOc1e9iw5uD6e6jobn7929F3
eck-lab elasticsearch endpoint: https://104.198.33.66:9200
eck-lab kibana endpoint: https://35.222.68.152:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

fleet

  • Stands up the standard stack & installs a Fleet server and an elastic-agent for each of the kubernetes hosts as a DaemonSet and reports back to the stack. Also configures an external output so that you can register elastic-agents from outside of the kubernetes environment. CA fingerprint is also added to the output so that you no longer have to provide the CA.
> ./deploy-elastick8s.sh fleet 8.4.1 2.3.0

.....SNIPPED....

[SUMMARY] ECK 2.3.0 STACK 8.4.1

                     NAME                                     READY   STATUS    RESTARTS      AGE
                     pod/eck-lab-es-default-0                 1/1     Running   0             4m53s
                     pod/eck-lab-es-default-1                 1/1     Running   0             4m53s
                     pod/eck-lab-es-default-2                 1/1     Running   0             4m53s
                     pod/eck-lab-kb-7fd7d55655-zwhlt          1/1     Running   0             2m53s
                     pod/elastic-agent-agent-jqvkq            1/1     Running   2 (57s ago)   104s
                     pod/elastic-agent-agent-wmkvg            1/1     Running   2 (44s ago)   104s
                     pod/elastic-agent-agent-zf85l            1/1     Running   1 (68s ago)   104s
                     pod/fleet-server-agent-5844f447c-h886x   1/1     Running   0             79s

                     NAME                               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
                     service/eck-lab-es-default         ClusterIP      None           <none>          9200/TCP         4m53s
                     service/eck-lab-es-http            LoadBalancer   10.88.3.136    34.132.208.41   9200:30654/TCP   4m54s
                     service/eck-lab-es-internal-http   ClusterIP      10.88.14.183   <none>          9200/TCP         4m54s
                     service/eck-lab-es-transport       ClusterIP      None           <none>          9300/TCP         4m54s
                     service/eck-lab-kb-http            LoadBalancer   10.88.13.193   34.70.145.200   5601:31994/TCP   3m29s
                     service/fleet-server-agent-http    LoadBalancer   10.88.6.52     35.223.44.173   8220:32091/TCP   106s
                     service/kubernetes                 ClusterIP      10.88.0.1      <none>          443/TCP          75m

                     NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
                     daemonset.apps/elastic-agent-agent   3         3         3       3            3           <none>          104s

                     NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/eck-lab-kb           1/1     1            1           3m28s
                     deployment.apps/fleet-server-agent   1/1     1            1           105s

                     NAME                                           DESIRED   CURRENT   READY   AGE
                     replicaset.apps/eck-lab-kb-7fd7d55655          1         1         1       2m53s
                     replicaset.apps/fleet-server-agent-5844f447c   1         1         1       79s

                     NAME                                  READY   AGE
                     statefulset.apps/eck-lab-es-default   3/3     4m54s

[SUMMARY] STACK INFO:
eck-lab elastic password: 643c9N6mVDaBe4H0jb39Uj6s
eck-lab elasticsearch endpoint: https://34.132.208.41:9200
eck-lab kibana endpoint: https://34.70.145.200:5601
eck-lab Fleet Server endpoint: https://35.223.44.173:8220

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

[NOTE] If you missed the summary its also in /Users/jlim/elastick8s/notes
[NOTE] You can start logging into kibana but please give things few minutes for proper startup and letting components settle down.

file

HELM Commands:

Just wanted to explain some common misinformation about helm.
You can run elasticsearch on kubernetes natively, via eck operator, or via helm charts. With helm charts you can use elastic provided helm charts or any 3rd party charts. Not all helm installs use the ECK operator. Easiest way to tell the difference is to look for the elastic-operator pod in all namespaces. If there are no elastic-operator pod that means you've just deployed elasticsearch using helm charts but not with an ECK operator. For the helm modes in the script, I am using the elastic provided helm charts without the ECK operator.

All of helm features on the script is limited to STACK 7.14.0+ & < 8.0.0 There are currently, as of this article, no 8.x versions of helm charts released by elastic.

helm

  • Stands up 3 instances of elasticsearch & 1 instance of kibana. The deployment is named helm-lab. All the endpoints are secured with SSL certificates just like ECK commands and the ca.crt is saved to <HOMEDIR>/elastick8s/.
> ./deploy-elastick8s.sh helm 7.17.5

 ********** Deploying ELASTIC STACK 7.17.5 with HELM CHARTS - without ECK operator **************

[DEBUG] Create elastic user password and create a secret
[DEBUG] Create certificates for the stack and add create secrets

[DEBUG] Creating ca.crt and certificate/key for helm-lab-es-default
[DEBUG] Creating secret helm-lab-es-default-certificates

[DEBUG] Creating ca.crt and certificate/key for helm-lab-kb-default
[DEBUG] Creating secret helm-lab-kb-default-certificates
[DEBUG] Create es-values.yaml
[DEBUG] Starting elasticsearch
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-es-default sts is showing 3 replicas ready

                     NAME                  READY   AGE
                     helm-lab-es-default   3/3     106s

[DEBUG] Creating kibana encryption key secret
[DEBUG] Create kb-vaules.yaml
[DEBUG] Starting kibana
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] helm-lab-kb-kibana deployment is showing 1 replicas ready

                     NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
                     helm-lab-kb-kibana   1/1     1            1           82s

[DEBUG] Grabbing elastic password for helm-lab: IxG62Fn1tfdb0u3BVGpFYpvcR
[DEBUG] Grabbing elasticsearch endpoint for  helm-lab: https://34.171.41.227:9200
[DEBUG] Grabbing kibana endpoint for helm-lab: https://34.66.121.117:5601

[DEBUG] Applying trial license.

[DEBUG] Stack is up but will take a minute or two to become healthy.  Please view kubectl get pods to ensure that all pods are up before trying to login

[SUMMARY] HELM STACK 7.17.5

                     NAME                                      READY   STATUS    RESTARTS   AGE
                     pod/helm-lab-es-default-0                 1/1     Running   0          3m21s
                     pod/helm-lab-es-default-1                 1/1     Running   0          3m21s
                     pod/helm-lab-es-default-2                 1/1     Running   0          3m21s
                     pod/helm-lab-kb-kibana-7b55c76fc5-vg7w2   1/1     Running   0          90s

                     NAME                                   TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                         AGE
                     service/helm-lab-es-default            LoadBalancer   10.88.7.51    34.171.41.227   9200:30301/TCP,9300:31484/TCP   3m21s
                     service/helm-lab-es-default-headless   ClusterIP      None          <none>          9200/TCP,9300/TCP               3m21s
                     service/helm-lab-kb-kibana             LoadBalancer   10.88.6.173   34.66.121.117   5601:32352/TCP                  90s
                     service/kubernetes                     ClusterIP      10.88.0.1     <none>          443/TCP                         89m

                     NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/helm-lab-kb-kibana   1/1     1            1           90s

                     NAME                                            DESIRED   CURRENT   READY   AGE
                     replicaset.apps/helm-lab-kb-kibana-7b55c76fc5   1         1         1       90s

                     NAME                                   READY   AGE
                     statefulset.apps/helm-lab-es-default   3/3     3m21s

[SUMMARY] HELM STACK INFO:
helm-lab elastic password: IxG62Fn1tfdb0u3BVGpFYpvcR
helm-lab elasticsearch endpoint: https://34.171.41.227:9200
helm-lab kibana endpoint: https://34.66.121.117:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

helmldap

  • Same as helm but adds an openldap server and reconfigures elasticsearch for ldap authentication. LDAP users are user01:password01 & user02:password02

helmbeats

  • Stands up a standard helm-lab deployment and filebeat & metricbeat to report on the kubernetes environment back to helm-lab deployment. For metricbeats the script will install kube-state-metrics to gather metrics.

helmlogstashbeats

  • Same as helmbeats however filebeat & metricbeat will output to logstash. Logstash will listen on port 5044 and send beats data to elasticsearch. All of the configurations and pipelines are stored in ConfigMaps

helmstackmonitor

  • Stands up a helm-lab environment and metricbeat to monitor helm-lab onto itself.

NATIVE Commands:

In the NATIVE commands both elasticsearch, kibana, & beats are stood up as just standard StatefulSets, Deployments, & ReplicaSets without using ECK or helm charts. Some of the features are limited from 7.3.0+ and others 7.5.0+ for native stack monitoring.

native

  • Stands up native-lab with 3 elasticsearch & 1 kibana
> ./deploy-elastick8s.sh native 8.4.1

 ********** Deploying ELASTIC STACK 8.4.1 Natively without helm or operator **************

[DEBUG] Create elastic user password and create a secret
[DEBUG] Create certificates for the stack and add create secrets

[DEBUG] Creating ca.crt and certificate/key for native-lab-es-default
[DEBUG] Creating secret native-lab-es-default-certificates

[DEBUG] Creating ca.crt and certificate/key for native-lab-kb-default
[DEBUG] Creating secret native-lab-kb-default-certificates
[DEBUG] Create elasticsearch.yaml
[DEBUG] Starting elasticsearch
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-default sts is showing 3 replicas ready

                     NAME                 READY   AGE
                     native-lab-default   3/3     105s

[DEBUG] Applying trial license.
[DEBUG] Creating kibana/kibana_system user.
[DEBUG] Creating kibana encryption key secret
[DEBUG] Create kibana.yaml
[DEBUG] Starting kibana
[DEBUG] native-lab-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-kibana is starting. Checking again in 10 seconds.  If this does not finish in few minutes something is wrong. CTRL-C please
✓ Sleeping 10 seconds Done
[DEBUG] native-lab-kibana deployment is showing 1 replicas ready

                     NAME                READY   UP-TO-DATE   AVAILABLE   AGE
                     native-lab-kibana   1/1     1            1           72s

[DEBUG] Grabbing elastic password for native-lab: 4w7ehkfhizQXeQ4vRjqXxWweb
[DEBUG] Grabbing elasticsearch endpoint for  native-lab: https://104.154.19.8:9200
[DEBUG] Grabbing kibana endpoint for native-lab: https://34.134.74.193:5601

[DEBUG] Stack is up but will take a minute or two to become healthy.  Please view kubectl get pods to ensure that all pods are up before trying to login

[SUMMARY] NATIVE STACK 8.4.1

                     NAME                                     READY   STATUS    RESTARTS   AGE
                     pod/native-lab-default-0                 1/1     Running   0          3m16s
                     pod/native-lab-default-1                 1/1     Running   0          3m16s
                     pod/native-lab-default-2                 1/1     Running   0          3m16s
                     pod/native-lab-kibana-6958dcfd4c-vxpsm   1/1     Running   0          79s

                     NAME                                  TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                         AGE
                     service/kubernetes                    ClusterIP      10.88.0.1     <none>          443/TCP                         97m
                     service/native-lab-default            LoadBalancer   10.88.7.43    104.154.19.8    9200:32446/TCP,9300:32733/TCP   3m18s
                     service/native-lab-default-headless   ClusterIP      None          <none>          9200/TCP,9300/TCP               3m17s
                     service/native-lab-kibana             LoadBalancer   10.88.10.56   34.134.74.193   5601:30442/TCP                  80s

                     NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
                     deployment.apps/native-lab-kibana   1/1     1            1           80s

                     NAME                                           DESIRED   CURRENT   READY   AGE
                     replicaset.apps/native-lab-kibana-6958dcfd4c   1         1         1       80s

                     NAME                                  READY   AGE
                     statefulset.apps/native-lab-default   3/3     3m17s

[SUMMARY] NATIVE STACK INFO:
native-lab elastic password: 4w7ehkfhizQXeQ4vRjqXxWweb
native-lab elasticsearch endpoint: https://104.154.19.8:9200
native-lab kibana endpoint: https://34.134.74.193:5601

[SUMMARY] ca.crt is located in /Users/jlim/elastick8s/ca.crt

nativebeats

  • Stands up native-lab with filebeat & metricbeats to monitor the kubernetes environment

nativestackmonitor

  • Stands up native-lab and uses metricbeats to monitor native-lab onto itself.

Enjoy!

jlim0930

2 Comments

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.