February 2022 – Justin Lim

0

Create secrets to use with your elastic stack to create keystore in kubernetes – minio example – helm example included

Posted on February 25, 2022 by jlim0930

For this example I will stand up a very simple minio server on my localhost. Create kubernetes secrets for the s3.client.default.access_key and s3.client.default.secret_key. Configure my elasticsearch pod with initContainer to install the repository-s3 plugin and secureSettings to create the keystore. minio server This is a very simple, not secure setup just for testing $ mkdir data $ wget https://dl.min.io/server/minio/release/linux-amd64/minio $ chmod +x minio $ ./minio server ./data API: http://192.168.1.251:9000 http://172.17.0.1:9000 http://192.168.122.1:9000 http://192.168.49.1:9000 http://127.0.0.1:9000 RootUser: minioadmin RootPass: minioadmin Console: http://192.168.1.251:36012 http://172.17.0.1:36012 http://192.168.122.1:36012 http://192.168.49.1:36012 http://127.0.0.1:36012 RootUser: minioadmin RootPass: minioadmin Command-line: https://docs.min.io/docs/minio-client-quickstart-guide $ mc alias set myminio http://192.168.1.251:9000 minioadmin minioadmin Instead of getting… Continue Reading →

2

elasticsearch stack monitoring on kubernetes(ECK)

Posted on February 25, 2022 by jlim0930

UPDATE: Since ECK operator 1.7 there is a new way to deploy stack monitoring. I’ve added the section at the end to cover for this. The old method still works and depending on your use case you can still use the old method versus new. The main difference between the OLD and NEW is that the OLD method uses filebeat and metricbeat pods while the new method uses filebeat and metricbeat sidecar containers. Lets get started with monitoring your elastic stack in kubernetes. There are multiple ways of doing this and I will divide this up into 3 sections, using… Continue Reading →

1

elasticsearch 8.x configure roles and users for stack monitoring

Posted on February 17, 2022 by jlim0930

A lot of changes were made with elasticsearch 8.0 release but the main one that affected me the most was the ability to use the default elastic(superuser role) to access system indices. According to https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights.html#_better_protection_for_system_indices we no longer have write access to system indices with the superuser role. When configuring stack monitoring using metricbeats this is an issue if you’ve been just using the elastic user to configure the monitoring. You can add allow_restricted_indices to the role to enable it back however its the lazy and insecure method. I’ve summarized the creation of roles and users for metricbeats and filebeats… Continue Reading →

0

How to run setup for various beats(filebeat, metricbeat, & more) in kubernetes to load dashboards and more

Posted on February 8, 2022 by jlim0930

Whenever you install beats or update beats its best if you run the setup so that the setup will update your index templates, ILM, visualizations, etc. I found that its always best before installing or upgrading to use a seed host to just run the setup before deploying or updating across your environment and to turn off template updates and dashboard loading on the beats locally so that your elasticsearch cluster is not flooded. In a baremetal environment this is easy to do but in docker and in kubernetes it gets a bit difficult. Sure you can setup a initContainer… Continue Reading →

1

ECE(elastic cloud enterprise) snapshot repository using object storage(minio) with self-signed TLS

Posted on February 3, 2022 by jlim0930

Just FYI this is a complete rewrite of the previous article. It should be better organized and more closely align with real world situations. We will stand up a minio server with self-signed or internally signed SSL certificate that is not publicly trusted. This guide will work with any s3 compliant object storage not just minio. The certificate that we will create will be for internal use but you can and should create a more secure certificate to be used in your environment. If you have s3 compliant object storage with publicly trusted signed SSL certificate then all you would… Continue Reading →

0

Creating a custom CA and certificates with SAN signed by the CA for testing

Posted on February 2, 2022 by jlim0930

There are endless guides on the internet that uses endless methods to generate certificates and CA but I have not found one that is simple to use to create a custom CA and CA signed certificate with SAN (Subject Alternative Name) to perform some testing. So I created simple set of steps: Create a ca.key ❯ openssl genrsa -out ca.key 2048 Generating RSA private key, 2048 bit long modulus …………………………………………………………………………….+++ …………….+++ e is 65537 (0x10001) ❯ ls ca.key ❯ cat ca.key —–BEGIN RSA PRIVATE KEY—– MIIEowIBAAKCAQEArTBn8M9NBHmmVKOCcKl75EYZqv2LvNvNQjSJ0YDHLrXt2CpL x3N7IDgriLU4TbFVb13yQCPXESQOOBuzNJMqG8Ca5K56RYNBY6QP5k8z195385Qd QJODgut3A+ACjkWB9iVMHuN+KVXyEcPhl4/UFvYW6+ybvSSmUgfZJ/u3fCh6YaWN 15nVWRI40yDAgvM8EKuPew734ENF1GdSVF+S+m2QhDKR2gn8NkpdvYjKDtKFN2Rh VjREEb3TuQqttZNykkCira93dr3/ILdTGVxOIkXhdESFdiRMZ9dXOUqozQSALGfw cXKY2MlrkN+20F/ojoh3IqZs2gJS05udJOaarQIDAQABAoIBADdnBckmN6gX1lq7 F848mZJzzmBBzcLzuZzVO8VWYeGSd2ywUx+R1LCA54RLHKDV+tOuhQF5taIZG6dd TR2jelP4cFR5cEnubCuY3zE44wfKdiroldcDmY13D9KghZDHsYRxeAFlmwVUJiUC uZcHfrx9quV8AnImWEJjmldNEexYa94tLM+SahbNEG3H6s2hQE+GvshoVp+IoPPg sIVZo7AyEj/Z6sdY67YX92aNHeFr1TvSO/knxAryBW1knui8ZlmpFIroEZ8eR/Cn 1SYIWaJZpzz5pmbKIOWHc2uM7rUJzW3Rm8hFzjOCiUgmmYjQROVvLKFMTtiV6mcl IATnsqECgYEA0pR5fjqdf9NG3C+k8bh/PxKX2MzBmaFMjMdini081l9oN6D1Hjiz Pw3NZ0iVb+pQceeVLss8v/mX/zhkQ0P+rbiz4Lh191few5/uAxhqVJUVbMPjCYMz qxTTK3pcpui4PPM8pfFOBFbuXc4QqUf5HYW6HUq0eWWy86U9cB0W0wUCgYEA0otU r8LlxclOYZkcxqbYe+0/8xxCXaYw5slTbyxlh40wX2vgjmSkeOli8peGfMD07Cfm… Continue Reading →

0

Rescuing pods from CrashLoopBackOff

Posted on February 1, 2022 by jlim0930

When pods behave badly in your cluster, looping over and over, it is known as CrashLoopBackOff. If the pod contains important data or you just need to edit something on the pod to get it fixed like running some checker on a program installed on the pod or moving things around or whatever the reason is but you need to get into the pod to fix it but there is no way you can just kubectl -n namespace exec -it name bash to it then what do you do? dang you CrashLoopBackOff!! initContainer method Advantage: the pod runs before the… Continue Reading →