0

Generate test data using elastic’s makelogs

Updated 5/11/2023

When I am testing for various issues I find that I need to create some test data. The built in sample data is great but sometimes you need to keep ingesting data or ingest some older data to reproduce issues. I found this repo and it was the answers to my issue.

I did not want to install this and wanted to containerize it so that I can leave my system clean yet be able to use this and re-use it again.

makelogs runs fine as it is however if your ES endpoint is secured with TLS and does not have proper certificates it makelogs will fail.

Please follow the steps below to create the docker image

$ git clone https://github.com/elastic/makelogs.git
$ cd makelogs

NOTE: below step is no longer needed as source was updated to add --insecure flag

##########
We will be editing _client.js.

Find the line

  host: url

and replace it with

  host: url,
  ssl:{ rejectUnauthorized: false, pfx: [] }

##########

Now run the following to build the docker image

$ docker build -t makelogs .
[+] Building 14.8s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                              0.0s
 => => transferring dockerfile: 37B                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => => transferring context: 34B                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/node:16.13.0                                                                                                                   1.0s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                 0.0s
 => => transferring context: 4.72kB                                                                                                                                               0.0s
 => CACHED [1/4] FROM docker.io/library/node:16.13.0@sha256:580a0850049c59a48f06090edd48c9f966c5e6572bbbabc369ba3ecbc4855dba                                                      0.0s
 => [2/4] COPY / /app                                                                                                                                                             0.0s
 => [3/4] WORKDIR /app                                                                                                                                                            0.0s
 => [4/4] RUN yarn install                                                                                                                                                       12.3s
 => exporting to image                                                                                                                                                            1.4s
 => => exporting layers                                                                                                                                                           1.4s
 => => writing image sha256:d181f96bb76508bac1e84fe806aeeb8d95c609c0b816e27c0665d6cae62f4dd5                                                                                      0.0s
 => => naming to docker.io/library/makelogs

Some examples on how to use this.

Lets say that our endpoint is https://192.168.1.10:9200 and our login is elastic:password for all of our examples

  • To get help docker run --rm makelogs --help
  • To connect and send events docker run --url "https://elastic:password@192.168.1.10:9200"

Various options you can add and remove

  • -c – defaults to 14000 documents but you can specify it -c 1t for 1k records
  • -d – defaults to today but you can set X/Y where X is number of days before and Y is number of days after
  • --indexPrefix – default logstash- however you can specify it
  • -s – default 1 – number of shards
  • -r – default 0 – number of replica shards
  • --dry – dry run
  • --reset – clear all indexPrefix-* indicies and recreate
  • --no-reset – does not reset but just adds to the existing indexPrefix-*
  • --insecure – if your endpoint is self-signed or private

Sample run

$ docker run --rm makelogs --url "https://elastic:password@192.168.1.10:9200" --reset --indexPrefix "test-"
Generating 14000 events from 2022-01-25T00:00:00Z to 2022-01-27T23:59:59Z --insecure
creating index template for "test-*"

created 14000 events in 10 seconds.

file

file

If you are using my deploy-elastic.sh script to run your ES cluster in docker containers please use your host’s IP address to run the makelogs.

$ docker run --rm makelogs --url "https://elastic:password@192.168.1.10:9200"  --indexPrefix "testagain-" -c 1t --days "10/10" --insecure
Generating 1000 events from 2022-01-16T00:00:00Z to 2022-02-05T23:59:59Z
creating index template for "testagain-*"

created 1000 events in 1 seconds.

file

jlim0930

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.