Rahul Singhai Blog

In my previous blog post, I described how to install and configure an ElasticSearch 5.x cluster. In this blog post, we will look at how to migrate data. Consult this table to verify that rolling upgrades are supported for your version of Elasticsearch. Full cluster upgrade (2.x to 5.x) We will have to do full cluster upgrade and restart . Install Elasticsearch Migration Helper on old cluster. This plugin will help you to check whether you can upgrade directly to the next major version of Elasticsearch, or whether you need to make changes to your data and cluster before doing so. cd /work/elk/elasticsearch-2.4.3/ curl -O -L https://github.com/elastic/elasticsearch-migration/releases/download/v2.0.4/elasticsearch-migration-2.0.4.zip ./bin/plugin install file:///work/elk/elasticsearch-2.4.3/elasticsearch-migration-2.0.4.zip Start old ElasticSearch: ./bin/elasticsearch & Browse elasticsearch-migration Click on "Cluster Checkup" > "Run checks now". C...

ElasticSearch 5.x requires a minimum of Max file descriptors 65536 and Max virtual memory areas 262144. It throws an error on start-up if these are set to very low value. ERROR: bootstrap checks failed max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] Check current values using: $ cat /proc/sys/fs/file-max 16384 $ cat /proc/sys/vm/max_map_count 65530 $ ulimit -Hn 16384 $ ulimit -Sn 4096 To fix this, following files need to change/add below settings: Recommended: Add a new file 99-elastic.conf under /etc/security/limits.d with following settings: elasticsearch - nofile 800000 elasticsearch - nproc 16384 defaultusername - nofile 800000 defaultusername - nproc 16384 Alternatively, edit /etc/sysctl.conf with following settings: fs.file-max = 800000 vm.max_map_count=300000

Curator is a tool from Elastic to help manage your ElasticSearch cluster. For certain logs/data, we use one ElasticSearch index per year/month/day and might keep a rolling 7 day window of history. This means that every day we need to create, backup, and delete some indices. Curator helps make this process automated and repeatable. Installation Curator is written in Python , so will need pip to install it: pip install elasticsearch-curator curator --config ./curator_cluster_config.yml curator_actions.yml --dry-run Configuration Create a file curator_cluster_config.yml with following contents: --- # Remember, leave a key empty if there is no value. None will be a string, not a Python "NoneType" client: hosts: - "es_coordinating_01.singhaiuklimited.com" port: 9200 url_prefix: use_ssl: True # The certificate file is the CA certificate used to sign all ES node certificates. # Use same CA certificate to generate and sign the certificate running ...

ElasticSearch X-pack documentation a good description on how to secure your ElasticSearch cluster using SSL/TLS. I used certgen to generate certificates for all the nodes as below: Create a instances.yml file: vim /work/elk/elasticsearch-5.6.2/config/x-pack/instances.yml instances: - name: "hostname-00" ip: - "192.126.0.163" - "192.0.2.2" - "198.51.100.1" dns: - "hostname-00" - "hostname-00.mydomain.name" - name: "hostname-01" ip: - "192.126.0.164" dns: - "hostname-01" - "hostname-01.mydomain.name" - name: "hostname-02" - name: "CN=hostname-03,C=GB,ST=Greater London,L=London,O=OrgName,OU=OrgUnit,DC=mydomain,DC=com" dns: - "hostname-03.mydomain.name" - "hostname-03.internal" - "hostname-03" Run below command to generate a CA certificate and...

We recently upgraded from ElasticSearch 2.4.3 to 5.6.2. Below are the steps that we used to install and configure new ELK cluster. ElasticSearch Installation mkdir -p /work/elk/data/data-es562 mkdir -p /work/elk/data/logs-es562 mkdir -p /work/elk/data/repo-es562 cd /work/elk/ curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.2.tar.gz tar -zxvf elasticsearch-5.6.2.tar.gz curl -O https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.6.2.zip cd /work/elk/elasticsearch-5.6.2 ./bin/elasticsearch-plugin install file:///work/elk/x-pack-5.6.2.zip Configuration Settings for Master + Ingest node in elasticsearch.yml We have reused master nodes as ingest nodes, because we don't have any heavy ingest pipelines, and x-pack monitoring requires at-least one ingest node to be present in the cluster. cluster.name: ESDev562 node.name: "master_01" node.master: true # Enable the node.master role (enabled by default). node.data: false ...

Following up my previous blog post below is a sample Logstash config that can be used to connect to the ElasticSearch for the output of data: cd /work/elk/logstash-5.2.6/ vim ./config/twitter_feeds_consumer/twitter_feeds_consumer.conf input { kafka { topics => ["twitter_feeds_kafka_topic_name"] bootstrap_servers => "kafka-broker-1.domain.name:9092,kafka-broker-2.domain.name:9092" # consumer_threads => 5 # auto_offset_reset => "earliest" group_id => "logstash562_twitter_feeds_consumer_group" codec => json { charset => "ISO-8859-1" } } } output { # stdout { codec => "rubydebug" } elasticsearch { hosts => ["https://coord_01:9200"] index => "index-name-%{+YYYY.MM.dd}" ssl => true cacert => '/work/elk/logstash-5.6.2/config/ca.crt' user => logstash_internal password => logstash_internal_password } } Logs...

Watch: Compare Vs Script condition If you need to do comparisons with aggregation buckets in watchers, then instead of this: "condition": { "compare": { "ctx.payload.aggregations.agg1.buckets.0.doc_count": { "gte": 5 } } } please prefer this, as this takes care of "Index out of bound" exceptions: "condition": { "script": { "source": "return (ctx.payload.aggregations.agg1.buckets.size() > 0 && ctx.payload.aggregations.agg1.buckets.0.doc_count >= params.threshold)", "lang": "painless", "params": { "threshold": 5 } } } Hard-coded variables in scripts ElasticSearch throws circuit_breaking_exception error when it sees more than 15 new dynamic scripts within a minute. Sometimes this error is also thrown when master goes down abruptly. This is so because, it has to compile every new/unique script it s...