Rahul Singhai Blog

If you try to run wget or curl in Ununtu/Debian behind corporate proxy, you might receive errors like: ERROR: The certificate of 'apertium.projectjj.com' is not trusted. ERROR: The certificate of 'apertium.projectjj.com' doesn't have a known issuer. wget https://apertium.projectjj.com/apt/apertium-packaging.public.gpg ERROR: cannot verify apertium.projectjj.com's certificate, issued by 'emailAddress=proxyteam@corporate.proxy.com,CN=diassl.corporate.proxy.com,OU=Division UK,O=Group name,L=Company,ST=GB,C=UK': Unable to locally verify the issuer's authority. To connect to apertium.projectjj.com insecurely, use `--no-check-certificate'. To solution is to install your company's CA certificate in Ubuntu. In Windows, open the first part of URL in your web browser. e.g. open https://apertium.projectjj.com in web browser. If you inspect the certifcate, you will see the same CN (diassl.corporate.proxy.com), as reported by the error above ...

While trying to index a document, if ElasticSearch throws following exception: elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, 'cluster_block_exception', 'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];') then, this means that ElasticSearch has cluster-wide locked writes to all indices. As a temporary solution, you can unlock writes to the cluster (all indexes) using: curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' but, this might still result in locking of indices because underlying problem has not been fixed. Most often it is caused by exceeding the disk watermark / quota. In its default configuration, ElasticSearch will not allocate any more disk space when more than 90% of the disk are used overall (i.e. by ElasticSearch or other applications). We can reduce this watermark extremely low using: curl ...

By default, there is no encryption, authentication, or ACLs configured in Elasticsearch/Kafka. Any client can communicate to ES nodes / Kafka brokers via the PLAINTEXT port. It is critical that access via this port is restricted to trusted clients only. Network segmentation and/or authorization ACLs can be used to restrict access to trusted IPs in such cases. If neither is used, the cluster is wide open and can be accessed by anyone. While non-secured clusters are supported, as are a mix of authenticated, unauthenticated, encrypted and non-encrypted clients, it is recommended to secure the components in your cluster. Secure Sockets Layer (SSL) is the predecessor of Transport Layer Security (TLS) , and SSL has been deprecated since June 2015. However, generally people use the term SSL instead of TLS in configuration and code. SSL can be configured for encryption or authentication. You may configure just SSL encryption (by default SSL encryption includes certificate authenticatio...

While trying to install logstash plugin, I was getting below error: $ /work/logstash/logstash-5.5.2/bin/logstash-plugin install logstash-input-cloudwatch WARNING: A maven settings file already exist at ~/.m2/settings.xml, please review the content to make sure it include your proxies configuration. Validating logstash-input-cloudwatch Installing logstash-input-cloudwatch Error Bundler::InstallError, retrying 1/10 An error occurred while installing logstash-core (5.5.2), and Bundler cannot continue. Make sure that `gem install logstash-core -v '5.5.2'` succeeds before bundling. Error Bundler::InstallError, retrying 2/10 An error occurred while installing logstash-core (5.5.2), and Bundler cannot continue. Make sure that `gem install logstash-core -v '5.5.2'` succeeds before bundling. Here are the things I did to make it work: Created maven ~/.m2/settings.xml file <?xml version="1.0" encoding="UTF-8"?> <settings xmlns="http://maven.ap...

Let's say we want to read and parse a CSV file containing Key value pairs. We will be using Alpakka's CSVParser for this. A snippet of a file (src/main/resources/CountryNicCurrencyKeyValueMap.csv) that shows mapping from Country NIC code to currency code with pipe (|) as field delimiter: AD|EUR AE|AED AF|AFN AG|XCD AI|XCD AL|ALL AM|AMD AN|ANG AO|AOA AQ|AQD AR|ARS AS|EUR AT|EUR AU|AUD AW|ANG AX|EUR AZ|AZN BA|BAM BB|BBD BD|BDT BE|EUR BF|XOF BG|BGN BH|BHD BI|BIF BJ|XOF BL|EUR BM|BMD BN|BND BO|BOB BR|BRL BS|BSD BT|INR Following is the code: import java.io.File import java.nio.charset.StandardCharsets import akka.actor.ActorSystem import akka.stream._ import akka.stream.alpakka.csv.scaladsl.CsvParsing import akka.stream.scaladsl.{FileIO, Flow, Sink} import akka.util.ByteString import scala.collection.immutable import scala.concurrent.{ExecutionContext, _} import scala.concurrent.duration._ implicit val system: ActorSystem = ActorSystem("TestApplication") implicit ...

Performance Tuning of Kafka is critical when your cluster grow in size. Below are few points to consider to improve Kafka performance: Consumer group ID : Never use same exact consumer group ID for dozens of machines consuming from different topics. All of those commits will end up on the same exact partition of __consumer_offsets , hence the same broker, and this might in turn cause performance problems. Choose the consumer group ID to group_id+topic_name . Skewed : A broker is skewed if its number of partitions is greater that the average of partitions per broker on the given topic. Example: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2). Try to make sure that none of the brokers is skewed. Spread : Brokers spread is the percentage of brokers in the cluster that has partitions for the given topic. Example: 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic. Try to achieve 100% broker spread...

In my previous blog post, I described how to install and configure an ElasticSearch 5.x cluster. In this blog post, we will look at how to migrate data. Consult this table to verify that rolling upgrades are supported for your version of Elasticsearch. Full cluster upgrade (2.x to 5.x) We will have to do full cluster upgrade and restart . Install Elasticsearch Migration Helper on old cluster. This plugin will help you to check whether you can upgrade directly to the next major version of Elasticsearch, or whether you need to make changes to your data and cluster before doing so. cd /work/elk/elasticsearch-2.4.3/ curl -O -L https://github.com/elastic/elasticsearch-migration/releases/download/v2.0.4/elasticsearch-migration-2.0.4.zip ./bin/plugin install file:///work/elk/elasticsearch-2.4.3/elasticsearch-migration-2.0.4.zip Start old ElasticSearch: ./bin/elasticsearch & Browse elasticsearch-migration Click on "Cluster Checkup" > "Run checks now". C...

ElasticSearch 5.x requires a minimum of Max file descriptors 65536 and Max virtual memory areas 262144. It throws an error on start-up if these are set to very low value. ERROR: bootstrap checks failed max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] Check current values using: $ cat /proc/sys/fs/file-max 16384 $ cat /proc/sys/vm/max_map_count 65530 $ ulimit -Hn 16384 $ ulimit -Sn 4096 To fix this, following files need to change/add below settings: Recommended: Add a new file 99-elastic.conf under /etc/security/limits.d with following settings: elasticsearch - nofile 800000 elasticsearch - nproc 16384 defaultusername - nofile 800000 defaultusername - nproc 16384 Alternatively, edit /etc/sysctl.conf with following settings: fs.file-max = 800000 vm.max_map_count=300000

Curator is a tool from Elastic to help manage your ElasticSearch cluster. For certain logs/data, we use one ElasticSearch index per year/month/day and might keep a rolling 7 day window of history. This means that every day we need to create, backup, and delete some indices. Curator helps make this process automated and repeatable. Installation Curator is written in Python , so will need pip to install it: pip install elasticsearch-curator curator --config ./curator_cluster_config.yml curator_actions.yml --dry-run Configuration Create a file curator_cluster_config.yml with following contents: --- # Remember, leave a key empty if there is no value. None will be a string, not a Python "NoneType" client: hosts: - "es_coordinating_01.singhaiuklimited.com" port: 9200 url_prefix: use_ssl: True # The certificate file is the CA certificate used to sign all ES node certificates. # Use same CA certificate to generate and sign the certificate running ...

ElasticSearch X-pack documentation a good description on how to secure your ElasticSearch cluster using SSL/TLS. I used certgen to generate certificates for all the nodes as below: Create a instances.yml file: vim /work/elk/elasticsearch-5.6.2/config/x-pack/instances.yml instances: - name: "hostname-00" ip: - "192.126.0.163" - "192.0.2.2" - "198.51.100.1" dns: - "hostname-00" - "hostname-00.mydomain.name" - name: "hostname-01" ip: - "192.126.0.164" dns: - "hostname-01" - "hostname-01.mydomain.name" - name: "hostname-02" - name: "CN=hostname-03,C=GB,ST=Greater London,L=London,O=OrgName,OU=OrgUnit,DC=mydomain,DC=com" dns: - "hostname-03.mydomain.name" - "hostname-03.internal" - "hostname-03" Run below command to generate a CA certificate and...

We recently upgraded from ElasticSearch 2.4.3 to 5.6.2. Below are the steps that we used to install and configure new ELK cluster. ElasticSearch Installation mkdir -p /work/elk/data/data-es562 mkdir -p /work/elk/data/logs-es562 mkdir -p /work/elk/data/repo-es562 cd /work/elk/ curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.2.tar.gz tar -zxvf elasticsearch-5.6.2.tar.gz curl -O https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.6.2.zip cd /work/elk/elasticsearch-5.6.2 ./bin/elasticsearch-plugin install file:///work/elk/x-pack-5.6.2.zip Configuration Settings for Master + Ingest node in elasticsearch.yml We have reused master nodes as ingest nodes, because we don't have any heavy ingest pipelines, and x-pack monitoring requires at-least one ingest node to be present in the cluster. cluster.name: ESDev562 node.name: "master_01" node.master: true # Enable the node.master role (enabled by default). node.data: false ...

Following up my previous blog post below is a sample Logstash config that can be used to connect to the ElasticSearch for the output of data: cd /work/elk/logstash-5.2.6/ vim ./config/twitter_feeds_consumer/twitter_feeds_consumer.conf input { kafka { topics => ["twitter_feeds_kafka_topic_name"] bootstrap_servers => "kafka-broker-1.domain.name:9092,kafka-broker-2.domain.name:9092" # consumer_threads => 5 # auto_offset_reset => "earliest" group_id => "logstash562_twitter_feeds_consumer_group" codec => json { charset => "ISO-8859-1" } } } output { # stdout { codec => "rubydebug" } elasticsearch { hosts => ["https://coord_01:9200"] index => "index-name-%{+YYYY.MM.dd}" ssl => true cacert => '/work/elk/logstash-5.6.2/config/ca.crt' user => logstash_internal password => logstash_internal_password } } Logs...

Watch: Compare Vs Script condition If you need to do comparisons with aggregation buckets in watchers, then instead of this: "condition": { "compare": { "ctx.payload.aggregations.agg1.buckets.0.doc_count": { "gte": 5 } } } please prefer this, as this takes care of "Index out of bound" exceptions: "condition": { "script": { "source": "return (ctx.payload.aggregations.agg1.buckets.size() > 0 && ctx.payload.aggregations.agg1.buckets.0.doc_count >= params.threshold)", "lang": "painless", "params": { "threshold": 5 } } } Hard-coded variables in scripts ElasticSearch throws circuit_breaking_exception error when it sees more than 15 new dynamic scripts within a minute. Sometimes this error is also thrown when master goes down abruptly. This is so because, it has to compile every new/unique script it s...

Check MirrorMaker.scala for more details. Target cluster setup Download and install Kafka (target cluster). Select appropriate version and download its tgz from Kafka Downloads page. tar -zxf kafka_2.11-0.10.0.1.tgz cd kafka_2.11-0.10.0.1 Configure Target Kafka cluster's ZooKeeper vim ./config/zookeeper.properties # the directory where the snapshot is stored. dataDir=/work/kafka_2.11-0.10.0.1/zookeeper-data # the port at which the clients will connect clientPort=2181 # disable the per-ip limit on thseparatedof connections since this is a non-production config maxClientCnxns=0 Start Target Kafka cluster's ZooKeeper ./bin/zookeeper-server-start.sh config/zookeeper.properties Configure Target Kafka cluster's Server vim ./config/server.properties # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 # The number of threads handling network requests num.network.threads=3 # The number of threads doing disk I/O num.io.threads=8 # The sen...

The Subject Alternative Name field lets you specify additional host names (sites, IP addresses, common names, etc.) to be protected by a single SSL Certificate, such as a Multi-Domain (SAN) or Extend Validation Multi-Domain Certificate. Benefits Secure Host Names on Different Base Domains in One SSL Certificate: A Wildcard Certificate can protect all first-level subdomains on an entire domain, such as *.example.com. However, a Wildcard Certificate cannot protect both www.example.com and www.example.net. Virtual Host Multiple SSL Sites on a Single IP Address: Hosting multiple SSL-enabled sites on a single server typically requires a unique IP address per site, but a Multi-Domain (SAN) Certificate with Subject Alternative Names can solve this problem. Microsoft IIS and Apache are both able to Virtual Host HTTPS sites using Multi-Domain (SAN) Certificates. Greatly Simplify Your Server's SSL Configuration: Using a Multi-Domain (SAN) Certificate saves you the hassle and time invo...

ElasticSearch has a concept of bucket selection generated from aggregation. This works as a pipeline, where first aggregation generates buckets, and then bucket selection further filters out buckets. We have an ElasticSearch index ' daily_reports ', where a row represents a particular version of report. When a report is created a new row is inserted in the index with a new ' reportId ' field value and ' publishDate ' field representing the UNIX timestamp. Each report/row has multiple other fields representing properties of the report, for e.g., ' title ', ' activity ', ' reportStatus ', ' reportLevel ', etc. When the report is edited/deleted, a new row is inserted into the index, with same ' reportId ', but different '_id', 'publishDate', 'reportLevel' etc. Now if user wants to get the latest version for each report matching a particular filter criterion ( reportLevel = Monitoring AND repor...