Skip to main content

A sample Logstash config to connect to ElasticSearch with TLS

Following up my previous blog post below is a sample Logstash config that can be used to connect to the ElasticSearch for the output of data:
cd /work/elk/logstash-5.2.6/
vim ./config/twitter_feeds_consumer/twitter_feeds_consumer.conf

input {
  kafka {
    topics => ["twitter_feeds_kafka_topic_name"]
    bootstrap_servers => ","
    # consumer_threads => 5
    # auto_offset_reset => "earliest"
    group_id => "logstash562_twitter_feeds_consumer_group"
    codec => json { charset => "ISO-8859-1" }

output {
# stdout { codec => "rubydebug" }

  elasticsearch {
    hosts => ["https://coord_01:9200"]
    index => "index-name-%{+YYYY.MM.dd}"
    ssl => true
    cacert => '/work/elk/logstash-5.6.2/config/ca.crt'
    user => logstash_internal
    password => logstash_internal_password

Logstash 5.x onwards require that every logstash process specifies its own data path folder.
To do that follow below steps:
mkdir -p /work/elk/data/data-logstash562/twitter_feeds_consumer
mkdir -p /work/elk/data/logs-logstash562/twitter_feeds_consumer/
./bin/logstash -f ./config/twitter_feeds_consumer/twitter_feeds_consumer.conf -w 5 -l /work/elk/data/logs-logstash562/twitter_feeds_consumer &

Important flags used are:
-f Logstash config file
-w Sets the number of pipeline workers that will, in parallel, execute the filter and output stages of the pipeline. Logstash 5.x onwards, you need to specify different data folders for every Logstash process.
-t Check configuration for valid syntax and then exit
-r Reload config automatically
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location for changes, in seconds. The default is every 3 seconds.


Popular posts from this blog

ElasticSearch max file descriptors too low error

ElasticSearch 5.x requires a minimum of Max file descriptors 65536 and Max virtual memory areas 262144.
It throws an error on start-up if these are set to very low value.
ERROR: bootstrap checks failed max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Check current values using:
$ cat /proc/sys/fs/file-max 16384 $ cat /proc/sys/vm/max_map_count 65530 $ ulimit -Hn 16384 $ ulimit -Sn 4096
To fix this, following files need to change/add below settings:
Recommended: Add a new file 99-elastic.conf under /etc/security/limits.d with following settings:
elasticsearch - nofile 800000 elasticsearch - nproc 16384 defaultusername - nofile 800000 defaultusername - nproc 16384 Alternatively, edit /etc/sysctl.conf with following settings:
fs.file-max = 800000 vm.max_map_count=300000

Kafka performance tuning

Performance Tuning of Kafka is critical when your cluster grow in size. Below are few points to consider to improve Kafka performance:
Consumer group ID: Never use same exact consumer group ID for dozens of machines consuming from different topics. All of those commits will end up on the same exact partition of __consumer_offsets, hence the same broker, and this might in turn cause performance problems. Choose the consumer group ID to group_id+topic_name.
Skewed: A broker is skewed if its number of partitions is greater that the average of partitions per broker on the given topic. Example: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2). Try to make sure that none of the brokers is skewed.
Spread: Brokers spread is the percentage of brokers in the cluster that has partitions for the given topic. Example: 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic. Try to achieve 100% broker spread.
Leader skew…

Procedure for name and date of birth change (Pune)

For change of name, the form (scribd) is available free of cost at Government Book Depot (Shaskiya Granthagar), which is located near Collector’s office, next to Saint Helena's School. The postal address is:
Government Photozinco Press Premises and Book Depot,
5, Photozinco Press Road, Pune, MH, 411001.
Wikimapia link

Charges for name or date of birth change, in the Maharashtra Government Gazette:
INR 120.00 per insertion (for two copies of the Gazette)
For backward class applicants: INR 60.00
Charges for extra copy of the Gazette: INR 15.00 per copy (two copies are enough, so you may not want to pay extra for extra copies).

Backward class applicants are required to submit a xerox of caste certificate of old name as issued by the Collector of the District concerned.

Once the form is duly submitted, it normally takes 10 to 15 days for publication of advertisement in the Maharashtra Government Gazette. The Gazette copy reaches to the address filled in the form within next 7 to 15 day…